I just finished posting a neat syslog summary script that was passed to me by Phil Koontz. He uses it every day to keep track of syslog events. You can download it from the Netcraftsmen website: http://www.netcraftsmen.net/Resources/Technical Articles/Syslog Summary Scripts. (While you're there, you may want to look at other articles and blogs.)
The summary page produced by this script is shown below. I've found that it scales well on large networks that generate a lot of events. Phil has used it on a network of over 2500 routers and switches. I'm currently using it on a network of nearly 600 routers and switches. At those scales, the report is seldom longer than two or three pages.
Summary of Cisco syslog Messages on Sun Oct 11 23:59:01 2009
Cisco Messages:
18 LINEPROTO-5-UPDOWN
9 OSPF-5-ADJCHG
7 SNMP-3-AUTHFAIL
2 BGP-5-ADJCHANGE
2 LINK-3-UPDOWN
1 BGP-3-NOTIFICATION
Messages sorted by frequency and source device:
8 d04-3550-03 d04-3550-03 LINEPROTO-5-UPDOWN FastEthernet0/13
4 d19-3400-01 d19-3400-01 LINEPROTO-5-UPDOWN FastEthernet0/19
2 d02-2811-01 d02-2811-01 SNMP-3-AUTHFAIL
2 d03-2811-01 d03-2811-01 SNMP-3-AUTHFAIL
2 d45-3560-01 d45-3560-01 LINEPROTO-5-UPDOWN GigabitEthernet0/17
2 d19-3400-01 d19-3400-01 LINK-3-UPDOWN FastEthernet0/19
2 d48-7604-01 d48-7604-01 OSPF-5-ADJCHG
2 d16-7604-01 d16-7604-01 BGP-5-ADJCHANGE
2 d16-7604-01 d16-7604-01 SNMP-3-AUTHFAIL
2 d64-3550-05 d64-3550-05 LINEPROTO-5-UPDOWN FastEthernet0/2
2 d22-7604-01 d22-7604-01 OSPF-5-ADJCHG
1 d14-6504-01 d14-6504-01 OSPF-5-ADJCHG
1 d38-7604-01 d38-7604-01 OSPF-5-ADJCHG
1 d38-7604-01 d38-7604-01 SNMP-3-AUTHFAIL
1 d89-3560-01 d89-3560-01 LINEPROTO-5-UPDOWN Vlan3264
1 d89-3560-01 d89-3560-01 OSPF-5-ADJCHG
1 d16-7604-01 d16-7604-01 BGP-3-NOTIFICATION
1 d27-3560-01 d27-3560-01 LINEPROTO-5-UPDOWN Vlan3265
1 d27-3560-01 d27-3560-01 OSPF-5-ADJCHG
1 d92-3560-01 d92-3560-01 OSPF-5-ADJCHG
Included in the download zip file are scripts for web page links to the summary reports and for automatically running it from cron. It is intended to be installed on a Linux-type system. I've used syslog-ng as the underlying syslog server because it is free, scales to large syslog volume, automatically does log file rotation, and has great filtering and forwarding capabilities.
I use the report to look for rare, but critical events -- often the ones that appear only once per month. I also look for unstable interfaces or devices -- they have higher counts. Reading the summary is certainly better than scanning a long syslog log file from the prior day. I find that a quick scan of the summary is a good first step in monitoring what's happening on the network. It is easy to implement and use. I like to think of it as a quick event analysis system and the cost is certainly right - just the time and effort required to add to an existing syslog server.
If you improve the scripts, and don't mind sharing your improvements, please send them to me and I'll post them, with credit to you. I'd like to see the second column be the device IP address instead of another copy of the name. And I'd like to have it look up the interface descriptions, possibly taken from a config file repository, and add that after the interface name for interface events. The zip file includes the original script from Phil as well as a modified version that we've been using in customer implementations.
I'd like to know what you use to track events. In upcoming posts, I'll share what I think are the fundamental requirements for handling events.
-Terry