nv-l

RE: [nv-l] Cursed Cisco Trap Formats

2002-10-03 14:43:24
Subject: RE: [nv-l] Cursed Cisco Trap Formats
From: "Barr, Scott" <Scott_Barr AT csgsystems DOT com>
To: <nv-l AT lists.tivoli DOT com>
Date: Thu, 3 Oct 2002 13:43:24 -0500
Well yes, in a perfect world that would be true. BUT you can't necessarily do that. Some of the equipment does not support the newer versions of the IOS because of physical constraints (processor and memory) or the newer versions introduce bugs (such as port-scanning reboots the router). I don't disagree with you, but it is simply not possible to do. These routers are a variety of classes, (some 2500, some 3000, some 4000, some 7000, 7200, 7500 on and on) and they support a variety of protocols (some have ATM, some have frame, some have point to point serial, some have ISDN). It just can't be easily done.
 
 
So, let me go back to my original request - how can I handle this with automation?
-----Original Message-----
From: Stringfellow, William [mailto:William.Stringfellow AT bankofamerica DOT com]
Sent: Thursday, October 03, 2002 1:41 PM
To: Barr, Scott; nv-l AT lists.tivoli DOT com
Subject: RE: [nv-l] Cursed Cisco Trap Formats

If that is true, sounds like you need to standardize the IOS version you run on that type of device.
 
        Bill
-----Original Message-----
From: Barr, Scott [mailto:Scott_Barr AT csgsystems DOT com]
Sent: Thursday, October 03, 2002 10:16 AM
To: nv-l AT lists.tivoli DOT com
Subject: RE: [nv-l] Cursed Cisco Trap Formats

All three varities come in with the same enterprise ID. Not sure how this would help.
-----Original Message-----
From: Stringfellow, William [mailto:William.Stringfellow AT bankofamerica DOT com]
Sent: Thursday, October 03, 2002 12:01 PM
To: Barr, Scott; nv-l AT lists.tivoli DOT com
Subject: RE: [nv-l] Cursed Cisco Trap Formats

Scott,
    We have seen this many times because the particular Cisco device is sending it's own version of Link UP/Down traps (there are many devices that have unique ways of sending what should be a generic trap.)   We learned about this when we first put MLMs in place and started seeing the raw trap varbinds. 
    Anyway, to fix it, figure out what the oid is for the device that is giving you the wrong number of varbinds, create a new trapd.conf entry for it in the enterprise piece, then add LinkUp and LinkDown specific traps to your menu for that enterprise.
    Under the "Event Log Message" use the generic "enterprise: $E args($#):\n$*"
 
    The $* part will give you each of the varbinds in an individual line in your trapd.log.  Then you can see what information is being provided and change the Event Log Message format so that it makes sense to your operators.
     I go through the log once a day looking for "no known format" or "FMT ERROR" messages and massage the trapd.conf to accomodate them.  We have found many traps where the original log entry had nothing to do with the real trap, remember that the definition of the trap stops at the last piece of the oid that NetView can interpret.  So pay attention to the first part of the trap where it says "received from enterprist AAAA" that AAAA is the name you will see in the list of enterprises when you bring up the trap definition window.
 
        Good luck,
        Bill
 
 
 
 
 -----Original Message-----
From: Barr, Scott [mailto:Scott_Barr AT csgsystems DOT com]
Sent: Thursday, October 03, 2002 9:27 AM
To: nv-l AT lists.tivoli DOT com
Subject: [nv-l] Cursed Cisco Trap Formats

NetView 7.1.1 on Solaris 2.8
 
Okay guys, I am looking for a way to skin a Cisco cat. The problem is due to the fact that we run a wide variety of protocols and routers, we often do not run the latest Cisco IOS versions. I recently had a situation where I observed this in trapd.log:
 
1033361376 3 Sun Sep 29 23:49:36 2002 <routernamehere> A Cisco_Link_Down trap received from enterprise cisco with 3 arguments: ifIndex=24; ifDescr=ATM1/0.8-aal5 layer; ifType=49; locIfReason=FMT ERROR: accessing element #4, only 3 available                                                               
Notice the format error. The reason this occurs is because under most circumstances the cisco IOS is delivering only 3 elements and the trap format in trapd.conf has 4 elements defined. So I opened TAC case on this with Cisco and they told me to use the following command on the routers:
 
snmp-server trap link ietf
 
Now, the trap comes in and looks like this:
 
1033478849 3 Tue Oct 01 08:27:29 2002 <routernamehere> A Cisco_Link_Down trap received from enterprise cisco with 5 arguments: ifIndex=26; ifDescr=2; ifType=2;  locIfReason=ATM1/0.9-aal5 layer                  
 
Now we get five arguments (still only 4 defined in trapd.conf) Okay, first problem is the format is still wrong since trapd.conf is not matching up with the IETF standard (which I have not been able to find yet). But thats no big deal, since I assumed I was writing some code to catch the variables and make intelligent decisions about what to do with it.
 
But wait! There is more! A lot of the routers send in link up/down traps in this format:
 
1033480388 3 Tue Oct 01 08:53:08 2002 <routernamehere>  A Cisco_Link_Down trap received from enterprise cisco with 4 arguments: ifIndex=1; ifDescr=Serial0/0;  ifType=22;  locIfReason=administratively down
 
So, to sum it up, I get link up/down traps with either 3, 4, or 5 arguments depending on what router is sending it in. They all have the same cisco enterprise ID so using trapd.conf to bypass the issue is not possible. I use rulesets (not command for automatic action in trapd.conf) to suppress interface outages of less than 5 minutes. I lose this functionality if I just pass the trap via command for automatic action. So what I need is a script that I can run using an action node, that can decipher whether there are 3,4, or 5 arguments and then parse them out. I am paging/emailing in my ruleset using action nodes, I would have to move them to the parsing script (no problem - we use nvpage and mailx)
 
Suggestions on scripts? How to code trapd.conf? Where is Cisco headquarters and what is composition of the materials used to build it? I *am* not a script coder person, so if you send me a perl script write it the way any idiot C programmer could read it and not one of your fancy-only-takes-1-line-of-completely-unreadable code.
 
- Signed: stuck between a rock and  a hard place with a boulder on my head.
Scott Barr
Network Systems Engineer
CSG Systems
Phone: 402-431-7939
Fax: 402-431-7413