RE: [nv-l] SOLVED - Traps Limitation??
2006-04-18 07:47:20
Well stated. TS gave me the number 'sustained 6-8
traps/sec' that NV could handle without tipping over. Your overall performance
depends on what processing you have going on after trapd. The secret to managing
traps is process and (if possible) characterizing traps in a 'safe' environment.
Set up a dev box and have folks point traps there first. Some traps are
'well behaved', meaning every minute to 5 minutes; some HP SIM traps send every
30 seconds, F5's can send once/second, and a misconfigured Cisco can send traps
at the rate of several per second. We've learned the hard way. Ain't it
fun??
Well, there is trap volume and then
there is trap volume. The product is designed to withstand quite an onslaught
but you are expected to deal with such an onslaught as a network problem,
because it IS a network problem. Ten traps per second, for instance, will
get handled. After a few minutes of this, thing start getting behind and then if
they stop, everything catches up. If it goes on and on, or escalates, things
cannot catch up.
If you look at
trapd.log or the events display, it is hard to miss the fact that an event storm
is going on. If you check the queue, you will probably see it backing up. Here's
a health check: netstat -a
| grep \.162 There will also be messages
in trapd.log about applications connecting and disconnecting from trapd. So that
is something else to look for.
Then,
netmon also is trying to send status events to trapd. It will naturally have
trouble operating normally. When I see netmon not responding, my first suspicion
is always bad name resolution, but in environments where traps are configured on
(especially authentication failure traps), I've learned recently to check that
first.
Cordially,
Leslie
A. Clark IT Services Specialist, Network Mgmt Information Technology
Services Americas IBM Global Services (248) 552-4968 Voicemail, Fax,
Pager
usman.taokeer AT s-iii DOT com Sent by: owner-nv-l AT lists.us.ibm DOT com
04/15/2006 01:44 AM
|
To
| nv-l AT lists.us.ibm DOT com
|
cc
| nv-l AT lists.us.ibm DOT com,
owner-nv-l AT lists.us.ibm DOT com
|
Subject
| Re: [nv-l] SOLVED
- Traps Limitation?? |
|
Hi,
I figured it out... the netview was flooded with
traps, and it was preventing Netview (netmon) to work properl, now since we have
limited the traps from devices everything is back to normal. There seems like a
limitation in netview on number of traps that it can process, IBM should
document it if there is one!
Regards, Usman Taokeer Si3.
James Shanks
<jshanks AT us.ibm DOT com> Sent by:
owner-nv-l AT lists.us.ibm DOT com
15-03-06 11:35 PM
Please respond
to nv-l AT lists.us.ibm DOT com |
|
To
| nv-l AT lists.us.ibm DOT com
|
cc
|
|
Subject
| Re: [nv-l] Traps
Limitation?? |
|
Usman, I doubt that you can ever
solve this problem by merely guessing at probable causes. Stop guessing,
turn on the full netmon trace (netmon -M -1), and call Support. They will
be able to tell you what netmon is doing, or else pass the trace to someone
else who can.
James Shanks Level 3 Support for Tivoli NetView
for UNIX and Windows Tivoli Software / IBM Software Group
usman.taokeer@s-i
ii.com
Sent by:
To
owner-nv-l@lists. nv-l AT lists.us.ibm DOT com
us.ibm.com
cc
nv-l AT lists.us.ibm DOT com,
owner-nv-l AT lists.us.ibm DOT com
03/15/2006 09:47
Subject AM
Re: [nv-l] Traps Limitation??
Please respond to
nv-l AT lists.us DOT ibm
.com
Gareth,
Ok! here is the
scenario:
NetView 7.1.4 FP04 Windows 2003 SP1 Hosted on a Dell Dual
XEON Processor with 2GB RAM!
There are around 1200 Nodes discovered in
the network, and we have around 400 Routers which are sending different traps
to the Netview server. The problem is whenever we try to "Demand Poll, Quick
Test etc" any device it just keeps saying "Waiting for netmon to respond" !!!
Any clues what's causing this? There is plenty of Memory available and the
CPU utilization is also between 4-10% only!
Regards,
Usman
Taokeer Si3.
Gareth Holl
<gholl AT us.ibm DOT com>
Sent by: owner-nv-l AT lists.us.ibm DOT com
To
nv-l AT lists.us.ibm DOT com 15-03-06 08:21 AM
cc
Subject
Please respond to
Re: [nv-l] Traps
nv-l AT lists.us.ibm DOT com
Limitation??
There is always going to be a limit of some
sort, whether with the hardware, OS, or trapd's ability itself. This is most
likely dependent on the resources (CPU speed, number of CPUs, and available
memory per process) available on the system hosting NetView.
trapd may
end up consuming most, if not all cycles of a single CPU during heavy trap
reception. So a multi-CPU box would be essential so that other processes
could continue to run. High CPU utilization caused by trap floods and even
the subsequent processing of the traps by other daemons such as nvcorrd could
well affect netmon's ability to keep up if it cannot get the CPU cycles it
needs.
trapd will try caching/queuing all events received (with the goal
to eventually process every single one of them), hence the need for a
large amount of memory and an adequate queue size. The cached events will
be processed when there is a break in trap reception.....this could be
some time after the trap was originally generated. So while trapd is
still receiving traps, it is possible for NetView's internal events
(including those from netmon) to be caught up in this process and thus
stayed queued/unprocessed for some time. This probably isn't a direct affect
on netmon but instead more of a perceived affect as status events are
not processed in a timely fashion and thus nodes don't change color on the
map in a timely fashion
That's all I
have.
Gareth
usman.taokeer AT s-iii DOT com
Sent by:
owner-nv-l AT lists.us.ibm DOT com
To 03/14/2006 09:57 PM
nv-l AT lists.us DOT ibm.c
om
cc Please respond to
nv-l
Subject
[nv-l] Traps
Limitation??
Hi List,
Just wanted to know
are there any limitations On Netview on the number of traps (coming from
different nodes) it can handle? If there is any would it effect the behaviour
of netmon?
Regards,
Usman Si3.
This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message.
Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited. [v.E.1]
|
|
|