Information
for my NBU colleagues.
We
came across something in our environment about a month or more ago where we had
a very large number of jobs fail with various network related issues. And
it was random clients or random media servers. We went thru and upgraded
our backup infrastructure, built new media server etc etc. All with no
good results. Our issue started around the end of November and anything
you can think of we have tried. We eventually had a Symantec Consultant
come onsite for two weeks to find our issue, in conjunction with Symantec
support we FINALLY believe we found our culprit…and here it is…McAfee
McShield 8.5 patch 4. Broke us bad…real bad.
And
simply disabling the services WILL NOT fix the issue, we had to manually remove
McAfee from all of our Media and the Master server.
Below
is the tech note. The gentleman who found it, reads these forums.
So publicly I like to thank-you, really appreciate your efforts.
McAfee is supposed to have fixes for this
issue.
3RD PARTY: NetBackup Services randomly
shut down.
Details:
Vendor/Product:
McAfee McShield 8.5 patch 3 or patch 4.
***NOTE*** If confirmation of this issue has occurred, it is Highly Recommended for the customer to
open a ticket directly with McAfee Support for the latest update on how to
handle this problem.
Detail/Symptom(s):
* NetBackup services randomly shut down including:
NetBackup Resource Broker Service
NetBackup Notification Service
NetBackup Policy Execution Manager Service
NetBackup Service Layer Service
* Active Jobs finish but tapes are not moved from drives back to slots
* Active Jobs which need to span media sit at "Waiting for next media:
Any"
* Queued Jobs do not go active
The above symptoms can happen once or twice per day and can occur on the
smallest or largest installations.
We have been able to associate these symptoms with the existence of McAfee
McShield 8.5 patch 3 or patch 4 running concurrently with NetBackup 6.0.
The NetBackup Services are shutting themselves down because the inter-process
sockets are being disconnected. The processes attempt to reconnect, but
are unable to do so and the services shut down.
Log Files:
12/20/07 01:53:56.227 137 PID:7184
TID:7556 [TAO] ACE_Select_Reactor_Notify::notify [handle=0x1f8]: write to
notification pipe handle failed: An existing connection was forcibly closed by
the remote host. (10054)
12/20/07 01:53:56.227 137 PID:7184
TID:7556 [TAO] sleep_hook failed: An existing connection was forcibly closed by
the remote host.
12/20/07 01:53:56.242 137 PID:7184
TID:920 [TAO] handle_notify_pipe_close - taking action REOPEN
12/20/07 01:54:17.336 137 PID:7184
TID:920 [TAO] handle_notify_pipe_close: failed to re-open notification pipe: A
connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because connected host
has failed to respond.
Workaround:
There several workarounds which can be implemented:
1. Uninstall McAfee McShield 8.5 and reboot. (simply stopping McShield services
is not sufficient)
2. Roll-back to McAfee McShield 8.0.
3. McAfee recommends:
A. Opening a ticket with McAfee Support
on this issue.
B. Renaming this driver file MFETDIK.sys
and rebooting. The lost functionality is, Port Blocking access protection
rules, and identification of Source IP address for a remote attacker.
4. Implement these excludes (mixed
success with this potential workaround)
McAfee McShield can exclude by directory structure or by process - its up to
the customer:
Good luck all and figured I’d pass
this along in the event anyone out there is running NBU 6.0 and MacAfee…
Dan Cruice.
