Several have asked for an update...
Problem ha snot been fixed, well not really but maybe?. Because we do
not know what the root cause was. Veritas says it was a network problem.
They were seeing pack reordering issues in their trace. But we have had
our network team looking, tracing and scanning and no issues were found.
Our network guys said that even if their were issues of packet
reordering that would not stop the client from sending data to the media
servers and the drives. Since we were running in a "unsupported
configuration" according to Veritas we stripped down everything to match
what they do support (they would not help us unless we were in a support
configuration). We removed the second nic, ip address, turned off IPMP
and ether channel to the media and master servers. Our backup network
stabilized then but is running half the speed. Veritas recommended some
minor changes to our configuration in Netbackup which we will look at
next week. Our Active directory team said they were doing some DHCP and
routing changes on their servers in the vlan where 90% of the backup
failures were but that does not explain why we got this same problem
with the other 10%, which were at another location. So we are in the
process of adding back in what we removed last week to see if it
"breaks" but so far so good. Adding back in the last change today so
tonight's run will tell us if all is ok. If all is ok then we will make
the changes that Veritas recommended (active-passive for IPMP on our
master instead of active-active ) and fix a required host setting that
was wrong. They also recommended a smaller shared memory size but has
failed to tell us what they would like to see it set to, so our Unix
guys are scratching their heads wondering why. Unless there is a bug in
their software that does not like to see a huge amount of shared memory
out their. Quit frankly it has the company stumped. Because no one thing
would have caused a cascade failure like we were seeing. We are running
Netbackup 5.0 MP3 Solaris 8 (which we upgraded to MP4 per Veritas. We
wanted to go to MP5 but the tech recommended not because there are other
"issues" with MP5). However, during this process when everything was
under a microscope we did find some minor tuning issues that will help
with overall network performance. So I guess it was not a total loss.


 
Greg

 
Greg



>>> The information contained in this e-mail transmission is privileged and/or 
>>> confidential intended solely for the exclusive use of the individual 
>>> addressee. If you are not the intended addressee you are hereby notified 
>>> that any retention, disclosure or other use is strictly prohibited. If you 
>>> have received this notification in error, please immediately contact the 
>>> sender and delete the material.