--== Warning: Wall of text incoming ==--
I have a NetBackup environment consisting of:
-= Local Site =-
1 Red Hat Linux AS 4 Master running NBU 6.0 MP7
2 Red Hat Linux AS 4 Media Servers running NBU 6.0 MP7
3 Tru64 V5.1B (Rev. 2650) SAN Media running NBU 6.0 MP7 (Mix between
O/S patch kit 6 and 7)
1 Spectra Logic T380 with 12 IBM LTO4 drives running latest BlueScale
patches and drive firmware.
1 NetApp 1400 VTL running latest firmware.
-= DR Site =-
1 Red Hat Linux AS 4 Master running NBU 6.0 MP7
1 Tru64 V5.1B (Rev. 2650) SAN Media running NBU 6.0 MP7
1 Spectra Logic T200 with 12 IBM LTO4 drives running latest BlueScale
patches and drive firmware.
Last July we replaced our ADIC i2000 library (LTO2 drives) with a
Spectra Logic T380. Once we got the library deployed we noticed that
our Linux systems are able to write to the library at LTO4 speeds, and
the regular network clients even get decent throughput over a 1gb
ethernet network. But the 3 Tru64 SAN media servers absolutely crawl.
In spite of the fact that I have the SAN media server license
installed, I can only get about 10 - 20 MB/s on the policies using the
Tru64 storage units.
Our main production database sits on a GS1280 (30 CPUS ,114 GB
memory), and we have a ES80 attached to another Spectra Logic library
at our DR site. Every Sunday morning, I backup an RMAN backup to tape,
mail the tapes to my DR site, and restore the RMAN files using a
Spectra Logic T200 attached to the ES80, which also has the SAN Media
Server software installed. My GS1280 system takes 15-20 hours to
backup, but my DR system can restore the same files in 6-7 hours
running at 80 - 110 MB/s. I'm completely baffled how the smaller
system gets such awesome throughput while my production box plods
along at sub-ethernet speeds.
I've spent the past several months researching performance and tuning
suggestions and I've applied settings 1 at a time when I can get an
outage.
To speed up testing, we have another GS1280 with 1/2 the CPU and
memory as the production system, and it only runs test databases, so
it's easier to ask to reboot it if I want to try tuning a particular
kernel parm or what not. I installed the SAN media server software on
this second 1280 and I've been trying to tune it to NetBackup for the
last couple of months.
Within NetBackup, I've tuned the Size and Number of data buffers, and
it has no visible effect.
I've used the hwmgr command to look at the driver and firmware level
of just about every piece of equipment on both systems, up to and
including the individual busses. The GS1280 has everything the ES80
does, it just has more of it.
I've verified HBA drivers on all boxes and all appear to be at the
latest firmware.
I've asked my SAN guys to double check the zoning, LUN masking,
configuration and firmware levels on the SAN switches here and at my
DR site to see if there's anything that might be preventing Tru64 from
writing to either of my libraries at SAN speeds. They have checked and
everything seems to be in order on both SAN environments. Furthermore,
I've asked them to look at port utilization on the SAN switches during
test backups from the 1280 and they tell me that the HBAs are hardly
being utilized.
We recently deployed a NetApp VTL, and I was curious if perhaps the
VTL got better performance (which would indicate some type of
incompatibility between Tru64 and Spectra Logic). There isn't one that
I can find. If I setup a test policy to write to the VTL from my test
GS1280 and let it write to all 80 virtual drives, no one stream
exceeds about 10 - 20 MB/s.
Next, I looked at the fragmentation level of the AdvFS domains on both
systems. While some are heavily fragmented, the I/O performance on
both systems is 100% for every file domain I've checked.
The fact that all my clients (Windows, Linux and the handful of
Solaris 10) work well with both libraries makes me think that this is
something in Tru64. If that's true, then I'm trying to figure out what
is set correctly on my DR ES 80 that's jacked up on my local 1280.
According to section 1.9 of the Tru64 tuning manual
(http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51B_HTML/ARH9GCTE/TITLE.HTM)
the 5 most commonly tuned kernel subsystems are: vm, ipc, proc, inet,
and socket. Furthermore,
http://seer.entsupport.symantec.com/docs/235845.htm is a technote
advising Tru64 kernel changes for NetBackup. I have examined the
values across all my systems. In most cases, the values on both
systems meet or exceed tuning suggestions I have found in manufacturer
documentation. The two or three values I have tuned so far have had no
effect.
http://www.scribd.com/doc/19213788/Net-Backup-6
I found this TechNote which recommends setting the sem_mni and sem_msl
values to 1,000. sem_msl is currently set to 500 on my local 1280, and
I think this is perhaps the only kernel parm I have yet to tune. I'm
going to ask for an outage this week to increase this setting to
1,000. If that doesn't work, then I believe I will be officially
stumped.
I've also watched the EVM channels and the binary error log and
haven't seen anything alarming. The tape drives aren't throwing errors
and appear to be working fine.
This is leading me to believe that there is something not tuned
correctly between the Tru64 O/S and the NetBackup client. If it's not
in the kernel then I simply don't know where else to look.
I'll be posting this to the NetBackup forums on Symantec.com, the ITRC
forums on HP.com and the NetBackup mailing list.
Can anyone think of any stone I've left unturned? Thanks.
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
