ADSM-L

Re: [ADSM-L] Please tell me about your LTO3 / LTO4 performance

2008-07-01 11:53:44
Subject: Re: [ADSM-L] Please tell me about your LTO3 / LTO4 performance
From: "Thach, Kevin G" <KThach AT COVHLTH DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 1 Jul 2008 11:51:51 -0400
I am set up very similar to you.  My TSM LPAR HBAS connect to a director
class switch which has an ISL to each of the edge switches that the tape
drives themselves connect to (odd drives on one and even on the other
like yourself.)

Therefore, I have 64 rmt devices at the AIX level for my LTO3 drives, as
each tape HBA sees each of the 14 drives.  I am not using the alternate
pathing.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Kauffman, Tom
Sent: Tuesday, July 01, 2008 11:42 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Please tell me about your LTO3 / LTO4 performance

How are your tape drives attached to your TSM HBAs? Presumably by SAN
switch, so how do you have the drives zoned? Ideally, every drive should
be visible on every fiber and alternate path support should be enabled
(chdev -l rmtx -a alt_pathing=yes) (do NOT do for the SMC if you do not
have path failover; may not work for LTO3 if you do not have path
failover).

I have 10 LTO4 and 6 LTO2 drives, and 10 fibers to tape from my TSM
LPAR; two SAN switches, with the even-numbered drives in one and the
odd-numbered drives in the other. The result is 80 rmt (tape) devices
for the LPAR.

I know I'm network limited - so I only get a maximum of 110 MB/sec per
drive/network interface in my nightly SAP backups. (dedicated Gb
networks, one per concurrent backup session - Gigabit Ethernet NICs are
cheap!) My off-site copy processes run at LTO2 drive speed (the 'twos
are only used for offsite tapes).

This is for 4 concurrent sessions over two network interfaces:
BKI1215I: Average transmission rate was 762.364 GB/h (216.850 MB/sec).
BKI1227I: Average compression factor was 1.000.
BKI0020I: End of program at: Mon Jun 30 20:55:08 EDT 2008 .
BKI0021I: Elapsed time: 01 h 52 min 00 sec .
BKI0024I: Return code is: 0.

So I averaged 108 MB/sec over the NIC, and 54 MB/sec to the drive.

Tom Kauffman
NIBCO, Inc

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Thach, Kevin G
Sent: Tuesday, July 01, 2008 10:41 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Please tell me about your LTO3 / LTO4 performance

Hi all-



For quite some time now, I have been trying to track down an elusive
bottleneck in my TSM environment relating to disk-to-tape performance.
This is a long read, but I would be very greatful for any suggestions.
Hopefully some of you folks much smarter than me out there will be able
to point me in the right direction.



If any other LTO3 or LTO4 users out there could give me some examples of
their real-world performance along with a little detail on their config,
that would be most helpful as well!



My current environment consists of:



*         TSM server = p570 LPAR w/4  1.9GHz processors and 8GB RAM, (6)
2Gb HBAS (2 for disk and 4 for tape traffic), and a 10Gb Ethernet
adapter.

*         TSM 5.4.1.2 on AIX 5.3 TL6

*         3584 w/14 LTO3 drives at primary site

*         3584 w/12 LTO1 drives at DR/hotsite (copypool volumes are
written directly to this library via SAN routing)

*         DB (80GB -- 4GB DBVOL size) residing on IBM DS8300 behind IBM
SVC

*         Log (11GB - single LOGVOL) residing on IBM DS8300 behind IBM
SVC

*         Primary Storage pool in question (2.5TB -- 20GB volume size),
DISK device class, residing on IBMDS8300 behind IBM SVC



I currently back up about 4.5TB / night, of which ~2TB is written
directly to my primary LTO3 tape pool with a simultaneous write to my
copypool across town.  So, each morning I'm left with about 2.5TB of
data to copy and migrate from my disk pool(s) to copypool and onsite
tape respectively.



My backup stg performance to LTO1 tape (copypool) is about what I would
expect.  I run 5 threads for this process (5 mount points used), and I
consistently average 20-25MB/sec/drive.  Fair enough.  I don't know of
anyone getting a whole lot more than that out of an LTO1 drive.



After that is complete, I then migrate that data to my LTO3 tape here
onsite.  That performance is pretty lousy compared to what I would
expect to get out of LTO3.  I run  6 migration threads (6 mount points
used), and I average around 25MB/sec/drive going to LT03 as well.



All SAN links between the TSM server and the LT03 drives are a minimum
of 2Gb, so that is my lowest common denominator.  I've tried using less
threads to see if perhaps I was saturating an HBA rather than the drive.
Same speed.  I've tried separating my DB and STG pools on different
storage subsystems.  Same speed.  I've opened PMR's with IBM support,
and they have poured over all of my TSM server settings / config and
found nothing to go on.  We've had IBM ATS teams evaluate the situation,
and they've never been able to pinpoint a problem.



I've tried various tools--tapewrite, nmon, filemon, etc. and I've not
found a smoking gun.



At this point, my gut is that SVC is the bottleneck, but for those of
you familiar with SVC, you know that trying to obtain meaningful
performance statistics on the SVC cluster itself is frustrating.



I know there are folks out there getting much better performance out of
LTO3 drives, so please tell me how you're doing it!



Suggestions?  Questions?



Thank you!

-Kevin








-----------------------------------------
This E-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION
intended only for the use of the Individual(s) named above.  If you
are not the intended recipient of this E-mail, or the employee or
agent responsible for delivering it to the intended recipient, you
are hereby notified that any dissemination or copying of this
E-mail is strictly prohibited.  If you have received this E-mail in
error, please immediately notify us at (865)374-4900 or notify us
by E-mail at hdesk AT covhlth DOT com
CONFIDENTIALITY NOTICE:  This email and any attachments are for the 
exclusive and confidential use of the intended recipient.  If you are
not
the intended recipient, please do not read, distribute or take action in

reliance upon this message. If you have received this in error, please 
notify us immediately by return email and promptly delete this message 
and its attachments from your computer system. We do not waive  
attorney-client or work product privilege by the transmission of this
message.