ADSM-L

Re: Daily Processing Performance (very slow - any ideas?)

2002-11-07 11:22:54
Subject: Re: Daily Processing Performance (very slow - any ideas?)
From: "Mark D. Rodriguez" <mark AT MDRCONSULT DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 7 Nov 2002 10:19:12 -0600
Thach, Kevin wrote:

I'm very dissapointed with the performance of our TSM environment, and I was
curious what kinds of numbers some of you with similar environments have
experienced.  I've worked extensively with IBM to try and tune things, but
apparently we've got everything adjusted correctly.  We are in the process
of cleaning up System Object stuff, so I'm wondering if I should expect
things to improve dramatically once we trim all that fat.

I apologize for the lenght of the post, but I want to include as much info
as possible.  I'm in dire need of a solution.

Our basic setup:

IBM 6H1 - 6 Processors / 4GB RAM
TSM 4.2.3 on AIX 4.3.3-09

LTO 3584 Tape Library with 10 drives
Fiber-arbitrated loop going through McData ES-1000 switches and McData 6064
Directors

500GB Non-Collocated Disk Pool on ESS
250GB Collocated Disk Pool on ESS
37GB TSM database

Approximately 200 Clients (mixture of AIX and WinNT/2K) running at various
client versions

200GB total / night backed up on average.

Daily Processing is slow, slow, slow.

Here are the steps for our daily processing (its all scheduled, but I'm just
showing you what runs when):

1) 7:00:00 - Daily processing starts
backup stg nocodisk copypool maxproc=4 wait=yes
backup stg colodisk copypool maxproc=4 wait=yes

2) Once that is finished the migrations start (I have the maxproc on both
pools set to 5)
update stg nocodisk hi=0 lo=0
update stg colodisk hi=0 lo=0

3) Once Migration is finished
update stg nocodisk hi=90 lo=70
update stg colodisk hi=90 lo=70
backup stg nocotape copypool maxproc=3 wait=yes
backup stg colotape copypool maxproc=3 wait=yes

4) Once that is finished
expire inventory

5) Once that is finished
backup db devclass=ltotape type=full

6) Then
backup volhist
backup devconfig
prepare

So, the big disappointment is on steps 1 and 2.  Our disk to tape
performance averages about 20GB/hour per tape drive.  If I reduce the number
of mount points, that number goes down even more.  Are LTO's really this
slow?  IBM says these suckers will do 50-100GB/hour.  With 10 drives, we
were told we could handle about 1-2 TB/day, and we're only dealing with
200GB and the entire daily processing takes more than 6 Hours!!!

At one time we were using disk caching, and I was told that slowed down disk
to tape performance, so we turned it off.  I saw a slight improvement, but
nothing major.  The Noncollocated disk pool still has data in it that has
not expired yet since I turned off caching.  Could that still be slowing
things down if the pool isn't completely flushed?

What can I really expect to get as far as performance?  How long should
daily processing really take for only 200GB worth of data?

Any help is greatly, greatly appreciated!

Thanks!
-Kevin

This E-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended only
for the use of the Individual(s) named above.  If you are not the intended
recipient of this E-mail, or the employee or agent responsible for
delivering it to the intended recipient, you are hereby notified that any
dissemination or copying of this E-mail is strictly prohibited.  If you have
received this E-mail in error, please immediately notify us at (865)
374-4900 or notify us by E-mail at hdesk AT covhlth DOT com.


Kevin,

Although you have given us a lot of good info I think we might need some
more.  I suspect that the performance problems maybe related to file I/O
performance.  I would like to know how your disk storage pools, DB and
Log files are laid out.  Remember, file I/O tuning is like realestate
there are only 3 rules Location, Location, Location!  Well that might
not be completely true, but anyway.  Give us layout info like RAID
levels, array layouts and connection info, are you using vpath etc.
Also, send the output of vmtune command with no parms.  There has been
a lot of discussion on the list about tuning and your environment is
fairly common.  Paul Seay has done a lot of stuff with the ESS as well
so he will probably have some suggestions.  Once we have the info we
will probably be able to help.

--
Regards,
Mark D. Rodriguez
President MDR Consulting, Inc.

===============================================================================
MDR Consulting
The very best in Technical Training and Consulting.
IBM Advanced Business Partner
SAIR Linux and GNU Authorized Center for Education
IBM Certified Advanced Technical Expert, CATE
AIX Support and Performance Tuning, RS6000 SP, TSM/ADSM and Linux
Red Hat Certified Engineer, RHCE
===============================================================================