ADSM-L

Re: Daily Processing Performance (very slow - any ideas?)

2002-11-07 11:14:53
Subject: Re: Daily Processing Performance (very slow - any ideas?)
From: David Longo <David.Longo AT HEALTH-FIRST DOT ORG>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 7 Nov 2002 11:13:56 -0500
I have a very similar environment.  I don't have the ED-6064 in loop -
yet.
Have only One Non-coll disk pool on an IBM FAStT 200 HA used
exclusively by TSM.
Have 6H1 with 2 procs and 2GB RAM, have 8 FC-AL drives in my 3584.

I get very fast performance on my LTO drives.  Tape to Tape I have
gotten
115G/hour - I use compression on drives, not on clients.  With
migrations
and backup stg from disk to tape I easily get 40GB/hour per tape
drive,
sometimes more - haven't really checked lately.  I never have more than
TWO
processes of these for a stg pool running at once - don't need to.

Several things you didn't mention:
A.  TSM DB.  Where is it stored and is it tuned across disks etc.  If
you do
a "q db f=d" what is the "Cache Hit Pct"?  Ideally is around 99%.

B.  The disk pools on ESS.  How well setup are they.  How many TSM
volumes do you have on each disk pool.

I am not an expert at ESS, but for instance if you just assigned one
big
500GB disk from ESS and have one or a very few TSM volumes that is
a problem.  Ideally break ESS down into smaller disks and have TSM
volumes
assigfned to each ESS disk (AIX hdisk).

Just a few quick thoughts.


David B. Longo
System Administrator
Health First, Inc.
3300 Fiske Blvd.
Rockledge, FL 32955-4305
PH      321.434.5536
Pager  321.634.8230
Fax:    321.434.5509
david.longo AT health-first DOT org


>>> KThach AT COVHLTH DOT COM 11/07/02 10:47AM >>>
I'm very dissapointed with the performance of our TSM environment, and
I was
curious what kinds of numbers some of you with similar environments
have
experienced.  I've worked extensively with IBM to try and tune things,
but
apparently we've got everything adjusted correctly.  We are in the
process
of cleaning up System Object stuff, so I'm wondering if I should
expect
things to improve dramatically once we trim all that fat.

I apologize for the lenght of the post, but I want to include as much
info
as possible.  I'm in dire need of a solution.

Our basic setup:

IBM 6H1 - 6 Processors / 4GB RAM
TSM 4.2.3 on AIX 4.3.3-09

LTO 3584 Tape Library with 10 drives
Fiber-arbitrated loop going through McData ES-1000 switches and McData
6064
Directors

500GB Non-Collocated Disk Pool on ESS
250GB Collocated Disk Pool on ESS
37GB TSM database

Approximately 200 Clients (mixture of AIX and WinNT/2K) running at
various
client versions

200GB total / night backed up on average.

Daily Processing is slow, slow, slow.

Here are the steps for our daily processing (its all scheduled, but I'm
just
showing you what runs when):

1) 7:00:00 - Daily processing starts
backup stg nocodisk copypool maxproc=4 wait=yes
backup stg colodisk copypool maxproc=4 wait=yes

2) Once that is finished the migrations start (I have the maxproc on
both
pools set to 5)
update stg nocodisk hi=0 lo=0
update stg colodisk hi=0 lo=0

3) Once Migration is finished
update stg nocodisk hi=90 lo=70
update stg colodisk hi=90 lo=70
backup stg nocotape copypool maxproc=3 wait=yes
backup stg colotape copypool maxproc=3 wait=yes

4) Once that is finished
expire inventory

5) Once that is finished
backup db devclass=ltotape type=full

6) Then
backup volhist
backup devconfig
prepare

So, the big disappointment is on steps 1 and 2.  Our disk to tape
performance averages about 20GB/hour per tape drive.  If I reduce the
number
of mount points, that number goes down even more.  Are LTO's really
this
slow?  IBM says these suckers will do 50-100GB/hour.  With 10 drives,
we
were told we could handle about 1-2 TB/day, and we're only dealing
with
200GB and the entire daily processing takes more than 6 Hours!!!

At one time we were using disk caching, and I was told that slowed down
disk
to tape performance, so we turned it off.  I saw a slight improvement,
but
nothing major.  The Noncollocated disk pool still has data in it that
has
not expired yet since I turned off caching.  Could that still be
slowing
things down if the pool isn't completely flushed?

What can I really expect to get as far as performance?  How long
should
daily processing really take for only 200GB worth of data?

Any help is greatly, greatly appreciated!

Thanks!
-Kevin

This E-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended
only
for the use of the Individual(s) named above.  If you are not the
intended
recipient of this E-mail, or the employee or agent responsible for
delivering it to the intended recipient, you are hereby notified that
any
dissemination or copying of this E-mail is strictly prohibited.  If you
have
received this E-mail in error, please immediately notify us at (865)
374-4900 or notify us by E-mail at hdesk AT covhlth DOT com.


"MMS <health-first.org>" made the following
 annotations on 11/07/2002 11:15:21 AM
------------------------------------------------------------------------------
This message is for the named person's use only.  It may contain confidential, 
proprietary, or legally privileged information.  No confidentiality or 
privilege is waived or lost by any mistransmission.  If you receive this 
message in error, please immediately delete it and all copies of it from your 
system, destroy any hard copies of it, and notify the sender.  You must not, 
directly or indirectly, use, disclose, distribute, print, or copy any part of 
this message if you are not the intended recipient.  Health First reserves the 
right to monitor all e-mail communications through its networks.  Any views or 
opinions expressed in this message are solely those of the individual sender, 
except (1) where the message states such views or opinions are on behalf of a 
particular entity;  and (2) the sender is authorized by the entity to give such 
views or opinions.

==============================================================================