Okay, some of you have requested more info before trying to make a
diagnosis, so let me give you some more details.
The 500GB disk pool and the 250GB disk pool are on ESS--8GB vpaths. I have
144 of these vpaths allocated to our TSM server. ESS uses RAID5. Within
TSM, the volumes for the disk pools are 10GB volumes. So, I have 50 volumes
for the noncollocated disk pool and 25 volumes for the collocated pool.
My DB and LOG use 1GB volumes. Cache hit % on the DB is 98.5 today, but I
have seen it as low as 85%.
We have 4 fiber adapters in the TSM server. 2 are for disk and 2 are for
tape--so tape and disk are not operating on the same adapter.
Here is the output of vmtune from my TSM server:
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages
maxrandwrt
209505 838020 2 8 120 128 524288 0
-M -w -k -c -b -B -u -l
-d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket
defps
838841 16256 4064 1 93 2128 9 131072
1
-s -n -S -L -g -h
sync_release_ilock nokilluid v_pinshm lgpg_regions lgpg_size
strict_maxperm
0 0 0 0 0 0
number of valid memory pages = 1048551 maxperm=79.9% of real memory
maximum pinable=80.0% of real memory minperm=20.0% of real memory
number of file memory pages = 838012 numperm=79.9% of real memory
Thanks again. If you need more info still, just let me know.
Kevin
-----Original Message-----
From: Mark D. Rodriguez [mailto:mark AT MDRCONSULT DOT COM]
Sent: Thursday, November 07, 2002 11:19 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Daily Processing Performance (very slow - any ideas?)
Thach, Kevin wrote:
>I'm very dissapointed with the performance of our TSM environment, and I
was
>curious what kinds of numbers some of you with similar environments have
>experienced. I've worked extensively with IBM to try and tune things, but
>apparently we've got everything adjusted correctly. We are in the process
>of cleaning up System Object stuff, so I'm wondering if I should expect
>things to improve dramatically once we trim all that fat.
>
>I apologize for the lenght of the post, but I want to include as much info
>as possible. I'm in dire need of a solution.
>
>Our basic setup:
>
>IBM 6H1 - 6 Processors / 4GB RAM
>TSM 4.2.3 on AIX 4.3.3-09
>
>LTO 3584 Tape Library with 10 drives
>Fiber-arbitrated loop going through McData ES-1000 switches and McData 6064
>Directors
>
>500GB Non-Collocated Disk Pool on ESS
>250GB Collocated Disk Pool on ESS
>37GB TSM database
>
>Approximately 200 Clients (mixture of AIX and WinNT/2K) running at various
>client versions
>
>200GB total / night backed up on average.
>
>Daily Processing is slow, slow, slow.
>
>Here are the steps for our daily processing (its all scheduled, but I'm
just
>showing you what runs when):
>
>1) 7:00:00 - Daily processing starts
>backup stg nocodisk copypool maxproc=4 wait=yes
>backup stg colodisk copypool maxproc=4 wait=yes
>
>2) Once that is finished the migrations start (I have the maxproc on both
>pools set to 5)
>update stg nocodisk hi=0 lo=0
>update stg colodisk hi=0 lo=0
>
>3) Once Migration is finished
>update stg nocodisk hi=90 lo=70
>update stg colodisk hi=90 lo=70
>backup stg nocotape copypool maxproc=3 wait=yes
>backup stg colotape copypool maxproc=3 wait=yes
>
>4) Once that is finished
>expire inventory
>
>5) Once that is finished
>backup db devclass=ltotape type=full
>
>6) Then
>backup volhist
>backup devconfig
>prepare
>
>So, the big disappointment is on steps 1 and 2. Our disk to tape
>performance averages about 20GB/hour per tape drive. If I reduce the
number
>of mount points, that number goes down even more. Are LTO's really this
>slow? IBM says these suckers will do 50-100GB/hour. With 10 drives, we
>were told we could handle about 1-2 TB/day, and we're only dealing with
>200GB and the entire daily processing takes more than 6 Hours!!!
>
>At one time we were using disk caching, and I was told that slowed down
disk
>to tape performance, so we turned it off. I saw a slight improvement, but
>nothing major. The Noncollocated disk pool still has data in it that has
>not expired yet since I turned off caching. Could that still be slowing
>things down if the pool isn't completely flushed?
>
>What can I really expect to get as far as performance? How long should
>daily processing really take for only 200GB worth of data?
>
>Any help is greatly, greatly appreciated!
>
>Thanks!
>-Kevin
>
>This E-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended only
>for the use of the Individual(s) named above. If you are not the intended
>recipient of this E-mail, or the employee or agent responsible for
>delivering it to the intended recipient, you are hereby notified that any
>dissemination or copying of this E-mail is strictly prohibited. If you
have
>received this E-mail in error, please immediately notify us at (865)
>374-4900 or notify us by E-mail at hdesk AT covhlth DOT com.
>
>
Kevin,
Although you have given us a lot of good info I think we might need some
more. I suspect that the performance problems maybe related to file I/O
performance. I would like to know how your disk storage pools, DB and
Log files are laid out. Remember, file I/O tuning is like realestate
there are only 3 rules Location, Location, Location! Well that might
not be completely true, but anyway. Give us layout info like RAID
levels, array layouts and connection info, are you using vpath etc.
Also, send the output of vmtune command with no parms. There has been
a lot of discussion on the list about tuning and your environment is
fairly common. Paul Seay has done a lot of stuff with the ESS as well
so he will probably have some suggestions. Once we have the info we
will probably be able to help.
--
Regards,
Mark D. Rodriguez
President MDR Consulting, Inc.
============================================================================
===
MDR Consulting
The very best in Technical Training and Consulting.
IBM Advanced Business Partner
SAIR Linux and GNU Authorized Center for Education
IBM Certified Advanced Technical Expert, CATE
AIX Support and Performance Tuning, RS6000 SP, TSM/ADSM and Linux
Red Hat Certified Engineer, RHCE
============================================================================
===
This E-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended only
for the use of the Individual(s) named above. If you are not the intended
recipient of this E-mail, or the employee or agent responsible for
delivering it to the intended recipient, you are hereby notified that any
dissemination or copying of this E-mail is strictly prohibited. If you have
received this E-mail in error, please immediately notify us at (865)
374-4900 or notify us by E-mail at hdesk AT covhlth DOT com.
|
