ADSM-L

Re: Expiration performance

2004-09-13 16:54:20
Subject: Re: Expiration performance
From: Dave Canan <ddcanan AT ATTGLOBAL DOT NET>
Date: Mon, 13 Sep 2004 13:54:02 -0700
        We have seen this type of behavior on performance PMRs. Often, a
DB will backup within the select command backup guidelines, but expiration
objects examined per hour will be slow. There also is a correlation between
the amount of fragmentation and the expiration time. In some cases, we have
seen that a DB that is fragmented by 50% goes through expiration up to 5
times faster after being de-fragmented.
        And yet, I am hesitant to recommend that everyone rush right out
and do an unload/reload of the DB. Why? First, because of the time involved
(and the downtime!). Second, it appears that it is always a short-term
benefit. Databases, by their very nature, are fragmented. Third, I have
always wondered if this can actually hurt you performance wise. If a
database is de-fragmented, it will probably end up on fewer TSM database
volumes, and this could actually make performance worse in the short term
until it gets fragmented.
        I am certainly open to discussion on this one. In general, the ATS
performance team has found that if expire inventory runs well, than most
TSM "things" (DB backup, reclamation, backups in general, etc) run well.
At 08:32 AM 9/13/2004 -0600, you wrote:
        Hmm...

        So from the responses posted so far, it looks like those folks
with smaller databases (there are a couple posts from folks with ~3GB of
utilized DB space) are the only ones who seem to be getting the mythical
>5M objects/hour on expirations.

        Those of us with larger DBs seem to be getting only in the <1M
objects/hour range. Now that I see the other posts, I guess I don't feel
like such an under-achiever anymore ;-)


__________
Here's the stats from one of my servers:
IBM P-series M80
        8CPUs, 6GB RAM.
AIX 5.2 ML1
TSM Server version 5.2.1.3
TSM DB - ~35GB on SSA D40 drawers w/fastwrite cache turned on. JBOD
configuration.

ACTIVITY                     Date                    Pages backed Up/Hr
------------------     ----------     ---------------------------------
FULL_DBBACKUP          2004-09-02                              17272800
FULL_DBBACKUP          2004-09-03                              18475200
FULL_DBBACKUP          2004-09-04                              18666000
FULL_DBBACKUP          2004-09-05                              18961200
FULL_DBBACKUP          2004-09-06                              18442800
FULL_DBBACKUP          2004-09-07                              18640800
FULL_DBBACKUP          2004-09-08                              18608400
FULL_DBBACKUP          2004-09-09                              18597600
FULL_DBBACKUP          2004-09-10                              18885600
-----------
ACTIVITY                     Date                Objects Examined Up/Hr
------------------     ----------     ---------------------------------
EXPIRATION             2004-09-02                                964800
EXPIRATION             2004-09-03                                270000
EXPIRATION             2004-09-04                                306000
EXPIRATION             2004-09-05                                291600
EXPIRATION             2004-09-06                                248400
EXPIRATION             2004-09-07                                302400
EXPIRATION             2004-09-08                                288000
EXPIRATION             2004-09-09                                594000
EXPIRATION             2004-09-10                                306000

        FWIW, it seems like over the last few years, every time I
upgrade the TSM server to the next version (TSM 4.1 -> 5.1 -> 5.2) I've
noticed a slowdown in the speed of the expiration process.

Ben


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Mueller, Ken
Sent: Monday, September 13, 2004 7:48 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Expiration performance


Our TSM setup is not nearly as large as most of yours, but for what it's
worth, here are our values for the past month:

FULL_DBBACKUP          2004-08-14                               6537600
FULL_DBBACKUP          2004-08-15                               6051600
FULL_DBBACKUP          2004-08-16                               6303600
FULL_DBBACKUP          2004-08-17                               5511600
FULL_DBBACKUP          2004-08-17                              12970800
FULL_DBBACKUP          2004-08-18                               6739200
FULL_DBBACKUP          2004-08-19                               7081200
FULL_DBBACKUP          2004-08-20                               7657200
FULL_DBBACKUP          2004-08-21                               7927200
FULL_DBBACKUP          2004-08-22                               7945200
FULL_DBBACKUP          2004-08-23                               5875200
FULL_DBBACKUP          2004-08-24                               6343200
FULL_DBBACKUP          2004-08-25                               7992000
FULL_DBBACKUP          2004-08-26                               5904000
FULL_DBBACKUP          2004-08-27                               8514000
FULL_DBBACKUP          2004-08-28                               7419600
FULL_DBBACKUP          2004-08-29                               5922000
FULL_DBBACKUP          2004-08-30                               6379200
FULL_DBBACKUP          2004-08-31                               7578000
FULL_DBBACKUP          2004-09-01                               8089200
FULL_DBBACKUP          2004-09-02                               7473600
FULL_DBBACKUP          2004-09-03                               5533200
FULL_DBBACKUP          2004-09-04                               5644800
FULL_DBBACKUP          2004-09-05                               8298000
FULL_DBBACKUP          2004-09-06                               6627600
FULL_DBBACKUP          2004-09-07                               6937200
FULL_DBBACKUP          2004-09-08                               6778800
FULL_DBBACKUP          2004-09-09                               8236800
FULL_DBBACKUP          2004-09-10                               8859600
FULL_DBBACKUP          2004-09-11                               8974800
FULL_DBBACKUP          2004-09-12                               5958000
FULL_DBBACKUP          2004-09-13                               8762400


EXPIRATION             2004-08-14                               9637200
EXPIRATION             2004-08-15                              20984400
EXPIRATION             2004-08-16                               8056800
EXPIRATION             2004-08-17                              19476000
EXPIRATION             2004-08-18                               7131600
EXPIRATION             2004-08-19                              18943200
EXPIRATION             2004-08-20                              14011200
EXPIRATION             2004-08-21                              13417200
EXPIRATION             2004-08-22                              10011600
EXPIRATION             2004-08-23                              20188800
EXPIRATION             2004-08-24                              19026000
EXPIRATION             2004-08-25                              17978400
EXPIRATION             2004-08-26                              13017600
EXPIRATION             2004-08-27                              12211200
EXPIRATION             2004-08-28                               8910000
EXPIRATION             2004-08-29                              15472800
EXPIRATION             2004-08-30                              19191600
EXPIRATION             2004-08-31                              17380800
EXPIRATION             2004-09-01                              11451600
EXPIRATION             2004-09-02                              11592000
EXPIRATION             2004-09-03                              11980800
EXPIRATION             2004-09-04                              12006000
EXPIRATION             2004-09-05                              12412800
EXPIRATION             2004-09-06                              13010400
EXPIRATION             2004-09-07                              18284400
EXPIRATION             2004-09-08                              14076000
EXPIRATION             2004-09-09                              12236400
EXPIRATION             2004-09-10                              12283200
EXPIRATION             2004-09-11                              10602000
EXPIRATION             2004-09-12                              12808800

RedHat Enterprise Linux 3.2.3-34 SMP
TSM 5.2.2.4
IBM x335 Dual 2GHz Intel Xeon 2.5GB RAM
DB Size 3G (65% util - 99.7% Hit ratio) on mirrored 36G 15k disk volume
L3583-36 w/2 LTO-1 drives (dedicated FC HBA through SAN)

I think cache hit ratio would have a pretty strong correlation to
expiration performance... (having a database that's relatively tiny
doesn't hurt either
;-)

Ken Mueller
Magna Carta Companies


-----Original Message-----
From: Christo Heuer [mailto:christoh AT ABSA.CO DOT ZA]
Sent: Monday, September 13, 2004 3:13 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Expiration performance


Hi Ben,

I agree with you on these points - ran Dave's script on our main TSM
server
- and the Database backup performance is exceptionally good - the
slowest I'm seeing is  31000000:
FULL_DBBACKUP          2004-09-03                              62625600
FULL_DBBACKUP          2004-09-04                              56156400
FULL_DBBACKUP          2004-09-05                              59911200
FULL_DBBACKUP          2004-09-06                              51152400
FULL_DBBACKUP          2004-09-07                              57801600
FULL_DBBACKUP          2004-09-08                              60886800
FULL_DBBACKUP          2004-09-09                              52707600
FULL_DBBACKUP          2004-09-10                              59018400
FULL_DBBACKUP          2004-09-11                              57938400
FULL_DBBACKUP          2004-09-12                              58968000

The average seems to be in the region of 50000000 +, so from that point
of view it seems excellent - but the expiration sucks according to the
script.

ACTIVITY                     Date                Objects Examined Up/Hr
------------------     ----------     ---------------------------------
EXPIRATION             2004-08-14                                856800
EXPIRATION             2004-08-14                                867600
EXPIRATION             2004-08-15                                864000
EXPIRATION             2004-08-17                                770400
EXPIRATION             2004-08-18                                669600
EXPIRATION             2004-08-19                                846000
EXPIRATION             2004-08-20                                734400

So, yes, it seems that Ymmv, but is there anyone out there getting good
performance from expiration? If there are people out there, don't you
want to share your environment with us - might be able to collectively
cast some light on the lower expiration figures.

For the record, my environment is as follows:
AIX 5.2
TSM 5.2.2.5
DB size 105Gig - 93% utilized - 13 Lun's (One over the recommendation).
Disk subsys - FastT900 File system - RLV Tape technology: 3584 with
Lto-2 drives. P650 (4 Processors and 4Gig Ram)

Regards
ChristoH

===================================================================
        Yes, interesting stats. On all my TSM servers, they get above 5M
pages for the DB backup, but none of them are above 3.8M objects on the
expire inventory. Some in the 2M, others only in the .5 M range.

        These random thoughts pointed at the group, not necessarily Joe.

        Since my db backups (sequential reads) go well, but the expire
inventory (random reads and writes) are slow, might that point to DB
fragmentation? Improper tuning of TSM buffers? Overcommittal of the
fast-write disk cache? Bad karma?

        One would think that the more files deleted during the expire
inventory, the longer it will take for the expire inventory to progress?
No? I can run 2 expire inventories in a row and the second one goes much
much quicker because there are very few changes to the DB to be made. It
seems like the performance on the "number of objects examined" is really
one of those "your mileage may vary" kind of stats.

        Perhaps I'm not getting the performance I should out of the
expiration...

Ben



______________________________________________________
Important Notice:

Important restrictions, qualifications and
disclaimers ("the Disclaimer") apply to this email.

To read this click on the following address:

http://www.absa.co.za/ABSA/EMail_Disclaimer

The Disclaimer forms part of the content of this
email in terms of section 11 of the Electronic
Communications and Transactions Act, 25 of 2002.

If you are unable to access the Disclaimer,
send a blank e-mail to disclaimer AT absa.co DOT za and we
will send you a copy of the Disclaimer.

Dave Canan
TSM Performance
IBM Advanced Technical Support
ddcanan AT us.ibm DOT com
<Prev in Thread] Current Thread [Next in Thread>