ADSM-L

Re: Expiration after TSM upgrade from 5.1.6.4 to 5.2.2.1

2004-03-26 09:17:29
Subject: Re: Expiration after TSM upgrade from 5.1.6.4 to 5.2.2.1
From: Luke Dahl <ldahl AT JPL.NASA DOT GOV>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Fri, 26 Mar 2004 06:16:40 -0800
Hi Eric,
    From our activity log:
03/20/04 06:00:07     ANR0811I Inventory client file expiration started as proc
                       cess 26. (SESSION: 5656, PROCESS: 26)

03/25/04 17:24:44     ANR0812I Inventory file expiration process 26 completed:
                        examined 23979559 objects, deleting 21479290 backup obj
                       jects, 0 archive objects, 0 DB backup volumes, and 0 rec
                       covery plan files. 1 errors were encountered. (SESSION:
                        5656, PROCESS: 26)
03/25/04 17:24:44     ANR0987I Process 26 for EXPIRE INVENTORY running in the B
                       BACKGROUND processed 21479290 items with a completion st
                       tate of FAILURE at 17:24:44. (SESSION: 5656, PROCESS: 26)

We've seen much better *overall* performance after moving to raw volumes, but 
nowhere near what we saw on two very old AIX boxes.  We're running a Sun 420R 
with three cpu's.   Load averages nightly peak around 20.  We're moving to two 
Sun V480's with two cpu's to start.  We haven't run expiration yet again but 
I'm hoping it doesn't take nearly as long.  I'm still confused as to why after 
the upgrade we're seeing this.  If it's significantly different the next time 
around I'll report to the list with my findings.

Good luck!

Luke

"Loon, E.J. van - SPLXM" wrote:

> Hi Luke!
> Just because I'm currently trying to boost expiration processing: I see your
> database is rather large, how long did it take you to expire those 23194393
> objects?
> I really tried almost everything, from moving to 15k speed disks, multiple
> database volumes, moving to RAW's, the best I have seen in my environment is
> 60 objects per second.
> I posted a question some time ago (2002) where I requested for other
> people's performances, Tom Kauffman was the winner at that time with a
> dazzling 3112 objects per second!
> Kindest regards,
> Eric van Loon
> KLM Royal Dutch Airlines
>
> -----Original Message-----
> From: Luke Dahl [mailto:ldahl AT JPL.NASA DOT GOV]
> Sent: Thursday, March 25, 2004 21:39
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: Expiration after TSM upgrade from 5.1.6.4 to 5.2.2.1
>
> Hi Mark,
>     Output from q db f=d:
>           Available Space (MB): 118,780
>         Assigned Capacity (MB): 70,000
>         Maximum Extension (MB): 48,780
>         Maximum Reduction (MB): 15,924
>              Page Size (bytes): 4,096
>             Total Usable Pages: 17,920,000
>                     Used Pages: 11,215,255
>                       Pct Util: 62.6
>                  Max. Pct Util: 77.3
>               Physical Volumes: 1
>              Buffer Pool Pages: 32,768
>          Total Buffer Requests: 2,151,221,843
>                 Cache Hit Pct.: 99.04
>                Cache Wait Pct.: 0.00
>            Backup in Progress?: No
>     Type of Backup In Progress:
>   Incrementals Since Last Full: 0
> Changed Since Last Backup (MB): 2,785.80
>             Percentage Changed: 6.36
> Last Complete Backup Date/Time: 03/24/04 17:33:13
>
> This is the standard amount of expiration we see:
> 03/14/04 17:27:07     ANR0812I Inventory file expiration process 1692
> completed:
>                        examined 23194393 objects, deleting 1690917 backup
> objec
>                        cts, 0 archive objects, 0 DB backup volumes, and 0
> recov
>                        very plan files. 0 errors were encountered.
>
> This is currently running:
>  Process Process Description  Status
> s Number
> -------- --------------------
> -------------------------------------------------
>       26 Expiration           Examined 22853888 objects, deleting 20518954
> bac
>                                ckup objects, 0 archive objects, 0 DB backup
> vo
>                                olumes, 0 recovery plan files; 1 errors
> encount
>                                tered.
>
> I don't think the policies were correctly expiring data prior to the
> upgrade.  We've had numerous nodes that seemed to be storing much more than
> their policy dictated.  We tried to force expiration on the files and it
> didn't seem to make much of a difference.  Unfortunately we've been tweaking
> policies and trying to force a lot of data to drop off prior to the upgrade
> so I can't point to any particular reason why expiration is now expiring so
> much data.  You can see the db size has dropped as well due to the
> expiration - and we performed an unload/reload less than a month ago prior
> to the upgrade.  Being the db volume is one volume it could be disk
> contention, but it's a raw volume fiber attached (Sun T-3's).
>
> Luke
>
> "Stapleton, Mark" wrote:
>
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On 
> > Behalf Of
> > Luke Dahl
> > >    We upgraded our TSM Server (Solaris 9) from 5.1.6.4 to 5.2.2.1 last
> > >week.  Prior to the upgrade our expiration usually lasted about 24
> > hours
> > >(53Gb database).  It would generally post results like this:  examined
> > >16446439 objects, deleting 59582 backup objects
> > >Our 3494 library showed est. capacity of 60Tb and approximately 88 pct.
> > >util.
> >
> > What is your db's cache hit ratio? 24 hours for a 53GB database is
> > manifestly too long. It should get through expiration in 2-3 hours. Give
> > us the output from Q DB F=D.
> >
> > >Any idea why expiration wasn't running properly prior to the upgrade?
> > >Data integrity looks good but we've just cleared out loads of space.  I
> > >don't recall a known bug in expiration processing in 5.1.6.4 but may
> > >have missed something.  Thanks for your thoughts and input.
> >
> > How, exactly, did you "clear out loads of space"?
> >
> > --
> > Mark Stapleton
>
> **********************************************************************
> For information, services and offers, please visit our web site: 
> http://www.klm.com. This e-mail and any attachment may contain confidential 
> and privileged material intended for the addressee only. If you are not the 
> addressee, you are notified that no part of the e-mail or any attachment may 
> be disclosed, copied or distributed, and that any other action related to 
> this e-mail or attachment is strictly prohibited, and may be unlawful. If you 
> have received this e-mail by error, please notify the sender immediately by 
> return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij 
> NV (KLM), its subsidiaries and/or its employees shall not be liable for the 
> incorrect or incomplete transmission of this e-mail or any attachments, nor 
> responsible for any delay in receipt.
> **********************************************************************