On 06.04.07 at 06:29 naclos AT SWBELL DOT NET wrote:

Date: Fri, 7 Apr 2006 06:29:54 -0700
From: Andrew Carlson <naclos AT SWBELL DOT NET>
Reply-To: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: CLEANUP EXPTABLE / SHOW VERIFYEXPTABLE

What do these commands do?

--- Josh-Daniel Davis <xaminmo AT OMNITECH DOT NET> wrote:

Does anyone know how to tell how big the expiration table is?

The reason is that I ran CLEANUP EXPTABLE on Monday.
On one of my servers, it finished up almost immediately.
On the other, it's been running for almost 3 days.


Because of this, when I try to run EXPIRE INV, I get:

tsm: SERVER>expire inv
ANS8001I Return code 4.


tsm: SERVER>q act begint=-00:01
04/06/06 14:43:34 ANR2017I Administrator OPERATOR issued command:
EXPIRE
    INVENTORY (SESSION: 239372)
04/06/06 14:43:34 ANR4298I Expiration thread already processing -
unable
    to begin another expiration process. (SESSION: 239372)
04/06/06 14:43:34 ANR2017I Administrator OPERATOR issued command:
ROLLBACK
    (SESSION: 239372)


It doesn't show up in Q PROC, and tracing IM* and more only shows
failure
to obtain the lock.


I know it's running because of SHOW THREAD and Q ACT.


SHOW THREAD will shows this:

Thread 129: ImVerifyExpTabThread
  tid=33076, ktid=2588793, ptid=0, det=1, zomb=0, join=0, result=0,
sess=0
   Awaiting cond waitP->waiting (0x18d5ffe20), using mutex TMV->mutex

(0x111b091f8), in tmLock (0x100041a08)
   Stack trace:
     0x0900000000382554 _cond_wait_global
     0x0900000000382f64 _cond_wait
     0x0900000000383a2c pthread_cond_wait
     0x000000010000d91c pkWaitCondition
     0x0000000100041a10 tmLock
     0x000000010016cc5c ImLockFsId
     0x000000010016cafc ImLockFileSpace
     0x000000010067f168 LockFilespace
     0x0000000100681710 ImVerifyExpTabThread
     0x000000010000e9dc StartThread
     0x090000000036c50c _pthread_body



Q ACT has been showing many many of these message pairs:

04/06/06 13:19:04 CLEANUP EXPTABLE:  **** resetting 'hasactive',
objId=0:574058714 **** (SESSION: 134418)

04/06/06 13:19:04 CLEANUP EXPTABLE:  !!!! 'HasActive' flag set
incorrectly
for objId=0:574877220 (\ADSM.SYS\ÿþÿÿÿþCLUSTERDBÿþÿÿÿþ),
nodeName=NODENAME, fsName=\\nodename\c$ÿþÿÿÿþ !!!! (SESSION: 134418)

There are 125000 lines of this in the actlog in the last 67 hours.

The object IDs are not in order, so I have no idea how much longer
it's
going to run.

I'm assuming it will parse the entire Expiring.Objects table.


SHOW OBJDIR
... Expiring.Objects(78)


SHOW NODE 78
It's a b-tree root node with 99 subnodes.
It's 4 levels deep, and each node has a different number of children.
MaxCapacity is 1004, so potentially 1004^4.
I manually traversing the tree isn't feasible.


SHOW TREE Expiring.Objects
This just hangs for a long time.


I'm leaving it running, redirected to an outfile, but it's been 10
mins
for both the node that completed the CLEANUP quickly and the node
that
didn't.

When expiration was OK, it would take 6-10 hours with SKIPD=YES and
up to
a day and a half with SKIPD=NO, vs 4 hours on the "good" node.

There's no CANCEL CLEANUP or similar.

I'm hesitant to kill off the server and restart it simply because of
the
number of objects it's correcting.

So, should I just wait for the SHOW TREE to complete, or is there
some
other, faster and more simple way to see?


Thanks for any assistance.

-Josh-Daniel Davis

Andy Carlson
---------------------------------------------------------------------------
Gamecube:$150,PSO:$50,Broadband Adapter: $35, Hunters License: $8.95/month,
The feeling of seeing the red box with the item you want in it:Priceless.