ADSM-L

Re: [ADSM-L] Dealloc prohibited - transaction failed

2009-08-20 05:34:03
Subject: Re: [ADSM-L] Dealloc prohibited - transaction failed
From: "Loon, EJ van - SPLXM" <Eric-van.Loon AT KLM DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 20 Aug 2009 11:30:04 +0200
Hi KM!
I already tried the delete object command. In fact, the objectid is in
the error itself:

ANR9999D imutil.c(7001): ThreadId <51> unexpected rc=87 from bfDestroy
for objId=0.106696510

So when I issue a delete object 0 106696510 the same error is returned.
However, you might have put me on the right track:

show bfo 0 1066965103:

Bitfile Object: 0.1066965103
**Sub-bitfile 0.1066965103 is stored in the following aggregate(s)
  Super-bitfile: 0.1086967169, Offset: 0.0, Length 0.717

show bfo 0 1086967169:

Bitfile Object: 0.1086967169
**Super-bitfile 0.1086967169 contains following aggregated bitfiles
  0.1066965103

**Archival Bitfile Entry
  Bitfile Type: PRIMARY  Storage Format: 22
  Bitfile Size: 0.781  Number of Segments: 1
  Storage Pool ID: 6  Volume ID: 539755  Volume Name: L0301QL4
**Archival Bitfile Entry
  Bitfile Type: COPY  Storage Format: 22
  Bitfile Size: 0.781  Number of Segments: 1
  Storage Pool ID: -1  Volume ID: 540025  Volume Name: L041LLL4
**Archival Bitfile Entry
  Bitfile Type: COPY  Storage Format: 22
  Bitfile Size: 0.781  Number of Segments: 1
  Storage Pool ID: -2  Volume ID: 505610  Volume Name: *unknown*

Aha!!! There seems to be an orphaned entry to a second copypool! And
yes, we moved to new hardware about a month ago by starting a backup
stgpool to the new copypool. When it was in sync, I deleted all volumes
to the old copypool and deleted this pool. I guess this wasn't handled
correctly by the TSM server.
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
km
Sent: woensdag 19 augustus 2009 19:57
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Dealloc prohibited - transaction failed

On 19/08, Loon, EJ van - SPLXM wrote:
> I was hoping to be able to solve it myself, since the server is
running
> an unsupported TSM level (5.3.4.0), so I cannot open a PMR.
> Thanks anyway.
> Kind regards,
> Eric van Loon
> KLM Royal Dutch Airlines
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
Of
> km
> Sent: dinsdag 18 augustus 2009 18:34
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: Dealloc prohibited - transaction failed
>
> On 18/08, Loon, EJ van - SPLXM wrote:
> > Hi TSM-ers!
> > I have two nodes which I cannot delete. When I issue a DELETE
> FILESPACE
> > * for one of these node, I see the following errors in the log:
> >
> > ANR0984I Process 59041 for DELETE FILESPACE started in the
BACKGROUND
> at
> > 15:30:40.
> > ANR0800I DELETE FILESPACE * for node KL10143J started as process
> 59041.
> > ANR0802I DELETE FILESPACE * (backup/archive data) for node KL10143J
> > started.
> > ANR9999D ssalloc.c(1532): ThreadId <51> Dealloc prohibited -
> transaction
> > failed.
> > ANR9999D ThreadId <51> issued message 9999 from:
> > ANR9999D ThreadId <51>  0x0000000100017f78 outDiagf
> > ANR9999D ThreadId <51>  0x000000010032d480 ssDealloc
> > ANR9999D ThreadId <51>  0x0000000100365534 AfDeallocSegments
> > ANR9999D ThreadId <51>  0x0000000100365e68 AfDeleteBitfileFromPool
> > ANR9999D ThreadId <51>  0x00000001003663c8 AfDestroyAll
> > ANR9999D ThreadId <51>  0x00000001003608a4 bfDestroy
> > ANR9999D ThreadId <51>  0x00000001001816fc ImDeleteBitfile
> > ANR9999D ThreadId <51>  0x000000010018a510 imDeleteObject
> > ANR9999D ThreadId <51>  0x00000001003d3800 DeleteBackups
> > ANR9999D ThreadId <51>  0x00000001003d4c80 imFSDeletionThread
> > ANR9999D ThreadId <51>  0x00000001000063c8 StartThread
> > ANR9999D ThreadId <51>  0x090000000053650c _pthread_body
> > ANR9999D imutil.c(7001): ThreadId <51> unexpected rc=87 from
bfDestroy
> > for objId=0.1066965103
> > ANR9999D ThreadId <51> issued message 9999 from:
> > ANR9999D ThreadId <51>  0x0000000100017f78 outDiagf
> > ANR9999D ThreadId <51>  0x0000000100181740 ImDeleteBitfile
> > ANR9999D ThreadId <51>  0x000000010018a510 imDeleteObject
> > ANR9999D ThreadId <51>  0x00000001003d3800 DeleteBackups
> > ANR9999D ThreadId <51>  0x00000001003d4c80 imFSDeletionThread
> > ANR9999D ThreadId <51>  0x00000001000063c8 StartThread
> > ANR9999D ThreadId <51>  0x090000000053650c _pthread_body
> > ANR9999D imfsdel.c(1847): ThreadId <51> IM not able to delete object
> > 0.1066965103, rc: 19
> > ANR9999D ThreadId <51> issued message 9999 from:
> > ANR9999D ThreadId <51>  0x0000000100017f78 outDiagf
> > ANR9999D ThreadId <51>  0x00000001003d3840 DeleteBackups
> > ANR9999D ThreadId <51>  0x00000001003d4c80 imFSDeletionThread
> > ANR9999D ThreadId <51>  0x00000001000063c8 StartThread
> > ANR9999D ThreadId <51>  0x090000000053650c _pthread_body
> > ANR0987I Process 59041 for DELETE FILESPACE running in the
BACKGROUND
> > processed 25 items with a completion state of FAILURE at 15:30:40.
> >
> > I cannot find both the return codes 19 and 78 in the list of known
> > return codes for TSM...
> > It's reproducible. Anybody seen this before?
> > Thank you very much for any reply in advance!
> > Kind regards,
> > Eric van Loon
> >
>
> Looks like an IBM support case, probably a missing object or
something.
>
> -km

OK, then an AUDIT DB would probably fix the problem, but it needs
several hours to run which will incur downtime for the TSM server. But
this is the safe choice for fixing TSM DB problems.


Or, if you feel adventurous and dont really have anything to lose, you
could try to tracing the server while deleting the filespace.

trace disable *
trace enable ss bf af
trace begin /tmp/trace.out

delete filespace KL10143J *

trace flush
trace end

And then check the trace.out if you find anything that might be fixable
by you.

To see which file is causing the problem (could take a while):
SELECT ll_name, hl_name FROM backups WHERE object_id = 1066965103

And this will show you the bitfile object and all information on the
sub-bitfile object:
show bfo 0 1066965103

Then you could try to delete the offending object with:
delete object 1066965103

BUT, directly doing stuff with the DB is not really a good idea and
totally unsupported so you are on your own here. :)


-km
**********************************************************************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message. 

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286 
**********************************************************************