Crashing spectrum protect server

[DavrosDalek]

Hi Guys

I've got a windows based spectrum protect server here (Server Version 8, Release 1, Level 10.200) thats been running fine for the past 3 years, over the weekend it crashed. If i restart the services it runs for a few hours and crashes again with the following in the windows event log

TSM Server Diagnostic: ANR9999D: ADSM Exception Information: file = pkthread.c, line = 3824,Code = c0000005, Address = 00007FF9F7D44BE1
Attempt to read data at address 20~

I did capture the following just as it crashed this afternoon so suspect some sort of coruption thats causing the server to crash when it runs the defragmentation process.

ANR4392I CONTAINER DEFRAGMENTATION started processing stgpool PRIMARY_POOL.
Containers are only using 96.77% of 426508.82 GB due to fragmentation.
ANR0984I Process 4 for Move Container (Automatic) started in the BACKGROUND at
14:54:40.
ANR0984I Process 5 for Move Container (Automatic) started in the BACKGROUND at
14:54:40.
ANR0984I Process 6 for Move Container (Automatic) started in the BACKGROUND at
14:54:40.
ANR0984I Process 7 for Move Container (Automatic) started in the BACKGROUND at
14:54:40.
ANR0984I Process 8 for Move Container (Automatic) started in the BACKGROUND at
14:54:40.
ANR2017I Administrator ADMIN issued command: QUERY SESSION
ANR9999D_1199904567 SdWriteContainer(sdio.c:1571) Thread<94>: Improper setup
for writing to container 0000018F9DF9B610 (nil) True
ANR9999D Thread<94> issued message 9999 from:
ANR9999D Thread<94> 7ffc4f9545b4 OutDiagToCons()+b4
ANR9999D Thread<94> 7ffc4f94dc22 outDiagfExt()+112
ANR9999D Thread<94> 7ffc4f6a3ff9 SdWriteContainer()+449
ANR9999D Thread<94> 7ffc4f6a49bf SdWrite()+94f
ANR9999D Thread<94> 7ffc4f68086c ProcessDefragData()+27c
ANR9999D Thread<94> 7ffc4f67f3d3 ProcessStreamBuffer()+223
ANR9999D Thread<94> 7ffc4f67e550 SdCntrStreamThread()+e20
ANR9999D Thread<94> 7ffc4ed0c443 startThread()+153
ANR9999D Thread<94> 7ffc5e394f7f beginthreadex()+107
ANR9999D Thread<94> 7ffc5e395126 endthreadex()+192
ANR9999D Thread<94> 7ffc699c84d4 BaseThreadInitThunk()+14
ANR9999D Thread<94> 7ffc6c411791 RtlUserThreadStart()+21
ANR0204I The container state for N:\Pool1\d9\000000000000d9a8.dcf is updated
from AVAILABLE to UNAVAILABLE.
ANR3660E An unexpected error occurred while opening or writing to the
container. Container UNKNOWN in stgpool PRIMARY_POOL has been marked as
UNAVAILABLE and should be audited to validate accessibility and content.
ANR1181E sdtxn.c(1314): Data storage transaction 0:1270035 was aborted.
ANR2183W sdstream.c(2859): Transaction 0:1270035 was aborted.
ANR9999D_3977819583 SdGetNextEmptyBuffer(sdbuf.c:980) Thread<89>: Invalid sd
transaction descriptor. sd session is 0000018FAE60F780.
ANR9999D Thread<89> issued message 9999 from:
ANR9999D Thread<89> 7ffc4f9545b4 OutDiagToCons()+b4
ANR9999D Thread<89> 7ffc4f94dc22 outDiagfExt()+112
ANR9999D Thread<89> 7ffc4f6923d1 SdGetNextEmptyBuffer()+131
ANR9999D Thread<89> 7ffc4f67d1bb StreamSinkGetBuffer()+2ab
ANR9999D Thread<89> 7ffc4f67b228 sdStreamSink()+5a8
ANR9999D Thread<89> 7ffc4f677a88 RtrvChunk()+1a78
ANR9999D Thread<89> 7ffc4f66e645 sdRtrv()+6a5
ANR9999D Thread<89> 7ffc4f6385f9 SdMoveContainer()+2e59
ANR9999D Thread<89> 7ffc4f63a142 MoveContainerThread()+92
ANR9999D Thread<89> 7ffc4ed0c443 startThread()+153
ANR9999D Thread<89> 7ffc5e394f7f beginthreadex()+107
ANR9999D Thread<89> 7ffc5e395126 endthreadex()+192
ANR9999D Thread<89> 7ffc699c84d4 BaseThreadInitThunk()+14
ANR9999D Thread<89> 7ffc6c411791 RtlUserThreadStart()+21
ANR0986I Process 5 for Move Container (Automatic) running in the BACKGROUND
processed 457 items for a total of 154,390,528 bytes with a completion state
of FAILURE at 14:54:45.
ANR1893E Process 5 for Move Container (Automatic) completed with a completion
state of FAILURE.

I ran an audit container on the above mentioned container and it reported back no issues so i restarted the services and it ran for roughly 3 hours before crashing again.

Any ideas where to start ?

Thanks
Craig

 

[DavrosDalek]

Further info from q act

06/20/2022 17:28:20 ANR4392I CONTAINER DEFRAGMENTATION started processing
stgpool PRIMARY_POOL. Containers are only using 96.78%
of 426498.64 GB due to fragmentation.
06/20/2022 17:28:20 ANR0984I Process 3 for Move Container (Automatic) started
in the BACKGROUND at 17:28:20. (PROCESS: 3)
06/20/2022 17:28:20 ANR0984I Process 4 for Move Container (Automatic) started
in the BACKGROUND at 17:28:20. (PROCESS: 4)
06/20/2022 17:28:20 ANR0984I Process 5 for Move Container (Automatic) started
in the BACKGROUND at 17:28:20. (PROCESS: 5)
06/20/2022 17:28:20 ANR0984I Process 6 for Move Container (Automatic) started
in the BACKGROUND at 17:28:20. (PROCESS: 6)
06/20/2022 17:28:20 ANR0984I Process 7 for Move Container (Automatic) started
in the BACKGROUND at 17:28:20. (PROCESS: 7)
06/20/2022 17:28:23 ANR9999D_1199904567 SdWriteContainer(sdio.c:1571)
Thread<414>: Improper setup for writing to container
00000164877B4790 (nil) True (PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> issued message 9999 from: (PROCESS:
5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7ff45b4 OutDiagToCons()+b4
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7fedc22 outDiagfExt()+112
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7d43ff9 SdWriteContainer()+449
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7d449bf SdWrite()+94f
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7d2086c
ProcessDefragData()+27c (PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7d1f3d3
ProcessStreamBuffer()+223 (PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f7d1e550
SdCntrStreamThread()+e20 (PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ff9f73ac443 startThread()+153
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ffa0c1c4f7f beginthreadex()+107
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ffa0c1c5126 endthreadex()+192
(PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ffa196184d4
BaseThreadInitThunk()+14 (PROCESS: 5)
06/20/2022 17:28:23 ANR9999D Thread<414> 7ffa1bf11791
RtlUserThreadStart()+21 (PROCESS: 5)
06/20/2022 18:05:51 ANR3339I Default Label in key data base is TSM Server
SelfSigned SHA Key.
06/20/2022 18:05:51 ANR4726I The ICC support module has been loaded.
06/20/2022 18:05:51 ANR0990I Server restart-recovery in progress.

 

[marclant]

Appears it might be a known defect. Disable defrag or upgrade to 8.1.12 or higher.

More info here:

IT36573: ANR9999D SDWRITECONTAINER(SDIO.C:1590) THREAD<XXXX>: IMPROPER SETUP FOR WRITING TO CONTAINER XXXXXXXXX (NIL) TRUE

Starting with version 8.1.10.000 the IBM Spectrum Protect Server crashes with these symptoms during a directory container

www.ibm.com

 

[DavrosDalek]

Thanks Marclant, i have disabled the auto defag for the time being and the server has stopped crashing.