TSM hanging on ANR0990I Server restart-recovery in progress.

[MrGerryDee]

Hello everyone,

I'm new to ADSM, may you kindly assist.. I am having a challenge whereby TSM suddenly hangs and the only way I can get in is to reboot the server. The TSM service is configured to start automatically, so if I do the ps command I can see the service is running but a dsmadmc does not give me a password prompt. So I cancel the process using kill -9 and then start the dsmserv direcly on console and this is what I get:

dsmserv
ANR7800I DSMSERV generated at 12:39:55 on Oct 10 2011.

Tivoli Storage Manager for AIX

Version 6, Release 3, Level 0.0


Licensed Materials - Property of IBM


(C) Copyright IBM Corporation 1990, 2011.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.


ANR7801I Subsystem process ID is 553012.
ANR0900I Processing options file /home/tsminst2/dsmserv.opt.
ANR7811I Using instance directory /home/tsminst2.
ANR4726I The Async I/O support module has been loaded.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.

The server does not move from that line. Anyone with a solution to this?

Regards

 

[marclant]

If the server crashed, DB2 is likely doing a crash-recovery which can take a few hours. Check db2diag.log to see what DB2 is doing.

 

[MrGerryDee]

Hi marclant,

I didn't find anything pointing to this error in the file /home/tsminst2/sqllib/db2dump/db2diag.log, it was last written in October 2013.

 

[RecoveryOne]

You may have log rotation on for db2diag. Run an ls -ltr in your db2dump directory. The newest file should be at the bottom of your output. For example, my current db2diag.log is actually db2diag.7.log due to log rotation.

 

[MrGerryDee]

Thanks RecoveryOne, the file had these entries:

2017-06-19-14.56.08.387693-300 I90286609A505 LEVEL: Event
PID : 376902 TID : 11567 PROC : db2sysc 0
INSTANCE: tsminst2 NODE : 000 DB : TSMDB1
APPHDL : 0-66 APPID: *LOCAL.DB2.170619195309
AUTHID : TSMINST2
EDUID : 11567 EDUNAME: db2stmm (TSMDB1) 0
FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20
CHANGE : STMM CFG DB TSMDB1: "Database_memory" From: "5808176" <automatic> To: "5569746" <automatic>

2017-06-19-14.56.08.389975-300 I90287115A491 LEVEL: Event
PID : 376902 TID : 11567 PROC : db2sysc 0
INSTANCE: tsminst2 NODE : 000 DB : TSMDB1
APPHDL : 0-66 APPID: *LOCAL.DB2.170619195309
AUTHID : TSMINST2
EDUID : 11567 EDUNAME: db2stmm (TSMDB1) 0
FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20
CHANGE : STMM CFG DB TSMDB1: "Sortheap" From: "1052" <automatic> To: "940" <automatic>

2017-06-19-14.59.08.435549-300 I90287607A491 LEVEL: Event
PID : 376902 TID : 11567 PROC : db2sysc 0
INSTANCE: tsminst2 NODE : 000 DB : TSMDB1
APPHDL : 0-66 APPID: *LOCAL.DB2.170619195309
AUTHID : TSMINST2
EDUID : 11567 EDUNAME: db2stmm (TSMDB1) 0
FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20
CHANGE : STMM CFG DB TSMDB1: "Sortheap" From: "940" <automatic> To: "1051" <automatic>

 

[marclant]

You'll have to find where "db2start" was issued in the db2diag.log and work your way down.

These technotes will help you determine if it's a crash recovery and how to monitor the progress if that is the case:
http://www-01.ibm.com/support/docview.wss?uid=swg21634679
http://www-01.ibm.com/support/docview.wss?uid=swg21610522
http://www-01.ibm.com/support/docview.wss?uid=swg21655286

 

[MrGerryDee]

Thank you marclant.. After waiting for a long time I issued a reboot of the AIX server and it then started successfully.