Insufficient log space - TSM server wouldn't start

jojo72

ADSM.ORG Member
Joined
Aug 13, 2009
Messages
29
Reaction score
0
Points
0
PREDATAR Control23

HI!

Went through troubleshooting. This shows that the recovery log is full. Located at 12 GB at c:\tsm\server1\log1.dsm and the maximum limit should be 13 GB as i know about it.

ANS1316E The server does not have enough recovery log space to continue the current operation

I attempted to restart the server but received an error

ANR9999D adminit.c(1269): ThreadId<0> Insufficient log space to update table Administrative.Attributes.


Tried to start command by expanding of recovery logfile:

C:\Program Files\Tivoli\TSM\server>dsmserv extend log /tsmlog/server1/log1.dsm 1
000
The server has not been properly installed.
Could not find the output logfile key.
Issue dsmserv -help for a usage statement

ANR0900I Processing options file C:\Program Files\Tivoli\TSM\server\dsmserv.op-
t.
ANR7800I DSMSERV generated at 08:35:01 on Jul 24 2008.

Tivoli Storage Manager for Windows
Version 5, Release 5, Level 1.0

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2007.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR4726I The ICC support module has been loaded.
ANR0200I Recovery log assigned capacity is 12288 megabytes.
ANR0201I Database assigned capacity is 65536 megabytes.
ANR0306I Recovery log volume mount in progress.
ANR2441E EXTEND LOG: Volume C:\TSMLOG\SERVER1\LOG1.DSM is already defined as a
recovery log volume.
ANR7835I The server thread 1 (tid 4876) terminated in response to server
shutdown.
ANR7835I The server thread 7 (tid 5728) terminated in response to server
shutdown.
ANR7835I The server thread 8 (tid 6128) terminated in response to server
shutdown.
ANR0991I Server shutdown complete.


How can i slove this? Please help me.

System: TSM server 5.5
 
PREDATAR Control23

Try to start dsmserv from commandline:
dsmserv extend log [new_volume_name] [new_volume_size] with size in megabytes

Then immediately start a backup db to free log space

http://publib.boulder.ibm.com/infoc...tm?path=2_2_6_10#dqx2r_cmd_dsmserv_extend_log

Jeroen was faster :)

When TSM runs again you should define a db backup trigger to ensure automatic db backup when the log usage grows. On my servers I use log full percentage 60%.
And then it is important to reduce log size again to have 1 or better 2 GB as a reserve
 
PREDATAR Control23

13 GB is the max limit for recovery log in TSM v.5 and a recommendation is not to extend to exactly 13 GB because then it's impossible to get TSM server working again if a similar failure occurs (probably you haven't run "backup db type=full" completely since a long time ago).

So try creating a new dbvolume and only extend about 600 MB.

Once TSM Server is up and running (start it in the foreground "dsmserv") disable client sessions and run a complete TSM DB backup which will empty the log.

Good Luck!
 
PREDATAR Control23

HI!

Thank you for your feedbacks, but I have still trouble to run the command and got errors.

C:\Program Files\Tivoli\TSM\server>dsmserv extend log c:\tsmlog\server1\log2.dsm
640
The server has not been properly installed.
Could not find the output logfile key.
Issue dsmserv -help for a usage statement
ANR0900I Processing options file C:\Program Files\Tivoli\TSM\server\dsmserv.op-
t.
ANR7800I DSMSERV generated at 08:35:01 on Jul 24 2008.
Tivoli Storage Manager for Windows
Version 5, Release 5, Level 1.0
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2007.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR4726I The ICC support module has been loaded.
ANR0200I Recovery log assigned capacity is 12288 megabytes.
ANR0201I Database assigned capacity is 65536 megabytes.
ANR0306I Recovery log volume mount in progress.
ANR9969E Unable to open volume C:\TSMLOG\SERVER1\LOG2.DSM. The most likely
reason is that another TSM server is running and has the volume allocated.
ANR2404E EXTEND LOG: Volume C:\TSMLOG\SERVER1\LOG2.DSM is not available.
ANR7835I The server thread 1 (tid 2184) terminated in response to server
shutdown.
ANR7835I The server thread 7 (tid 4012) terminated in response to server
shutdown.
ANR7835I The server thread 8 (tid 3976) terminated in response to server
shutdown.
ANR0991I Server shutdown complete.
 
PREDATAR Control23

try to specify explicitly server configuration:
dsmserv -k Server1 extend log c:\tsmlog\server1\log2.dsm 640
or
dsmserv -optfile="C:\path\to\your\dsmserv.opt" extend log c:\tsmlog\server1\log2.dsm 640
 
PREDATAR Control23

thanks - I am running in extend operation, but should the process take so long time? This feels nonended time....

Got the following messages:
)).
ANR0433W Session 10 for OCS01 (WinNT) refused - insufficient recovery log
space.
ANR0403I Session 10 ended for node OCS01 (WinNT).
ANR2997W The server log is 97 percent full. The server will delay transactions
by 300 milliseconds.
ANR0406I Session 11 started for node VC-WIN-01 (WinNT) (Tcp/Ip vc-win-01.signo-
.no(61710)).
ANR0433W Session 11 for VC-WIN-01 (WinNT) refused - insufficient recovery log
space.
ANR0403I Session 11 ended for node VC-WIN-01 (WinNT).
ANR2997W The server log is 97 percent full. The server will delay transactions
by 300 milliseconds.
ANR2997W The server log is 97 percent full. The server will delay transactions
by 300 milliseconds.
ANR0406I Session 12 started for node MGMT01 (WinNT) (Tcp/Ip mgmt01.signo.no(48-
36)).
ANR0433W Session 12 for MGMT01 (WinNT) refused - insufficient recovery log
space.
ANR0403I Session 12 ended for node MGMT01 (WinNT).
ANR2997W The server log is 97 percent full. The server will delay transactions
by 300 milliseconds.
ANR2997W The server log is 97 percent full. The server will delay transactions
by 300 milliseconds.
 
PREDATAR Control23

may you post output of?
q log f=d
q logv f=d
q pr
 
PREDATAR Control23

No - i can't start TSM server before I have extended the logfile
 
PREDATAR Control23

what about
dsmserv -optfile="C:\path\to\your\dsmserv.opt" display logvolumes

PS: you can enlarge DB log to 13000 MB and it is working
 
PREDATAR Control23

The server has not been properly installed.
Could not find the output logfile key.
Issue dsmserv -help for a usage statement
ANR0900I Processing options file C:\Program Files\Tivoli\TSM\server\dsmserv.opt.
..
ANR9969E Unable to open volume C:\TSMLOG\SERVER1\LOG2.DSM. The most likely
once more time. Are you sure you are loading the correct server?
Because that one is the default one. Look on drive for "dsmserv.opt" if you are not sure, might be in location like "C:\Program Files\Tivoli\TSM\server1\dsmserv.opt" or elsewhere
 
PREDATAR Control23

I have ran the following

cd C:\Program Files\Tivoli\TSM\server\
..\console\dsmfmt -log emerglogg.dsm 500
dsmserv -o ..\server1\ C:\Program Files\Tivoli\TSM\server\dsmserv.opt extend log emerglogg.dsm 400
dsmserv -o ..\server1\dsmserv.opt

what was the fault?
 
PREDATAR Control23

C:\Program Files\Tivoli\TSM\server>dsmserv -o ..\server1\ C:\Program Files\Tivol
i\TSM\server1\dsmserv.opt extend log emerglogg.dsm 400
The server has not been properly installed.
Could not find the output logfile key.
Issue dsmserv -help for a usage statement

ANR0905W Options file ..\server1\ not found.
ANR7800I DSMSERV generated at 08:35:01 on Jul 24 2008.

Tivoli Storage Manager for Windows
Version 5, Release 5, Level 1.0

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2007.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0200I Recovery log assigned capacity is 8 megabytes.
ANR0201I Database assigned capacity is 65536 megabytes.
ANR0306I Recovery log volume mount in progress.
ANR0353I Recovery log analysis pass in progress.
ANR0354I Recovery log redo pass in progress.
ANR0355I Recovery log undo pass in progress.
ANR0352I Transaction recovery complete.
ANR1635I The server machine GUID, 7c.36.fd.a1.a1.c9.11.dd.81.5e.00.1a.64.d4.0d-
.f8, has initialized.
ANR2100I Activity log process has started.
ANR4726I The NAS-NDMP support module has been loaded.
ANR1792W HBAAPI vendor library failed to load or is missing.
ANR8843E Initialization failed for SCSI library AUTOLTLIB - the library will be
inaccessible.
ANR8843E Initialization failed for SCSI library AUTODLTLIB - the library will
be inaccessible.
ANR8441E Initialization failed for SCSI library AUTODLTLIB.
ANR8441E Initialization failed for SCSI library AUTOLTLIB.
ANR1414W Volume 171BAAL4 access mode is "read-only" due to previous write
error.
ANR1412W Volume 606BAAL4 access mode is "unavailable".
ANR1414W Volume 172BAAL4 access mode is "read-only" due to previous write
error.
ANR1414W Volume 346BAAL4 access mode is "read-only" due to previous write
error.
ANR1414W Volume 349BAAL4 access mode is "read-only" due to previous write
error.
ANR1412W Volume 173BAAL4 access mode is "unavailable".
ANR1414W Volume 983BABL4 access mode is "read-only" due to previous write
error.
ANR1412W Volume 982BABL4 access mode is "unavailable".
ANR1414W Volume 037BAAL4 access mode is "read-only" due to previous write
error.
ANR1414W Volume 038BAAL4 access mode is "read-only" due to previous write
error.
ANR1414W Volume 984BABL4 access mode is "read-only" due to previous write
error.
ANR1414W Volume 345BAAL4 access mode is "read-only" due to previous write
error.
ANR1412W Volume 857BDSL4 access mode is "unavailable".
ANR0984I Process 1 for EXPIRATION started in the BACKGROUND at 12:45:59.
ANR2803I License manager started.
ANR8260I Named Pipes driver ready for connection with clients.
ANR0811I Inventory client file expiration started as process 1.
ANR0993I Server initialization complete.
ANR2560I Schedule manager started.
ANR0916I TIVOLI STORAGE MANAGER distributed by Tivoli is now ready for use.
ANR2828I Server is licensed to support IBM System Storage Archive Manager.
ANR2828I Server is licensed to support Tivoli Storage Manager Basic Edition.
ANR2828I Server is licensed to support Tivoli Storage Manager Extended Edition.
ANR8200I TCP/IP Version 4 driver ready for connection with clients on port
1500.
ANR4391I Expiration processing node EXCH02, filespace EXCH02\SystemState\NULL\-
System State\SystemState, fsId 1, domain STANDARD, and management class DEFAULT
- for BACKUP type files.
ANR4391I Expiration processing node EXCH02, filespace EXCH02\SystemState\NULL\-
System State\SystemState, fsId 1, domain STANDARD, and management class DEFAULT
- for BACKUP type files.
ANR8439I SCSI library TS3200 is ready for operations.
TSM:BACKUP01_SERVER1>
ANR0984I Process 2 for SPACE RECLAMATION started in the BACKGROUND at 12:46:01.A
NR4930I Reclamation process 2 started for primary storage pool LTOPOOL1
automatically, threshold=60, duration=None.
ANR1040I Space reclamation started for volume 347BAAL4, storage pool LTOPOOL1
(process number 2).
ANR1044I Removable volume 347BAAL4 is required for space reclamation.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\ SENTRUM
CASH AND CARRY.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 1 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 2 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 3 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 4 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 5 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 6 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ BILDE 7 C & C.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ IMG_6130.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ IMG_6133.JPG.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\BILDER\SENTRUM
CASH AND CARRY\ THUMBS.DB.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\ALLE\MALER\BREVMAL\
BREVMAL SIGNO DOKKEN AS.DOCX.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\MØTER I LEDERTEAMET\REFERA-
TER\ REFERAT FRA MØTE I LEDERTEAMET 30.11.10.DOCX.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\MØTER MED SIGNO\ MØTEPLAN
SIGNO 2011.DOC.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\ÅRSPLAN\2011\ INNSPILL TIL
ÅRSPLAN FOR 2011 ETTER PERSONALMØTE 04.12.10.DOCX.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\ÅRSPLAN\2011\ STRATEGISK
ÅRSPLAN FOR 2011, SIGNO.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\ÅRSPLAN\2011\ VEDLEGG TIL
STRATEGISK ÅRSPLAN FOR 2011, SIGNO.PDF.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\1.
ARBEIDSARKIV\A) FASTE SAKER (MØTER, BUDSJETT, M.M.)\ÅRSPLAN\2011\ ÅRSPLAN 2011
FOR DEN ENKELTE VIRKSOMHET, SKISSE TIL ELEMENTER FRA HOVEDKONTORET.DOCX.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\BRUKERE\ASBKRA\2.
REFERANSEARKIV\F) ALFABET\ MALER.
ANR1162W Space reclamation skipping damaged file on volume 347BAAL4: Node
BRG01, Type Backup, File space \\brg01\d$, File name \ASB\GRUPPER\DIVERSE ASVO
MAPPER\ADMINISTRASJON\BUDSJETT 2011\ KOPI AV BUDSJETT 2011 24 NOV.XLSX.
ANR1041I Space reclamation ended for volume 347BAAL4.
ANR4932I Reclamation process 2 ended for storage pool LTOPOOL1.
ANR0985I Process 2 for SPACE RECLAMATION running in the BACKGROUND completed
with completion state FAILURE at 12:46:01.
ANR4936I Reclamation of storage pool LTOPOOL1 has ended. Files reclaimed: 0,
Bytes reclaimed: 0, Files reconstructed: 0, Unreadable files: 0.
ANR1305I Disk volume E:\TSMDATA\SERVER1\DISK1.DSM varied online.
ANR2998W The server log is 81 percent full. The server has cancelled the oldest
transaction in the log.
ANR2997W The server log is 82 percent full. The server will delay transactions
by 3 milliseconds.
ANR0314W Recovery log usage exceeds 92 % of its assigned capacity.
 
PREDATAR Control23

If you can start TSM server and log in from dsmadmc:
task 0)
after login disable sessions to avoid increasing log usage and cancel processes
disable sessions
cancel process [number]
Cancel also sessions if there are already some.
Don't cancel DB backup if it already started.

task 1)
you need to create new log volume by using:
define logv C:\tsm\server1\LOG2.DSM 640

task 2)
you need to extend log:
extend log 640

If you can't start TSM server:
task 1)
you need to create new log volume by using:
C:\Program Files\Tivoli\tsm\console\dsmfmt -log C:\tsm\server1\LOG2.DSM 640

task 2)
you need to extend log:
C:\Program Files\Tivoli\tsm\server\dsmserv extend log C:\tsm\server1\LOG2.DSM 640
 
PREDATAR Control23

JoJo72 when and if you get the TSM instance running complete a db backup as soon as possible inorder to decrease the Recovery Log utilization(!)

1) disable session
2) backup db type=full devc=<Tape or File deviceclass>
3) Verify complete db backup, q volhist type=dbb begind=today
4) enable ses

After this everything should work normal again.

Regards,
Nicke
 
Top