TSM Sceduled backup failed

[cnukvd100]

Hi ,

I am facing a backup issue .
can any one please reply .

Below is the error log from the DB2 server. No activity log on the TSM .
-----------------------------

2014-01-29-01.00.09.997000+060 E3826747F496 LEVEL: Info
PID : 6584 TID : 7812 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7812 EDUNAME: db2agent (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:1687
MESSAGE : Starting an online db backup.

2014-01-29-01.00.13.007000+060 E3827245F556 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluvput, probe:963
DATA #1 : TSM RC, PD_DB2_TYPE_TSM_RC, 4 bytes
TSM RC=0x0000009D=157 -- see TSM API Reference for meaning.

2014-01-29-01.00.13.023000+060 I3827803F898 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluMapVend2MediaRCWithLog, probe:656
DATA #1 : String, 128 bytes
Vendor error: rc = 25 returned from function sqluvput.
Return_code structure from vendor library D:\IBM\SQLLIB\bin\db2tsm.dll:

DATA #2 : Hexdump, 48 bytes
0x0000000F2AE0CD38 : 9D00 0000 3936 3320 3135 3700 0000 0000 ....963 157.....
0x0000000F2AE0CD48 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0000000F2AE0CD58 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

2014-01-29-01.00.13.039000+060 I3828703F527 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluMapVend2MediaRCWithLog, probe:696
MESSAGE : Error in vendor support code at line: 963 rc: 157

2014-01-29-01.00.13.054000+060 E3829232F556 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluvend, probe:1134
DATA #1 : TSM RC, PD_DB2_TYPE_TSM_RC, 4 bytes
TSM RC=0x0000000B=11 -- see TSM API Reference for meaning.

2014-01-29-01.00.13.070000+060 E3829790F530 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:970
MESSAGE : Media controller -- problem during writing to vendor device

2014-01-29-01.00.13.070000+060 E3830322F597 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:970
MESSAGE : SQL2033N An error occurred while accessing TSM during the processing
of a database utility. TSM reason code: "".

2014-01-29-01.00.13.085000+060 E3830921F490 LEVEL: Error
PID : 6584 TID : 6120 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 6120 EDUNAME: db2med.7812.2 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:970
DATA #1 : String, 2 bytes
11

2014-01-29-01.00.13.101000+060 E3831413F609 LEVEL: Severe
PID : 6584 TID : 7812 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7812 EDUNAME: db2agent (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqlubHandleTSMError, probe:1319
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
-2033
DATA #2 : Hexdump, 7 bytes
0x0000000F2ABC5788 : 0FF8 FFFF 3131 00 ....11.

2014-01-29-01.00.13.101000+060 E3832024F1024 LEVEL: Severe
PID : 6584 TID : 7812 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7812 EDUNAME: db2agent (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqlubHandleTSMError, probe:1319
MESSAGE : SQL2033N An error occurred while accessing TSM during the processing
of a database utility. TSM reason code: "".
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -2033 sqlerrml: 2
sqlerrmc: 11
sqlerrp : sqlubHan
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:

2014-01-29-01.00.16.096000+060 E3940210F511 LEVEL: Error
PID : 6584 TID : 7356 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7356 EDUNAME: db2bm.7812.5 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluCheckIfAgentInterrupted, probe:1705
DATA #1 : Hex integer, 4 bytes
0x84000002

2014-01-29-01.00.16.112000+060 I3940723F553 LEVEL: Error
PID : 6584 TID : 7356 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7356 EDUNAME: db2bm.7812.5 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluWriteToQueue, probe:356
MESSAGE : SQL2001N The utility was interrupted. The output data may be
incomplete.

2014-01-29-01.00.16.112000+060 E3941278F511 LEVEL: Error
PID : 6584 TID : 7356 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7356 EDUNAME: db2bm.7812.5 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluCheckIfAgentInterrupted, probe:1705
DATA #1 : Hex integer, 4 bytes
0x84000002

2014-01-29-01.00.16.128000+060 I3941791F553 LEVEL: Error
PID : 6584 TID : 7356 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7356 EDUNAME: db2bm.7812.5 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluWriteToQueue, probe:356
MESSAGE : SQL2001N The utility was interrupted. The output data may be
incomplete.

2014-01-29-01.00.16.128000+060 E3942346F511 LEVEL: Error
PID : 6584 TID : 7356 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7356 EDUNAME: db2bm.7812.5 (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqluCheckIfAgentInterrupted, probe:1705
DATA #1 : Hex integer, 4 bytes
0x84000002

2014-01-29-01.05.53.788000+060 E3942859F474 LEVEL: Severe
PID : 6584 TID : 7812 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : ARCHIVE
APPHDL : 0-9352 APPID: *LOCAL.DB2.140129000002
AUTHID : ODADMIN HOSTNAME: localhost
EDUID : 7812 EDUNAME: db2agent (ARCHIVE) 0
FUNCTION: DB2 UDB, database utilities, sqlubcka, probe:887
MESSAGE : Backup terminated.

 

[Harry_Redl]

Hi,

what does the server say in the actlog? From the RC errors (157 and 11) I would say your server has run out of space ....

0157 S: DSM_RC_WILL_ABORT The transaction will be aborted.
0011 E: DSM_RC_ABORT_NO_REPOSIT_SPACE Server out of data storage space

Again - check the server activity log.

Harry

 

[cnukvd100]

Hi Harry ,
Thank you so much for the quick reply.
you are correct . this error happening from last couple of days . Yesterday i have created new volume with 102 GB space. even though . it was happened in the night 1 AM.

but one of the file space is still showing 100% pct util. how to increase the file space . is the error because of this?
Below are my available space now.

tsm: localhost_SERVER1>q volume


Volume Name Storage Device Estimated Pct Volume
Pool Name Class Name Capacity Util Status
------------------------ ----------- ---------- --------- ----- --------
G:\TSMVOL\VOL007 ARCHIVEPOOL DISK 100.0 G 27.1 On-Line
G:\TSMVOL\VOL008 ARCHIVEPOOL DISK 100.0 G 29.7 On-Line
G:\TSMVOL\VOL009 BACKUPPOOL DISK 100.0 G 99.5 On-Line
G:\TSMVOL\VOL01 BACKUPPOOL DISK 100.0 G 100.0 On-Line
G:\TSMVOL\VOL010 BACKUPPOOL DISK 100.0 G 100.0 On-Line
G:\TSMVOL\VOL011 BACKUPPOOL DISK 100.0 G 100.0 On-Line
G:\TSMVOL\VOL012 BACKUPPOOL DISK 100.0 G 100.0 On-Line
G:\TSMVOL\VOL02 BACKUPPOOL DISK 100.0 G 100.0 On-Line
H:\TSMVOL\VOL000 ARCHIVEPOOL DISK 100.0 G 26.7 On-Line
I:\TSMVOL\VOL007 ARCHIVEPOOL DISK 100.0 G 26.2 On-Line
I:\TSMVOL\VOL008 ARCHIVEPOOL DISK 100.0 G 27.0 On-Line
I:\TSMVOL\VOL009 BACKUPPOOL DISK 100.0 G 100.0 On-Line
I:\TSMVOL\VOL010 BACKUPPOOL DISK 100.0 G 99.5 On-Line
I:\TSMVOL\VOL011 BACKUPPOOL DISK 100.0 G 100.0 On-Line
I:\TSMVOL\VOL012 BACKUPPOOL DISK 100.0 G 100.0 On-Line
I:\TSMVOL\VOL013 BACKUPPOOL DISK 100.0 G 100.0 On-Line
I:\TSMVOL\VOL014 BACKUPPOOL DISK 100.0 G 100.0 On-Line
I:\TSMVOL\VOL015 BACKUPPOOL DISK 102.4 G 0.0 On-Line(new)


tsm: localhost_SERVER1>q stg


Storage Device Estimated Pct Pct High Low Next Stora-
Pool Name Class Name Capacity Util Migr Mig Mig ge Pool
Pct Pct
----------- ---------- ---------- ----- ----- ---- --- -----------
ARCHIVEPOOL DISK 1,953 G 26.8 26.8 90 70
BACKUPPOOL DISK 1,272 G 92.1 92.1 90 70 ((Util Pct was 100% before creating Volo15))
SPACEMGPOOL DISK 0.0 M 0.0 0.0 90 70


tsm: Localhost_SERVER1>q FI


Node Name Filespace FSID Platform Filespace Is Files- Capacity Pct
Name Type pace Util
Unicode?
--------------- ----------- ---- -------- --------- --------- ----------- -----
A55A4051 \\e75a4051- 1 WinNT NTFS Yes 99 GB 0.5
\h$
A55A4100-DB2 /ARCHIVE 3 DB2/NT64 APIB2/- No 4,404 GB 100.0(worried about this, how to increase free space )
NT64

Activity log : we have one worming

01/29/2014 01:00:13 ANR0522W Transaction failed for session 2787 for node
A55A4100-DB2 (DB2/NT64) - no space available in storage
pool BACKUPPOOL and all successor pools. (SESSION: 2787)

 

[cheffern]

So what this means is that the particular node you are accessing for the DB2 backup was not hitting the correct pool - (do you want it to hit BACKUPPOOL?)
or like you mentioned - the pool itself was too full to accept the backup from the beginning.
If just full, you would have to extend the diskspace to be able to handle the DB2 sizes - as each DB2 file will likely be large here - you need to make sure you do have the space ahead of time.
Else you can also possibly hang the DB2 system during the backup, should you run into issues.

 

[cnukvd100]

As per Activity log message .. BACKUPPOOL was used for backup. We had space issue for BACKUPPOOL. but after adding new volume free space got increased(I have not restarted the server, Do we need after adding new volume to BACKPOOL?).

And my File space (/ARCHIVE) is showing Utl Pct is 100%, does this causing the problem?
How to increase the free space for File Space (/ARCHIVE)?

Please reply.

 

[Harry_Redl]

Hi,

ignore the "q filespace" output - the percentage reported is the usage on the client and although I do not know how it is computed for the for the API clients (like DB2) it has nothing to do with the usage/free space in the TSM server.
Regarding the free space in the storage pool - if you add the volume to the storage pool, you do NOT need to restart the TSM server. It is available immediately (for DISK based stgpools - your case).
As Cheffern mentioned - the space needed for backup is allocated before the object is sent to TSM. If your DB2 is large enough, 102GB of free space can be insufficient and the "out of space" can be reported immediately - before client sends any data to the TSM.

Harry

 

[cnukvd100]

Thank you Harry/cheffern,

I have created one more storage volume with 100GB for safer side as we had one backup failure yesterday night (yes it is DISK). the volume file has created and file size shows 100GB even it is empty. We don't have disk space issue , I have 400GB more free space after creation of new volume.

I am waiting for schedule job to run.
Can i change the schedule job? I could not see how it was configured (it was configured by some one else).
can i change the schedule time ?
Can I run the incremental backup in business hours manually (how)?

sorry I am asking too many questions ?
I am new to TSM.

Thanks a lot for your efforts.

-Srinivas

 

[Harry_Redl]

Hi,

client schedules can be checked by "query sched * *" (with optional "format=detailed") - see "help query sched" to see the details. You can modify the schedule with "update sched" (see help as well) command.
Usually it is necessary to restart the scheduler service on the client after the schedule change - as the client can poll for the schedule in long interval by default. Restarting the scheduler service gets the information from the server immediately.
Running the job manually - can be done but the method may vary for various client types. It depends on the scheduler service used - if the client "listens" using the DSMCAD etc.

Harry

 

[cnukvd100]

The issue was resolved .
It took new volume (i.e 2nd new) to backup.

Thanks Harry...