SQL2062N An error occurred while accessing media "". Reason code:""

shaaha85

Active Newcomer
Joined
Nov 23, 2010
Messages
37
Reaction score
0
Points
0
PREDATAR Control23

Hi All,

I am having issue with TDP SAP backup.

We are having daily,weekly full and log backup for this server . But we are facind issue with only weekly full backup.Backup is starting, but it failing after some time.IBM said it network issue and closed.But we are not havind any issue with network.

Total size of DB is around 8 TB.

Please assist on this one.


Error.log
09/16/12 23:53:45 sessSendVerb: Error sending Verb, rc: -50
09/16/12 23:53:45 sessSendVerb: Error sending Verb, rc: -50
09/16/12 23:53:45 ANS1017E Session rejected: TCP/IP connection failure
09/23/12 17:40:06 sessSendVerb: Error sending Verb, rc: -50
09/23/12 17:40:06 sessSendVerb: Error sending Verb, rc: -50
09/23/12 17:40:06 ANS1017E Session rejected: TCP/IP connection failure
09/30/12 22:51:36 ANS1329S Server out of data storage space
09/30/12 22:51:49 ANS1329S Server out of data storage space
10/08/12 14:39:00 sessSendVerb: Error sending Verb, rc: -50
10/08/12 14:39:00 sessSendVerb: Error sending Verb, rc: -50
10/08/12 14:39:00 ANS1017E Session rejected: TCP/IP connection failure
10/15/12 01:46:45 sessSendVerb: Error sending Verb, rc: -50
10/15/12 01:46:45 sessSendVerb: Error sending Verb, rc: -50
10/15/12 01:46:45 ANS1017E Session rejected: TCP/IP connection failure

Backom log:
BKI8806I: Data Protection for SAP(R) - BackOM 6.1.0.2 (367T) Apr 24 2009
BKI2027I: Using TSM-API version 5.5.1.13 (compiled with 5.3.0.0).
BKI8652I: DB2 version 'SQL09055' with 64 bits detected.

BKI8511I: The command is: -c b_db -a FBP -B 14 -S 4 -P 8 -e /db2/FBP/dbs/initFBP_WEEKLY.utl -O
BKI8637I: Full online backup of 'FBP' started ...
BKI8643I: Using vendor library at '/usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a' ...
BKI8654I: Using an autonomic buffer size with 14 buffers ...
BKI8641I: Using 4 session(s) ...
BKI8642I: Using a degree of parallelism of 8 ...
SQL2062N An error occurred while accessing media "VENDOR". Reason code:
"18".
BKI8548I: Elapsed time: 21 h 15 min 54 sec
BKI8634E: Backup command failed due to an error.
BKI8512I: Return code is: 605
#####################

db2diag.log:
2012-10-15-01.45.10.839861+060 E753020648A546 LEVEL: Warning
PID : 15335632 TID : 178248 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000 DB : FBP
APPHDL : 0-13591 APPID: 10.144.69.238.47149.12101422450
AUTHID : SAPFBP
EDUID : 178248 EDUNAME: db2agent (FBP) 0
FUNCTION: DB2 UDB, data management, sqldIndexCreate, probe:1
MESSAGE : ADM5541W Rebuilding index with IID "2" in object with ID "1058" and
table space ID "19" on table "SAPFBP ./BI0/ABBP_SC40".

2012-10-15-01.45.10.888015+060 E753021195A520 LEVEL: Warning
PID : 15335632 TID : 178248 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000 DB : FBP
APPHDL : 0-13591 APPID: 10.144.69.238.47149.12101422450
AUTHID : SAPFBP
EDUID : 178248 EDUNAME: db2agent (FBP) 0
FUNCTION: DB2 UDB, data management, sqldEndIndexCreate, probe:1
MESSAGE : ADM5542W Index rebuild of index(es) on table "SAPFBP
./BI0/ABBP_SC40" completed successfully.

2012-10-15-01.46.45.991244+060 I753021716A754 LEVEL: Error
PID : 15335632 TID : 173629 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000
EDUID : 173629 EDUNAME: db2med.156189.3 (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqluMapVend2MediaRCWithLog, probe:646
DATA #1 : String, 142 bytes
Vendor error: rc = 18 returned from function sqluvput.
Return_code structure from vendor library /usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a:

DATA #2 : Hexdump, 48 bytes
0x070000090AE07110 : 0000 0012 5453 4D45 5250 3A20 7772 6974 ....TSMERP: writ
0x070000090AE07120 : 652D 3E45 5843 2100 0000 0000 0000 0000 e->EXC!.........
0x070000090AE07130 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

2012-10-15-01.46.46.639814+060 E753022471A376 LEVEL: Error
PID : 15335632 TID : 173629 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000
EDUID : 173629 EDUNAME: db2med.156189.3 (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:901
MESSAGE : Media controller -- problem during writing to vendor device

2012-10-15-01.46.46.640067+060 E753022848A399 LEVEL: Error
PID : 15335632 TID : 173629 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000
EDUID : 173629 EDUNAME: db2med.156189.3 (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:901
MESSAGE : SQL2062N An error occurred while accessing media "". Reason code:
"".

2012-10-15-01.46.46.640276+060 E753023248A342 LEVEL: Error
PID : 15335632 TID : 173629 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000
EDUID : 173629 EDUNAME: db2med.156189.3 (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:901
DATA #1 : String, 9 bytes
VENDOR.18

2012-10-15-01.46.46.640478+060 E753023591A556 LEVEL: Severe
PID : 15335632 TID : 156189 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000 DB : FBP
APPHDL : 0-14192 APPID: *LOCAL.db2fbp.121014033159
AUTHID : DB2FBP
EDUID : 156189 EDUNAME: db2agent (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqlubMWResponse, probe:873
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
-2062
DATA #2 : Hexdump, 14 bytes
0x0700001802403678 : FFFF F7F2 5645 4E44 4F52 FF31 3800 ....VENDOR.18.

2012-10-15-01.46.46.640678+060 E753024148A920 LEVEL: Severe
PID : 15335632 TID : 156189 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000 DB : FBP
APPHDL : 0-14192 APPID: *LOCAL.db2fbp.121014033159
AUTHID : DB2FBP
EDUID : 156189 EDUNAME: db2agent (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqlubMWResponse, probe:873
MESSAGE : SQL2062N An error occurred while accessing media "". Reason code:
"".
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -2062 sqlerrml: 9
sqlerrmc: VENDOR 18
sqlerrp : sqlubMWR
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:

2012-10-15-01.46.46.640893+060 E753025069A499 LEVEL: Error
PID : 15335632 TID : 156189 PROC : db2sysc 0
INSTANCE: db2fbp NODE : 000 DB : FBP
APPHDL : 0-14192 APPID: *LOCAL.db2fbp.121014033159
AUTHID : DB2FBP
EDUID : 156189 EDUNAME: db2agent (FBP) 0
FUNCTION: DB2 UDB, database utilities, sqlubMWResponse, probe:875
MESSAGE : SQL2062N An error occurred while accessing media "VENDOR". Reason
code: "18".

Note:
everytime time of backup failed i can see following message in activity log before session failed.
But is not cancelling online full backup,session failed for log backup.

dose it impact the weely full backup.


15/10/12 01:46:45 ANR2998W The server log is 81 percent full. The server has
cancelled the oldest transaction in the log. (SESSION:
86600)
15/10/12 01:46:45 ANR0524W Transaction failed for session 72156 for node
FBP_SAP_WEEKLY (TDP R3 AIX) - data transfer interrupted.
(SESSION: 72156)
15/10/12 01:46:45 ANR2997W The server log is 81 percent full. The server
will delay transactions by 3 milliseconds. (SESSION:
86600)
15/10/12 01:46:46 ANR0514I Session 72156 closed volume DP3258. (SESSION:
72156)
15/10/12 01:46:46 ANR0483W Session 72156 for node FBP_SAP_WEEKLY (TDP R3
AIX) terminated - forced by administrator. (SESSION:
72156)
.
.
.

15/10/12 01:47:03 ANR0403I Session 72158 ended for node FBP_SAP_WEEKLY (TDP
R3 AIX). (SESSION: 72158)
15/10/12 01:47:03 ANR0403I Session 72152 ended for node FBP_SAP_WEEKLY (TDP
R3 AIX). (SESSION: 72152)
15/10/12 01:47:03 ANR0403I Session 87874 ended for node FBP_SAP_WEEKLY (TDP
R3 AIX). (SESSION: 87874)
15/10/12 01:47:03 ANR0403I Session 87875 ended for node FBP_SAP_WEEKLY (TDP
R3 AIX). (SESSION: 87875)
(these are weekly full backup sessions)
 
PREDATAR Control23

HI,

The error SQL2062N make reference to an error with the library / drive access


Tell us if this working fo you

best regards

http://www-903.ibm.com/kr/sevent/bbs/view.jsp?bbs_seq=582&bbs_table=Event_Tivoli

Hi, normally when you rerun the restore is complete, do you mean that? SQL2062N An error occurred while accessing media VENDOR Reason code: 18 is called out-of-memory error. draw looked to check the system's memory, check the value of ulimit on OS. worth noting at the bottom, but I do not know the exact OS type case. ===== =============================== SQL2062N An error occurred while accessing media VENDOR Reason code: 18 The error code 18 is described in the following way: 0018 E DSM_RC_ABORT_NO_MEMORY Explanation: The server ran out of memory System Action: TSM ended the current operation User Response: This is a temporary problem. Retry later or see your system administrator. This would indicate that the AIX system where the DB2 is located ran out of memory. Some cases have been solved by adjusting the ulimits on the O / S to (i: e ulimits-m unlimited) and unlimited retrying the restore If the failure recurs after adjusting ulimits can you also provide the TSM Actlog from the time of the failure as well as the current db2diag.log and dsmerror.log etc ==================================== ▤ Lee Jong Soo by what you've written is Welcome!, this time by configuring the TSM DB2 SAP backup and restore tests to have the following message would come out, and the same for the following commands once more take to restore normal was appears. ask you know why I'll db2test> db2 restore db Hep load / usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a taken at 20080218145911 SQL2539W Warning! Restoring to an existing database that is the same as the backup image database. The database files will be deleted Do you want to continue? (Y / n) y SQL2062N An error occurred while accessing media VENDOR Reason code: 18. DB2 Board also raised but did not answer so Tivoli side once again raise.
 
PREDATAR Control23

That is the same error we get if TSM requires the tape drive for something else. The classic problem is the TDP requires access to the logs to complete the backup, so it goes to restore the logs from TSM. At that time, all drives may be in use. As a restore is higher priority than a backup, it cancels the longest running backup to restore the logs for the backup it just cancelled. Doh! To get round this, we make sure we have about 3 days of logs on in a cached diskpool.

Anyway, you will need to look at the TSM Server logs at the time it stopped to see what happened here. My guess is another process stole your tape drive.
 
PREDATAR Control23

Whoops, sorry. You have the issue in the logs you provided.

15/10/12 01:46:45 ANR2998W The server log is 81 percent full. The server has
cancelled the oldest transaction in the log. (SESSION:
86600)
15/10/12 01:46:45 ANR0524W Transaction failed for session 72156 for node
FBP_SAP_WEEKLY (TDP R3 AIX) - data transfer interrupted.


The TSM log file is filling up so it automatically kills the longest running session. Increase the TSM log file if you can.
 
PREDATAR Control23

start by separating DB2 logs from data, create stg pool for logs (can be tape or disk->tape)
 
PREDATAR Control23

Thanks guys,

I am working on all the possiblities.


backup again started with following sessions,after running 20 hours, it got failed.

If the backup is failing due to TSM log filling

1) What are all the thing i can do
2) we dont have enough space to extend the log.
3) Increasing the sessions or lanfree will help in it?

tsm: TSM_BLUE>q log

Available Assigned Maximum Maximum Page Total Used Pct Max.
Space Capacity Extension Reduction Size Usable Pages Util Pct
(MB) (MB) (MB) (MB) (bytes) Pages Util
--------- -------- --------- --------- ------- --------- --------- ----- -----
12,988 12,388 600 12,356 4,096 3,170,816 6,969 0.2 92.2



228,286 Tcp/Ip RecvW 0 S 4.6 K 360.6 G Node TDP R3 FBP_SAP_WEEKLY
AIX
228,289 Tcp/Ip RecvW 0 S 4.6 K 366.1 G Node TDP R3 FBP_SAP_WEEKLY
AIX
228,291 Tcp/Ip RecvW 0 S 4.5 K 349.5 G Node TDP R3 FBP_SAP_WEEKLY
AIX
228,293 Tcp/Ip RecvW 0 S 4.6 K 372.7 G Node TDP R3 FBP_SAP_WEEKLY
AIX


Activity log:

22/10/12 01:33:35 ANR2998W The server log is 81 percent full. The server has
cancelled the oldest transaction in the log. (SESSION:
246106)
22/10/12 01:33:35 ANR0524W Transaction failed for session 228289 for node
FBP_SAP_WEEKLY (TDP R3 AIX) - data transfer interrupted.
(SESSION: 228289)
22/10/12 01:33:35 ANR8325I Dismounting volume DP2002 - 1 minute mount
retention expired.
22/10/12 01:33:35 ANR2997W The server log is 81 percent full. The server
will delay transactions by 3 milliseconds. (SESSION:
246106)
22/10/12 01:33:35 ANR8336I Verifying label of LTO volume DP2002 in drive
DRIVE44 (/dev/rmt240). (PROCESS: 234852)
22/10/12 01:33:35 ANR0514I Session 228289 closed volume DP4099. (SESSION:
228289)
22/10/12 01:33:35 ANR0483W Session 228289 for node FBP_SAP_WEEKLY (TDP R3
AIX) terminated - forced by administrator. (SESSION:
228289)
22/10/12 01:33:36 ANR8468I LTO volume DP2002 dismounted from drive DRIVE44
(/dev/rmt240) in library SP-VTL-2. (PROCESS: 234852)
22/10/12 01:33:36 ANR8325I Dismounting volume DP4094 - 1 minute mount
 
PREDATAR Control23

Are you able to complete 8 tb in 20hours

Slow backups usually tend to fill logs!
 
PREDATAR Control23

No ,it getting failed with below error.
I increased the backup session from 4 to 8.Does it help?
 
PREDATAR Control23

I would set the number of sessions to 0. DB2 will then automatically use what it thinks the the optimal number of sessions.

Settings I use are:

Number of buffers: 0
Buffer Size: 0
Parallelism: 0
Priority: 0
Number of I/O sessions: 2

Remember that the sessions setting in the initFBP_WEEKLY.utl relates to the parallelism setting in DB13. Max Sessions relates to the I/O sessions and is the number of tape drives used for the backup. In my init<SID>.utl, I have a max sessions of 2 and sessions of 34. The max number of mount points for the TSM node needs to match or exceed the I/O sessions.

The parallelism / sessions setting is used to determine how many tablespaces are backed up at one time. If most of you database resides in one tablespace, you will be waiting for that to complete. It may be help to balance data across tablespaces. Other things to consider are database reorgs, compression and archiving. We got our best backup speed inprovements once we got a top DBA to look at what was happening at the database level.

This is a BW system? Another thing to look at would be nearline storage to archive old data out but still have it accessable.
 
PREDATAR Control23

reduce log ....

How will that help? If you run out of log space reducing the log space will not help. Speeding up the backup will reduce the time the backup is holding the log open with an active transaction waiting to commit to the database. If the size of the log space cannot be increased then speeding up the backup would be the next sensible step.

Another option would be a log a support call with IBM for them to analysis why this backup is holding the logs open for so long. It may be normal behaviour. Remember though, it is not necessarily this backup causing all the logging. All other activity will be filling up the logs. I've had this happen from time to time, but in this case it seems to be regular.

Personally I'd increase the size of logs not reduce. What have I missed?
 
Top