VTL/TSM Media Mount Failed after 1 Tape is Full..

[faisalsiddiqui]

we are facing couple of errors during TSM/RMAN TDPO backup on VTL.
After taking 800GB data on VTL through SAN AGENT, TSM Server is unable to mount next tape in drive and drive path is taken offline.

Please note that:-
· We are emulating ADIC iScalar2000 with HP LTO3 from VTL and we are running TSM Server version 5.5.4.
· We have sufficient number of scratch Tapes in Storage Pool.

As usual i need support in this regard. please help me out asap.


[email protected]

RMAN Error LOG:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on t1 channel at 07/12/2010 21:01:33
ORA-19502: write error on file "11likjt3_1_1", blockno 136043009 (blocksize=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
ANS1312E (RC12) Server media mount not possible

TSM Activity Log
ANR0408I Session 1289 started for server EDL_10GNODE1 (AIX-RS/6000) (Tcp/Ip) for library sharing.
ANR0409I Session 1289 ended for server EDL_10GNODE1 (AIX-RS/6000).
ANR8336I Verifying label of LTO volume L30034 in drive EDL05_SAN_EOCRAC1 (/dev/mt5).
ANR8468I LTO volume L30034 dismounted from drive EDL05_SAN_EOCRAC1 (/dev/mt5) in library EDL.
ANR8485E No drives are available to be mounted in R/W mode with format 00000000 in library EDL.
ANR9790W Request to mount volume L30035 for library client EDL_10GNODE1 failed.
ANR0530W Transaction failed for session 1211 for node 10GNODE1_TDPO (TDP Oracle AIX) - internal server error detected.
ANR0514I Session 1211 closed volume L30035.
ANR0514I Session 1211 closed volume L30034.
ANR0415I Session 1293 proxied by EDL_10GNODE1 started for node 10GNODE1_TDPO.
ANR0408I Session 1294 started for server EDL_10GNODE1 (AIX-RS/6000) (Tcp/Ip)
for library sharing.
ANR0409I Session 1294 ended for server EDL_10GNODE1 (AIX-RS/6000).
ANR0535W Transaction failed for session 1293 for node 10GNODE1_TDPO (TDP Oracle AIX) - insufficient mount points available to satisfy the request.
ANR0408I Session 1287 started for server EDL_10GNODE1 (AIX-RS/6000) (Tcp/Ip) for library sharing.
ANR0409I Session 1287 ended for server EDL_10GNODE1 (AIX-RS/6000).
ANR8214E Session open with 10.25.1.85 failed due to connection refusal.
ANR0454E Session rejected by server EDL_10GNODE1, reason: Communication Failure.
ANR8390W Failure connecting to library client EDL_10GNODE1 to manage volume L30034.
ANR0408I Session 1288 started for server EDL_10GNODE1 (AIX-RS/6000) (Tcp/Ip) for library sharing.
ANR1787W Not able to open adapter com.ibm-df1000fd-2.
ANR1787W Not able to open adapter com.ibm-df1000fd-3.
ANR1787W Not able to open adapter com.ibm-df1000fd-4.
ANR8963E Unable to find path to match the serial number defined for drive EDL05_SAN_EOCRAC1 in library EDL.
ANR8873E The path from source TSMPRD to destination EDL05_SAN_EOCRAC1 (/dev/mt5) is taken offline.
ANR8779E Unable to open drive /dev/mt5, error number=16.

 

[mccleld]

Interesting errors, some of which I'd not come across before. You don't state your platforms/OS versions and TSM code levels here - these could help.

If looking in the IBM/TSM docs and a Google on the error codes, which I'm sure you've already done prior to posting here, have yielded nothing then I'd start by trying to identify which of your components is at fault here - a few questions to help: can you successfully perform TDP/RMAN backups of your database over the LAN rather than the SAN? Can you perform LANfree backups using the SAN Storage Agent of non-TDP data (i.e. performing an archive of some large files on your OS). Are any other similar LANfree clients experiencing any similar problems writing to the VTL?

If these all point to the SAN Storage Agent on your Oracle server being the likely culprit, double check that the tape device driver (Atape?) you're running is supported with the EDL's emulated tape drives and in line with your TSM Server's equivalent version. And double check that the paths to the drives are correct (the ANR8963E above leads me to suspect this might not be the case).

HTH,

//DMc, London

 

[faisalsiddiqui]

TSM is unable to change next tape after filling 1 tape

Hi,

Thanks for your response. actually im trying to take DB backup RMAN - TDPO using SAN Agent, which is upto 3.2 TB after 800GB backup TSM is unable to change the next tape.

TSM Server is running on AIX 5.3 5300-06

TSM Server Code:-
Storage Management Server for AIX-RS/6000 - Version 5, Release 5, Level 4.0

Integrated Solutions Console 6.0.1.1
Integrated Solutions Console 6.0.1
IBM Tivoli Storage Manager - Administration Center 5.5.0.0

can you successfully perform TDP/RMAN backups of your database over the LAN rather than the SAN?
Yes we can do TDP/RMAN LAN based Backup which is upto 200GB

Can you perform LANfree backups using the SAN Storage Agent of non-TDP data (i.e. performing an archive of some large files on your OS).
Yes we can perform LAN FREE Backups using Storage Agent BA/Archive. also performing archives which is upto 600GB
on others servers we are doing SAN BASED BACKUP using RMAN TDPO which up to 700-800 GB Daily.


Are any other similar LANfree clients experiencing any similar problems writing to the VTL?
NO

If these all point to the SAN Storage Agent on your Oracle server being the likely culprit, double check that the tape device driver (Atape?) you're running is supported with the EDL's emulated tape drives and in line with your TSM Server's equivalent version. And double check that the paths to the drives are correct (the ANR8963E above leads me to suspect this might not be the case).

well, EMC (DL3D300) VTL dont provide device driver for TSM all we have to do is to work around and discover devices supported by tsm. so changer and devices comes as lbx and /dev/mtx. all paths are working fine drives also paths are correct. all drives are added in TSM as /dev/mtx to nth...

we are using TSM 5.5.4 Device driver on client and also on TSM server.

Issue is:-
When TSM filled one tape, for some reasons TSM is unable to change the Next Tape... but why???


Looking for your support, IBM is simply saying to upgrade the TSM Server to 6.1.X that will resolve the issue.

Please help me out ASAP.

Regards,
Faisal

 

[mccleld]

Re-reading the top of your mail, I see you're emulating HP LTO3 drives, so TSM Device Driver is the most appropriate, and you're at the same 5.5.4 level on both server and LANfree client.

IBM saying 'upgrade to 6.x' is unacceptable, so push back on this. Your version 5.5.x is still supported and receiving fixes, so the most that they should be able to ask you for is to upgrade within your 5.5.x level. On Google, I see some related hits, one on IBM.com where the ANR8485 message isn't reporting quite enough information in its FORMAT field, to be fixed in 5.5.5.

I can't think of anything else at the moment - anyone else have any ideas?

//DMc, London UK

 

[faisalsiddiqui]

Thanks for your prompt reply,

basically i cannot push IBM for support our licenses has been expired so they are not going to support us at the moment.

to upgrade TSM from 5.5.4 to upgrade TSM 5.5.5 can your refer any download link.

Thanks and Regards,
Faisal

Re-reading the top of your mail, I see you're emulating HP LTO3 drives, so TSM Device Driver is the most appropriate, and you're at the same 5.5.4 level on both server and LANfree client.

IBM saying 'upgrade to 6.x' is unacceptable, so push back on this. Your version 5.5.x is still supported and receiving fixes, so the most that they should be able to ask you for is to upgrade within your 5.5.x level. On Google, I see some related hits, one on IBM.com where the ANR8485 message isn't reporting quite enough information in its FORMAT field, to be fixed in 5.5.5.

I can't think of anything else at the moment - anyone else have any ideas?

//DMc, London UK

 

[faisalsiddiqui]

ANR8390W Failure connecting to library client

Hi again,

Yesterday i've upgraded TSM 5.5.4 to 5.5.4.2 and issue is still the same.

but new error i found in tsm ACT LOG.

ANR8390W Failure connecting to library client 10GNODE1 to manage volume L30036

can you please help me out to fix this issue?

Regards,
Faisal

Thanks for your prompt reply,

basically i cannot push IBM for support our licenses has been expired so they are not going to support us at the moment.

to upgrade TSM from 5.5.4 to upgrade TSM 5.5.5 can your refer any download link.

Thanks and Regards,
Faisal

 

[TonyB]

Have you defined the LAN-Free agent to both the library client and library server TSM instances?

 

[faisalsiddiqui]

Hi,

Thanks for your reply, Yes i did the same.

first i did the on TSM Server (Enable Node LAN-FREE movement) the copy paste the out on client node and start dsmsta on client side.

in q ses and q server i can see my agent, i think if there might be any issue regarding this i cant do LAN FREE backup.

im also copy/paste the output of validate lan free data movement.

tsm: TSMPRD>validate lanfree EOCRAC1_ORC TS3_10GN1
ANR0387I Evaluating node EOCRAC1_ORC using storage agent TS3_10GN1 for LAN-free data movement.

Node Storage Operation Mgmt Class Destination LAN-Free Explanation
Name Agent Name Name capable?
----- -------- --------- ---------- ------------ --------- --------------------
EOCR- TS3_10G- BACKUP STANDARD AIX_ORC_DB_- Yes
AC1- N1 TAPE_POOL
_ORC


please let me know or if we can proceed further.

Regards,
Faisal

Have you defined the LAN-Free agent to both the library client and library server TSM instances?

 

[lapfrank]

Can you ping the lanfree client from both TSM Servers ?

tsm prompt> ping server storageagent_name