VTL I/O error when cartridge gets full.

R

ruccsito

ADSM.ORG Member
Joined
Aug 13, 2009
Messages
14
Reaction score
2
Points
0
Hi guys.

I need some help with a problem. We are running TSM 6.3.3.200 on RHEL 6.4 with ProtecTIER TS7620.

I have a VTL defined with 8 drives and 100 cartidges, LIN_Tape latest version is installed on the Linux server.

Almost everything is working ok, the problem is when a backup is running a the virtual cartridge gets full, we get an I/O error on the tape, mark it as read only and mount the next tape.

Backups are completing ok, but in the morning I have the full tape with FILLING status and READONLY access.
Anyone else have this problem? This is a new implementation so this wasn't working before, I noticed it when the first tape got full.

Thank you guys!

Error details:
ANR8337I LTO volume 000017L3 mounted in drive DRV1 (/dev/IBMtape0).
ANR0513I Process 8 opened output volume 000017L3.
ANR8311E An I/O error occurred while accessing drive DRV1 (/dev/IBMtape0) for
WRITE operation, errno = 5, rc = 1.
ANR9999D_3130205688 ErrorHandler(pvrntp.c:8378) Thread<254>: Unexpected RC=1,
VOL=000017L3, DRIVE=DRV1 (/dev/IBMtape0)
ANR9999D Thread<254> issued message 9999 from:
ANR9999D Thread<254> 0x00000000d6d4e3 OutDiagToCons
ANR9999D Thread<254> 0x00000000d702e5 outDiagfExt
ANR9999D Thread<254> 0x00000000e581b5 ErrorHandler
ANR9999D Thread<254> 0x00000000e5ef70 DumpBlock
ANR9999D Thread<254> 0x00000000e60221 NtpWriteNC
ANR9999D Thread<254> 0x0000000096b509 AgentThread
ANR9999D Thread<254> 0x00000000dfeaf6 StartThread
ANR9999D Thread<254> 0x0000327d807851 *UNKNOWN*
ANR9999D Thread<254> 0x0000327d0e890d *UNKNOWN*
ANR1411W Access mode for volume 000017L3 now set to "read-only" due to write
error.
ANR0515I Process 8 closed volume 000017L3.
 
We are working a very similar problem right now. What HBA are you running with, Qlogic, emulex...?
 
ruccsito said:
Hi guys.

I need some help with a problem. We are running TSM 6.3.3.200 on RHEL 6.4 with ProtecTIER TS7620.

I have a VTL defined with 8 drives and 100 cartidges, LIN_Tape latest version is installed on the Linux server.

Almost everything is working ok, the problem is when a backup is running a the virtual cartridge gets full, we get an I/O error on the tape, mark it as read only and mount the next tape.

Backups are completing ok, but in the morning I have the full tape with FILLING status and READONLY access.
Anyone else have this problem? This is a new implementation so this wasn't working before, I noticed it when the first tape got full.

Thank you guys!

Error details:
ANR8337I LTO volume 000017L3 mounted in drive DRV1 (/dev/IBMtape0).
ANR0513I Process 8 opened output volume 000017L3.
ANR8311E An I/O error occurred while accessing drive DRV1 (/dev/IBMtape0) for
WRITE operation, errno = 5, rc = 1.
ANR9999D_3130205688 ErrorHandler(pvrntp.c:8378) Thread<254>: Unexpected RC=1,
VOL=000017L3, DRIVE=DRV1 (/dev/IBMtape0)
ANR9999D Thread<254> issued message 9999 from:
ANR9999D Thread<254> 0x00000000d6d4e3 OutDiagToCons
ANR9999D Thread<254> 0x00000000d702e5 outDiagfExt
ANR9999D Thread<254> 0x00000000e581b5 ErrorHandler
ANR9999D Thread<254> 0x00000000e5ef70 DumpBlock
ANR9999D Thread<254> 0x00000000e60221 NtpWriteNC
ANR9999D Thread<254> 0x0000000096b509 AgentThread
ANR9999D Thread<254> 0x00000000dfeaf6 StartThread
ANR9999D Thread<254> 0x0000327d807851 *UNKNOWN*
ANR9999D Thread<254> 0x0000327d0e890d *UNKNOWN*
ANR1411W Access mode for volume 000017L3 now set to "read-only" due to write
error.
ANR0515I Process 8 closed volume 000017L3.
Click to expand...

Did this happen before (on an earlier TSM server version) using TSM 6.3.3.2?

If it did not, then there would be either a bug on TSM server 6.3.3.2 or on the lin_tape version. I highly doubt if the HBA version would be the case - but possible. Try updating the HBA firmware or drivers as needed.
 
Hi guys!, thanks for your quick answer.
We are running Emulex LP HBA's with the standard RHEL driver and the latest lin_tape driver (I also tried with a previous version). I will install the emulex driver today and see if it helps.
This is a fresh install, so we didn't have a previous TSM server, and I did had the problem from the beginning.
I will try the HBA driver/firmware update.. what do you think about upgrading the TSM to 6.3.4?? any other though is very welcome.
Thank you!.
Julio.
 
Last edited:
Hi again guys.

Tried upgrading the HBA drivers and is still not working.

Any idea is very welcome!

Ty!
 
ruccsito said:
Hi again guys.

Tried upgrading the HBA drivers and is still not working.

Any idea is very welcome!

Ty!
Click to expand...

All I could think of is that this maybe a TSM server issue at version 6.3.3.2. I am running 6.3.3.0 on RedHat 5.8 and I don't have this problem.

You may want to update to a newer TSM server version (I think there is one but not sure - too lazy to poke around!), or call IBM.

Also, this may well be a ProtecTIER issue. Do you have the latest compatibility matrix for the ProtecTIER and an updated firmware?
 
Are you using 6.3.3.0 with ProtecTier? Which version?

I just updated to TSM 6.3.4.0 and still no luck.. the error changed a bit, it doens't have the ANR9999D messages anymore..but is still setting the volumes to reado.

I think I am going for the PT upgrade from 3.2.2.0 to 3.2.4.0 and then..I am out of ideas :p
 
ruccsito said:
Are you using 6.3.3.0 with ProtecTier? Which version?

I just updated to TSM 6.3.4.0 and still no luck.. the error changed a bit, it doens't have the ANR9999D messages anymore..but is still setting the volumes to reado.

I think I am going for the PT upgrade from 3.2.2.0 to 3.2.4.0 and then..I am out of ideas :p
Click to expand...

No, I am not using the ProtecTIER. I have Data Domain.

How is your library setup? Is it set to Autolabel=yes, Relabel Scratch=yes? However, I don't think this really matters. I am just throwing in on what I have set it up on the Data Domain. These settings controls on how TSM hands-off tapes that goes from full to empty.

I think you should call IBM about this.
 
One other item that you can look at is how have you defined the Library for the VTL. Is it defined as SCSI or as VTL? If it is VTL try updating the library so that it is defined as SCSI and see if that helps at all.
 
Hi guys.

Thanks again for your help.. I did set autolabel and relabelscratch and also tried both, VTL and SCSI libtypes.
Unfortunately this is a new implementation and I don't have customer id yet..so no IBM support for the next few days.

I upgraded the PT to the latest version and that didn't help, so I decided to try if the problem was the RHEL server..I configured a windows box as storage agent, and guess what? from the Windows box tapes works just fine, once they reach end of volume the TSM server set them as full correctly.

So..the problem is on my RHEL.. I think I will keep playing with the lin_tape version..maybe try an older one.

Again thanks for your help guys!
 
Take a look in your messages file for a lin_tape entry with host_byte 0x07

Something like:
driver_byte 0x08, host_byte 0x07, msg_byte 0x00, driver_byte 0x02

If so this is an undefined error being returned by linux or the HBA to lin_tape. The current versions of lin_tape surface this to TSM as an I/O error.

Also as an aside RHEL 6.3 and 6.4 are not yet supported by lin_tape. Keep an eye on the system storage interop center for when support is added.
 
Hi Art!

That's exactly what's happening; here is /var/log/messages from lin_tape 2.0 without multipathing.

Jul 11 13:46:25 GCBA-TSM kernel: lin_tape: IBMtape10----63002 driver_byte 08, host_byte 07, msg_byte 00, status_byte 01
Jul 11 13:46:26 GCBA-TSM kernel: lin_tape: IBMtape11----63004 driver_byte 08, host_byte 07, msg_byte 00, status_byte 01
Jul 11 13:46:26 GCBA-TSM kernel: lin_tape: IBMtape2-----63005 driver_byte 08, host_byte 07, msg_byte 00, status_byte 01
Jul 11 13:46:53 GCBA-TSM kernel: lin_tape: IBMtape12----63006 driver_byte 08, host_byte 07, msg_byte 00, status_byte 01


So..there is nothing i can do? I had to donwgrade the RHEL version or wait for the driver to get updated? I don't think that's going to happen soon.

Thank you!
 
ruccsito,

Open a PMR against the protecTier/lin_tape once you get your account set up.

Alternately if you are feeling adventurous and are handy with C you could take a look at patching lin_tape yourself.
 
Thank you guys for all your help.

I will follow your advise Art, I am not that handy with C to fix the code myself :p.. but I will open a PMR and try to get some help from them, maybe they could patch it for me.

Thanks again!!
Julio.
 
Hey guys! .. Just wanted to keep you posted, I couldn't get any help from IBM.., I couldn't contact the right team to provide me support for the driver, so as a workaround I downgraded the RHEL 6.4 kernel to 2.6.32-220.el6.x86_64 (this one is shipped with RHEL 6.2) and now it's working correctly. Thanks again for all your help!!
 
Thanks for posting your solution.

I have an identical situation here.

I will try with what you've done.
 
ruccsito said:
Hey guys! .. Just wanted to keep you posted, I couldn't get any help from IBM.., I couldn't contact the right team to provide me support for the driver, so as a workaround I downgraded the RHEL 6.4 kernel to 2.6.32-220.el6.x86_64 (this one is shipped with RHEL 6.2) and now it's working correctly. Thanks again for all your help!!
Click to expand...

Now I see the picture - did not notice (or better yet - did not pay attention!) that you were running Red Hat 6.4. LIN_TAPE is not supported on Red Hat 6.4. The highest supported version is 6.2

However, it is strange that IBM DID NOT point this out to you. This is on the readme notes when you download lin_tape drivers for Red Hat.
 
Last edited:
Hey Flosebec!, let us know if that worked for you.

I noticed that the kernel wasn't support.., but I though i can give it a try.. and it didn't went well.. that's why I downgraded to 6.2 kernel.

About IBM.. after sorting all the entitlement issues.. I am getting support..even with the unsupported kernel.. they asked me for some tests during last week..and people from the lab are working on the driver..so I'll post again once I get more feedback.

Ty!
 
Last edited:
Hi,

the same with SLES11 SP3 (Kernel >3). SAP Backup breaks.

.. driver_byte 08, host_byte 07 and the other symptoms. I hope there is a new version next days. SLES 11 SP3 (and i think SP2 also affected) are here for half a year.
 
Hey Thom.
Try a lower version of the OS/kernel .. IBM is still working with my case..I got ProtecTIER 2nd level and lin tape devs onboard.
I think they are working with RHEL to review the problem.. so my suggestions would be to open a PMR to get help for your problem as well.
Let us know how it goes.
Ty!

thom7421 said:
Hi,

the same with SLES11 SP3 (Kernel >3). SAP Backup breaks.

.. driver_byte 08, host_byte 07 and the other symptoms. I hope there is a new version next days. SLES 11 SP3 (and i think SP2 also affected) are here for half a year.
Click to expand...
 
You must log in or register to reply here.

Data Privacy Impact Assessment

Compliance Assurance in Every Byte: Discover the Impact of Our DPIA Services.

Sponsor ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

Navigation Menu

Forums
   IBM TSM
   EMC Backup and Recovery
   Symantec NetBackup  
   SAN  
   NAS
   Platforms / OS
Mailing List Archives
   ADSM-L
User Groups

NordVPN 3 Months FREE

Help Support ADSM.ORG and Get NordVPN with 64% Off and 3 Months FREE.

NordVPN with 64% off + 3 months FREE

Forum statistics

Threads
31,971
Messages
135,441
Members
21,949
Latest member
tsheli
Back
Top