ANR8311E error on IBM TS3310 Tape Library

[RajeshR]

Please suggest,

I got below error on my TSM server : W2K12 R2 TSM7.1.6
Tape Library TS3310 LTO4 Drives
This is the 2nd time i got these errors from last week and around 10 cartridges went to Read-Only state.
1. I cleaned each tape drive manually.
2. Recycled all Tape drives (power recycle)
3. Restarted Tape library
4. Nothing much found on system event logs.
5. Moving the data from these read-only tapes and checking them out of library.


When the below error occurs, if the job is backup, obviously backup job getting failed, but if it's TSM server process continuing with other tapes making current tape to read-only without any issues,

Please suggest what more can be done.

06/26/2017 09:03:29 ANR8311E An I/O error occurred while accessing drive
DRIVE2 (mt0.0.0.2) for GETPOS operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:03:29 ANR8311E An I/O error occurred while accessing drive
DRIVE2 (mt0.0.0.2) for OFFL operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:05:23 ANR8311E An I/O error occurred while accessing drive
DRIVE2 (mt0.0.0.2) for RELEASE operation, errno = 3, rc
= 2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:08:12 ANR8311E An I/O error occurred while accessing drive
DRIVE3 (mt0.0.0.4) for WEOF operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:08:12 ANR8311E An I/O error occurred while accessing drive
DRIVE3 (mt0.0.0.4) for OFFL operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:09:30 ANR8311E An I/O error occurred while accessing drive
DRIVE3 (mt0.0.0.4) for RELEASE operation, errno = 3, rc
= 2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:12:10 ANR8311E An I/O error occurred while accessing drive
DRIVE4 (mt1.0.0.4) for WEOF operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:12:10 ANR8311E An I/O error occurred while accessing drive
DRIVE4 (mt1.0.0.4) for OFFL operation, errno = 3, rc =
2863. (SESSION: 31202, PROCESS: 127)
06/26/2017 09:13:59 ANR8311E An I/O error occurred while accessing drive
DRIVE4 (mt1.0.0.4) for RELEASE operation, errno = 3, rc
= 2863. (SESSION: 31202, PROCESS: 127)

Please suggest.

 

[RajeshR]

My question is why the above errors logging at same time on all tape drives, if the tape cartridges are really faulty this might happen in different times.
Please suggest.

 

[DazRaz]

Can the operating system see the tape drives?

 

[RajeshR]

All tape drives are visible from Windows OS and all working properly, No events found for tape drives from windows event log,
Operations happening fine on TSM server but atleast once in a day above errors logging on tsm server and 1 r 2 tapes going to READ-ONLY state due to write error.

Following event logging during the error:

System

- Provider

[ Name] ADSMServer

- EventID 21

[ Qualifiers] 49152

Level 2

Task 8311

Keywords 0x80000000000000

- TimeCreated

[ SystemTime] 2017-06-27T17:27:27.000000000Z

EventRecordID 114256

Channel Application

Computer TSMserver-x.com

Security


- EventData

Server: TSMserver-x ANR8311E An I/O error occurred while accessing drive DRIVE4 (mt1.0.0.4) for RELEASE operation, errno = 3, rc = 2863.

 

[Trident]

Hi,

There should be something like a 'self test' or preferably a read/write test you can perform from the library. Either way, download log's from tape drives and open a PMR with IBM. They can have a look and find the root cause of the error.

-= Trident =-

 

[RajeshR]

Hello,

I have tried doing read/write test using ITDT and everything seems to be fine, and also i have backup jobs running through out night to tapes and all are successful, i have even tested FS&DB restores successfully, But few minutes ago when i am running backup stg pool(tape-tape) jobs i got same error on all tape drives which are being used.

Something needs to check from Switch end ?

 

[Trident]

Hi,
And all the usual stuff looked at:
- IBM tape driver is up to date
- Static binding for tape drives (q path/q drive match up with the correct drive/path)
- Speed towards drives is ok (4/8/16Gb) . You may have to lock it to a speed.
- Switch firmware is OK
- Zoning errors?
- Any errors on the library/drives?
- The tapes that go RO, are they old?

-= Trident =-

 

[RajeshR]

Hi,
- IBM tape driver is up to date >> C7QH for LTO4 tape drives. - i think this is the last update for LTO4.
- Static binding for tape drives (q path/q drive match up with the correct drive/path) >> yes verified.
- Speed towards drives is ok (4/8/16Gb) . You may have to lock it to a speed. >> it's 4GB (i think that's the lowest)
- Switch firmware is OK >> ( updated recently, not sure about version)
- Zoning errors? >> SAN team says no errors for assigned tape drive ports and all ports are online..
- Any errors on the library/drives? >> no errors, except the i/o station false alerts.
- The tapes that go RO, are they old? >> few are brand new and few are 1-2 yrs old.

 

[Trident]

Hi,
Not easy to find related info about this. Check FC errors (on sanswitches (look and see if the error counter is increasing), tape drives and host). Could also aslo look at the involved SFP's. Also, the IBM tape driver loaded on windows host should be checked (updated?).

Library path is mapped to only one drive (use static mapping)? If both drives are library paths, and you do not have a license for it, strange errors may occur.

I am running out of options.

Call IBM, and have them analyze the tape drive dumps. They can see more errors than we can.

Best of luck,

-= Trident =-

 

[RajeshR]

No luck to find any error on FC side, will ask CE to check SFP's.

IBM tape driver loaded on windows host should be checked (updated?). > will check this.

Physical library having 2 logical libraries for 2 TSM servers, hence having 2 control paths, Not sure whether i have licence for it or no but using this since a year without any issue.

Call raised with IBM waiting for CE, Will see after sending drive logs, if IBM wants me to use only one control path, i will delete one logical library for old TSM server(where it's only taking old tsm server dbbackup)

I have seen a technote for this kind of issues, there its mentioned not to use more than one control path, but it's RC is different from mine, so i thought it's ok.

Thanks for great help and advice, will post the resolution once it's got resolved.

 

[RajeshR]

Tape library licence info:
Feature :
Capacity on demand - Activated
Transparent Encryption - Activated
Path Failover - Activated

 

[Trident]

Hi,
Check your library config. Look for shared drives (vs. dedicated drives). I always use dedicated drives per library.

Very curious about root cause in this matter.

-= Trident =-

 

[RajeshR]

No shared drives, out of 5 drives 1 tape drive using for logical library(this drive is control path), 4 tape drives using another logical library out of 1 drive is control path.
Things are getting worse, last 1 hr there r around 2k read errors occurred on 3 tapes while running space reclamation but operation isn't failing and completed. strange