LTO3 read errors - ANR8311E errno=78

[Plingy63]

Folks, I'm stuck. We recently installed a new LTO-3 [ADIC i2000] library with 6 LTO-3 drives. TSM was at 5.2.4. Backups from the TSM client seem fine... but when I subsequently try, for example, a MOVE DATA of that data to another tape, it just throws out constant "ANR8311E an I/O error occurred while accessing drive LTO12 (/dev/rmtx) for READ operation, errno=78" messages.
We upgraded the TSM fixpack level to 5.2.10.0...but the problem still persists.
I have defined the library as an Automated Library [SCSI]...I have defined the LTO device class as FORMAT=ULTRIUM. Still no joy.

The library is connected to an AIX host running AIX 5.3 ml2. We're using two hba's...going thru a Brocade 3800 to two blades at the back of the i2000. Each blade is connected to 3 drives...but with the first blade having the library autochanger mapped to it as well.

I realise TSM5.2 is out of support now, which is why I haven't logged a call yet... if anyone can think of anything I've setup incorrectly...or offer any advice that might lead to a solution, I'd be forever grateful. This is doing my head in! ;-)

 

[moon-buddy]

Only reason I could think of why this is not working is that the driver and the TSM version are not compatible with one another or 5.2 does not support LTO3 drivers.

I guess an update to 5.3.x or 5.4.x of TSM is needed.

 

[ewalker]

TSM 5.2 does support LTO3 drives we where running 5.2.6 for about a year on LTO3 drives before upgrading to 5.3.4

We have it set to:
Format: DRIVE

 

[Plingy63]

Eric... would you happen to know the name/version of the drivers we need? My AIX colleague tell me we're at ATAPE 9.3.3.0 and tivoli.tsm.devices.acsls is at 5.2.4.0. Does this seem reasonable? I'm still bemused why it writes data out...but refuses to read it back without errors. If we're at the right driver level, it should be good for both reads & writes I would have thought.

 

[ewalker]

You might see if there is an updated version since you upgraded to TSM 5.2.10.0 I am not sure when the drives where supported by TSM and at what level

can you post your drive info?

q drive f=d

Library Name: CP3584
Drive Name: CPDRIVE01
Device Type: LTO
On-Line: Yes
Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
RIUM2,ULTRIUMC,ULTRIUM
Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
RIUM2

 

[Plingy63]

Here's the drive info. I've taken them offline for now to prevent migration writing out to them... we've still got LTO1 to take that job on.LTO1 libr works fine...but our users are screaming coz they can't restore data that's somehow got to LTO3. :-(

Library Name: I2000
Drive Name: LTO10
Device Type: LTO
On-Line: No
Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTR
IUM2,ULTRIUMC,ULTRIUM
Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTR
IUM2
Element: 261
Drive State: EMPTY
Allocated to:
WWN: 500308C001E5001F
Serial Number: F001E5001F
Last Update by (administrator): PLINGA
Last Update Date/Time: 05/11/07 14:32:22
Cleaning Frequency (Gigabytes/ASNEEDED/NONE): 3,000

Library Name: I2000
Drive Name: LTO8
Device Type: LTO
On-Line: No
Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTR
IUM2,ULTRIUMC,ULTRIUM
Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTR
IUM2
Element: 259
Drive State: UNKNOWN
Allocated to:
WWN: 500308C001E50013
Serial Number: F001E50013
Last Update by (administrator): PLINGA
Last Update Date/Time: 05/11/07 14:29:05
Cleaning Frequency (Gigabytes/ASNEEDED/NONE): 3,000

 

[ewalker]

What does your Device class look like? you said that "I have defined the LTO device class as FORMAT=ULTRIUM" we have ours set to DRIVE

from help upd dev
FORMAT=
DRIVE -- The server selects the highest
format that is supported by the
drive on which a volume is mounted.
Attention: Avoid specifying DRIVE
when a mixture of drives is used
within the same library. For
example, do not use this option for
a library containing some drives
that support recording formats
superior to other drives.


Device Class Name: LTO1TAPE
Device Access Strategy: Sequential
Storage Pool Count: 1
Device Type: LTO
Format: DRIVE
Est/Max Capacity (MB): 204,800.0
Mount Limit: DRIVES
Mount Wait (min): 60
Mount Retention (min): 2
Label Prefix: ADSM
Library: FSR3584
Directory:
Server Name:
Retry Period:
Retry Interval:
Shared:
High-level Address:
Minimum Capacity:
WORM: No
Scaled Capacity:

Device Class Name: LTO3TAPE
Device Access Strategy: Sequential
Storage Pool Count: 1
Device Type: LTO
Format: DRIVE
Est/Max Capacity (MB): 819,200.0
Mount Limit: 10
Mount Wait (min): 60
Mount Retention (min): 2
Label Prefix: ADSM
Library: CP3584
Directory:
Server Name:
Retry Period:
Retry Interval:
Shared:
High-level Address:
Minimum Capacity:
WORM: No

 

[Plingy63]

My device class was originally set to DRIVE...but in one of many efforts to try and get this working, I changed it yesterday to ULTRIUM3C. Hasn't made any difference though.

[Ed.] Just a thought...since I updated my DEVCLASS, TSM hasn't been bounced. Should it be?

Device Class Name LTO3
Device Access Strategy Sequential
Storage Pool Count 8
Device Type LTO
Format ULTRIUM3C
Est/Max Capacity 800G
Mount Limit DRIVES
Mount Wait (min) 10
Mount Retention (min) 3
Label Prefix ADSM
Library I2000
Directory -
Server Name -
Retry Period -
Retry Interval -
Shared -
HLAddr -
Minimum Capacity -
WORM NO
Scaled Capacity -
Last Update by (administrator) PLINGA
Last Update Date/Time 2007-05-10 20:05:03.000000

 

[ewalker]

Do you have only LTO3 drives in the library?

Is there still some other versions of LTO drives in the library?

We use to get those errors when we where moving from LTO1 to LTO3 in one library and a LTO3 was mounted in one of my LTO2 tape drives.

 

[Plingy63]

We still have LTO1 drives in another library...that seems to have been unaffected. So, in effect we have two device classes ... "LTO" which is used for the LTO-1's in the original library ... and "LTO3" which I intended be used for the LTO-3's in the i2000 library. I based "LTO3" on the existing LTO class, merely changed the estimated capacity. But last night I updated it to change from FORMAT=DRIVE to FORMAT=ULTRIUM3C. As far as I'm aware, that's the only difference.

 

[ewalker]

What do the storage pools that are in the library look like.

Storage Pool Name: LTO3TAPE
Storage Pool Type: Primary
Device Class Name: LTO3TAPE

Storage Pool Name: FSRLTO1COPY
Storage Pool Type: Copy
Device Class Name: LTO1TAPE

do you still have LTO1 tapes in the new library? Or is it all LTO3 tapes?

 

[Plingy63]

All LTO3 tapes in the new library... no LTO-1's in there. Have various storage pools referencing Device Class Name LTO3...Storage pool type of primary: e.g.

Storage Pool Name: BACKLTO3
Storage Pool Type: Primary
Device Class Name: LTO3
Storage Pool Data Format: Native

 

[ewalker]

This one has me stumped, Everything looks normal and like it should.

 

[Plingy63]

Yeh..me too! :-( I've noticed that the LTO-3 drive firmware level is 64D0. Whether that's got anything to do with it I'm not sure. What what I've seen on the Internet, it looks like there's a leter firmware level - 69U2 - so it might be worth getting that applied. Other than that, there's only two courses open to me... one is to bypass the SAN/switch environment to see if that makes any difference...the other is to upgrade TSM from 5.2.10 to 5.3.x. And that latter option has me worried! Any tips on upgrading? ;-) Never ventured there before...we've always been 5.2.x

Cheers .. Paul.

 

[BBB]

Have you tried testing the drives without TSM being involved? You can use the tar or dd etc type commands to write data to a drive. If you give that a go (make sure you don't overwrite a TSM tape), and then read back the data, how does it go?

The "tapeutil" aix command may help you as well, if it works for your library.

Also, in your error message you have errno=78. In the AIX errno.h file this is defined as:

#define ETIMEDOUT 78 /* Connection timed out */

Which might suggest some kind of hardware/driver/driver config issue.

Is TSM definitely pointing at the right /dev/rmtX? What is the output from the AIX command "lsdev -Cc tape"? Although as you can write to the drives apparently its unlikely to be this.

Is there another error number just before the one you have posted in the actlog, for example a ANR8302E?

Is there anything in your AIX "errpt"?

Have you got IBM Ultrium Manufactured LTO3 drives installed in the library?

And have you checked the host's ports and tape drive ports on the SAN switches for high error counts? Write down what the error counts are, try the move data or an audit vol, and then compare the error counts. You can use the brocade porterrshow command.

 

[sharon]

Same Issue

Hello,

I have the same error - similar issue. Upgraded from TSM Server v5.2.x to TSM server v5.3.5.1. Changed the drives from SCSI attached LTO2 to Fibre connected LTO3. Now get the following error...

ANR8311E An I/O error occurred while accessing drive DRIVE1 (mt5.0.0.1) for
READ operation, errno=1117.

Have tried an upd path cmd. The cmd works but we still have the same error. Library is full of LTO2 tapes and now has only LTO3 drives. This should not cause an issue. Have the latest TSM drivers and tape firmware etc.

Any help appreciated.

Thanks,
S.

 

[Plingy63]

Sharon - what's your LTO-3 drive firmware level? Are they Vendor=IBM, Model=Ultrium TD3 ? That's what ours are... we're going to try ramping up from 64D0 to 69U2 or higher...see if that makes any difference.

Will let you know... meantime, if you get a fix, *please* post it here..

Cheers ...

 

[Plingy63]

BBB - yes, we've got IBM Ultrium-TD3 drives at firmare level 64D0.
As far as I know, we're mostly seeing ANR8311E msgs...although I think we have seen ANR8302E not long after we implemented the library. But since then we've gone from TSM 5.2.4.0 to 5.2.10.0 and bounced the library/AIX host a few times, & I've not seen 8302's since then. I've not tried the "Tapeutil" or tar method... I'll solicit the help of the AIX techie for that & post back. Another thought: Should I perhaps have chgd things like TXNGroupmax, MOVESizethres since using the LTO-3?? I haven't changed ANY TSM server options as yet... if it's a Timeout or size related problem only seen in TSM, might this be a reasonable thing to do to test?

Cheers .. PL

 

[sharon]

more info

My problem with this error message is on
Windows 2003 Service Pack 1
IBM Driver version 6.1.5.4
Tape drives are showing at Windows level as
IBM ULT3580-TD3 SCSI Sequential Device

IBM Support have asked that I upgrade the firmware on the tape
drives (two drives are at firmware v62D0 and one is at 69U2). I
don't believe this will help as I get the same error for all 3 drives.

They have also asked that I remove the drive and path from TSM and
from Windows for the tapes drives and the library, and then reboot
Windows so that it finds the drives and library again, and then
recreate the definitions in TSM. I am happy to do this - but don't
know that it will help.

I will keep you posted on how things pan out.........

S.

 

[sharon]

Fixed

Did anyone else find a fix for this issue.

We are using QLogic cards. There is a configuration option on these
HBA cards - optimize for tape. By default this is not ticked. Tick this
box (enable this option) and then the errors disappear.

Of course this means that your tape drives are not sharing a HBA port
with your disk. Seperate zone and HBA ports for the disk and the tape,
then optimize the HBA for tape - all fixed.

Thanks,
S.