library auding not succeeding with slot element error

[Patravane]

HI Tech members,
we are running tsm 5 realease 5 on windows server 2003, along with 3 stacked IBM libraries4560 SLX.
the issue is that we are not able to audit the library, getting the error:
ARN8912E unable to verify the label of volume from slot element 33 in drive 0 (mt0.0.0.1) in library 4560LB
ARN8912E unable to verify the label of volume from slot element 34 in drive 1 (mt0.4.0.1) in library 4560LB
ARN8912E unable to verify the label of volume from slot element 35 in drive 1 (mt0.5.0.1) in library 4560LB
can someone help me to fix this.
thanks in advance for your help

 

[moon-buddy]

Does this happen to only these three tapes? If yes, your tape has a label error or is defective. Are these LTO tapes? LTO tapes have chips in them that signals the library of its status. If these tape have data on them, move the data out (cancel the audit first), and toss the tapes out after the move.

If you cannot move the data, mark these as destroyed and recreate the data from the OFFSITE tapes (I am assuming that these are part of the online tape pool).

 

[Patravane]

HI Moon Budy
I ve removed tapes from those slots but to error is still occuring , pointing those slots again and again. I ve removed and check the related magazine from tape library to see if there is not a physical problem, but all looks fine there.
any suggestion?

 

[moon-buddy]

Is the error pointing to the slots or to the tapes?

 

[Patravane]

Dont really know. I ve now exchanged the magazines (swapped two magazines with one another) and the error occurs for the same location :ARN8912E unable to verify the label of volume from slot element 33 in drive 1 (mt0.5.0.1) in library 4560LB. not sure if it is related to the tape or the slot.
I am now suspecting my barcode reader or robotic arm. will exchange them see if it change things.

 

[jbiggers]

I assume you used the following TSM command:

YOURTSM> audit library 4560LB checklabel=yes

Here's the text of this error from the TSM 5.4 Messages manual (sorry I don't have the one for TSM 5.5).

ANR8912E Unable to verify the label of volume from slot-element (element-address) in drive (drive name) in library (library name).

Explanation: The volume from the slot that is identified by the element address is loaded into a drive. The drive cannot be opened following the load. The label cannot be read to process the volume.

System action: The volume is stored, and processing continues with the next available slot.

User response: Check the system error logs for errors reported by the drive that are related to this volume. Make sure that the cartridge is not a cleaner cartridge that would have been loaded by mistake. If the cartridge is a cleaner, issue the QUERY LIBVOL command to get the correct storage slot for the cleaner and move the cleaner to that slot. If the cleaner is not in the library’s inventory, use the CHECKIN LIBVOL to check the cleaner in. During Audit Library, remove the bad volume from the reported slot and run Audit Library again.

End of Message

The label that cannot be read is the magnetic label on the tape, not the barcode label. The fact that the robot can pick the tape and put it into the tape drive and put it back into the slot again, suggests this is a drive issue and not a robot/picker issue. However this involves multiple drives, which is weird, so the system error log might be helpful here. If the barcode reader couldn't read the tape barcode, I think you would get a different error as well. It's odd that the Library is fixated on these slots, and since you've changed the magazine already, it can't be the slots. Multiple slots and multiple drives being involved at the same time, suggests something different.

We have an older IBM 3583 tape Library, and it does weird things on occasion. USUALLY this Library has to be rebooted or power cycled to clear a strange error that doesn't otherwise make sense. OCCASIONALLY, IBM service has found some bad hardware (including the robot/picker) to fix a weird error. RARELY IBM service had to do an alignment/calibration of the robot/picker to correct a weird error. The cleaner cartridge solution is novel and could apply, but we've never had a cleaner in a regular tape slot before.

Hope this helps.

 

[Patravane]

HI JBIGGERS,
Thanks a lot for your contribution.
yeah that is the command we are running.
regarding robot/picker and barcode reader, you are right . I ve wrapped them all with the same error occuring in the same range of slots. I ve also disconnected middle library where the related slots were located, with no luck. the slots numbers reported, were now physically located to the third library while the same errors were reporting. now we are going to check in all our already used cleaning tapes and check them out again to see if it can fix the issue. Any other suggestions while waiting? thanks for your hepfull contribtutions in advance.

 

[jbiggers]

Patravane, You might check to see if the Library device configuration in both Windows 2003 and TSM 5.5 are correct for that Library. Something might have changed there.

 

[nalinsl]

can you run 'audit library' command after removing tapes from those slots successfully ? which means no errors from the mechanical point of view.also did you try to mount/dismount/checkin/checkout any tape from this library without any error ? If you get errors in that, which is most likely because some SCSI ID change.make sure those are same by running tsmdlst.exe and ' issuing 'q path library=<library name> f=d' command.

Good Luck !

 

[Patravane]

HI Nalinsh and others,
The audit command did not complete at all even when we isolated two libraries and remained just with the top master, where slots are numbered from 1 to 29, but the slot thirties are still appearing in the ARN8912E errors.
I ve also checked in all our cleaning tapes, and run the library audit, remove them one by one but this doesnt solve the problem.
we have succeeded to NT backup files into the three libraries tape drives. but tsm hangs on ARN8912E when trying to audit the library.
we are now waiting for another module and three tape drives to be shipped here from IBM service center. we will try those new elements to see if errors disappear.
thanks once more for your help

 

[jbiggers]

Some other things to look at:

1. Did anything configuration related change prior to the appearance of this error, such as a software or hardware update, or device driver change?
2. Is the Library at the latest firmware level? IBM service always asks that question.
3. Is your version of Tivoli licensed for the extended edition, which can use larger libraries with greater than 4 tape drives and 40 slots? Usually though Tivoli will run a larger Library, even if you're not licensed for it (at least the AIX version of Tivoli 5.5 does). It just reports a license audit failure.

Just some other ideas that might help.

 

[Patravane]

OK Guys, this is what we did to sort out the issue:
1- checked out all the volumes with the options checkl=yes while removing them manually from the mail slot: at this stage some tapes refused to come out . we then removed those tapes manually and store outside as suspected to have corrupted headers.
2- we have now checked in tapes that came out automatically by themself at stage 1, with the checkl=yes option.
3- we were now able to run the audit successfully.
now wonder how we can remove the data from those tapes as they cannot be checked in, thus cannot be mounted into drives.
someone has an idea here how we can recover those data from corrupted tapes? (suspected bad headers).
thanks a lot for your contributions.

 

[rmazzon]

Patravane,

Have you tried checking in one of the suspect tapes using the mail slot and with checklabel=yes? Did you get any errors? Do you have copy pools?

 

[jbiggers]

Complicated Answer

Patravane,

Somewhere along the way, did the configuration of the tape Library change? I stand to be corrected on this, but my understanding is that when the magnetic label is written to tape (LABEL command), it includes both the barcode (usually) and the Library identity the tape is labeled and used in. If the Library identity changed somewhere along the way, the tapes labeled earlier may not check in afterwards. The only way to recover the data from these tapes, so the tapes can be relabeled (which wipes out the data) and used again, is to recover the data from another online pool. In other words, you will have to empty the tapes before you can relabel them, to prevent data loss.

If these tapes contain copy pool data, then you can delete a tape volume (DISCARD=YES) and then run the BACKUP STGPOOL command and the same (unexpired) data will be read from the online primary pool and written back to another copy pool tape. Then the deleted tape can be labeled again with the LABEL command with OVERWRITE option (very dangerous option for data loss, use carefully). Then the tapes can be used again in the Library.

If the data on these tapes is primary pool data, then you'll need to have the copy pool data available. Use the RESTORE STGPOOL command, and change the access mode of these tapes to DESTROYED. The RESTORE STGPOOL command will then read the online copy pool data and write missing files from the "DESTROYED" tapes back to other primary pool tapes (assuming these are primary pool tapes). Then the now empty "DESTROYED" tapes can be relabeled and used again.

Hope this helps.

 

[rmazzon]

jbiggers,

We have on a number of occasions moved thousands of tapes between libraries. And we also used one library, with sufficient empty slots, to label new and used tapes with new barcodes for use in other libraries. All of the libraries were varying configurations of 3584. So if there is any library identification information within an internal tape label I have not seen it affect movement between libraries, at least of the same basic model.

 

[jbiggers]

rmazzon,

Nice to hear that. Several years ago (forget the TSM version then), we had to use a set of tapes from another TSM server and 3583, and they would not check in no matter what we did. The only way was to relabel the tapes, then they checked in. However that TSM server was not part of our TSM system. I assume your TSM servers are part of a server group with a primary library manager.

 

[rmazzon]

jbiggers,

We do have server groups and library managers and the tape movement between two different library mangers was the same as with movement between two libraries managed by the same library manager. I could foresee an issue if the tape drives in the two libraries had different native capacities (JA/JB/JC for example) and the devclasses controlling the format was something other than drives, say 3592 (J1A) trying to read JB media.

And of course encryption would play a huge factor if it was enabled and each library had different key managers or used different encryption methods.

 

[jbiggers]

Even though I've worked with TSM for 9 years, I have very little experience with checking in foreign tapes. It seems if TSM doesn't recognize the tape or the Library it was labeled in, it will reject the tape and not check it in. The only workaround I know of is to relabel the tapes, but that means no data can be imported from a foreign TSM library. I assume this is because the TSM database HAS to know what's on the tape, or it couldn't access the data. There may be data security issues with this as well. Funny though, I've never tried this with purely scratch tapes from a foreign TSM system, only data tapes, but that was years ago.

To relate this back to Patravane's original problem, if you use checkl=barcode when checking in tapes, then these tapes may later fail the Library audit if the barcode is recognized, but the labeling library is not recognized (i.e. there could be the same barcode in 2 completely different TSM installations) thus violating the data security model that TSM is built on (which I don't understand very well). All of this is only meaningful if the Library identification changed somewhere along the way, and the old Library identification is no longer recognized by TSM.

 

[Patravane]

Yeah we ve tried that and the error we got from the begining of this post came out again ARN8912E with now slot element 445 while the highest number of slot is 87.
Now we are going to open a ticket with IBM tivoli team so see there is a way out of this.

 

[Patravane]

yeah we ve tried to check the corrupt tapes one at time from mail slots but still getting ARN8912E with slot element 445, while the highest slot number is 87.
Also, nothing has been changed on this library. others tapes are "checkeable" and "mountable" whiles others are not. not sure this is related to configurations change. what I can assume is that there was a bad tape drive we have changed that had certainly damaged those tapes.
we are going to raise a ticket with the IBM Tivoli team to check if there is a way out of this issue.