ADSM-L

Re: [ADSM-L] TSM library problem caused by IBM3584 virtual I/O

2008-09-05 15:31:40
Subject: Re: [ADSM-L] TSM library problem caused by IBM3584 virtual I/O
From: Robert R Price <rprice28 AT CSC DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Fri, 5 Sep 2008 15:30:39 -0400
Hi John,

We are seeing similar issues, albeit on a much smaller library.

Two of our TSM Servers run with 3584 libraries on version 5.4.2.0.  The
nine LTO3 library does not have virtual I/O enabled and we see no problems,
but the ten LTO3 library with virtual I/O enabled does get the ANR8300E
errors that you documented.  The effect we see are tapes going into
"unavailable" status.

To correct the problem, we need to idle tape mounts, audit the library and
update the tape status to "read/write".  But it would be better to find the
root cause of the problem and correct it.

Robert R. Price
ADSM/TSM Administrator
Computer Sciences Corporation
Phone: 412-374-3247
Fax: 412-374-6371
rprice28 AT csc DOT com

This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery.
NOTE: Regardless of content, this e-mail shall not operate to bind CSC to
any order or other contract unless pursuant to explicit written agreement
or government initiative expressly permitting the use of e-mail for such
purpose.



             "Schneider, John"
             <John.Schneider@M
             ERCY.NET>                                                  To
             Sent by: "ADSM:           ADSM-L AT VM.MARIST DOT EDU
             Dist Stor                                                  cc
             Manager"
             <[email protected]                                     Subject
             .EDU>                     TSM library problem caused by
                                       IBM3584 virtual I/O

             09/05/2008 12:51
             PM


             Please respond to
             "ADSM: Dist Stor
                 Manager"
             <[email protected]
                   .EDU>






Greetings,
    We are running TSM 5.4.3.0 on AIX 5.3ML5.  We have a library master
instance, and 9 other TSM instances that are library clients.  They all
share an IBM3584 with 24 LTO4 tape drives, and an EMC EDL virtual
library emulating an IBM3584 with 128 LTO1 drives.

    Recently our IBM CE told us we should be running with virtual I/O, a
feature of the IBM3584 library.  The reason he recommended it is because
we frequently have more than 32 outgoing tapes every day, and sometimes
the Operators don't get around to taking the tapes out of the I/O doors,
and checkouts have to wait.  With virtual I/O turned on, the checkouts
go ahead and run to completion, even though the tapes don't actually go
into the I/O doors.  Then later when the I/O doors get empty, the tape
library moves the rest of the tapes into the I/O doors.  That part seems
to be working as expected.

    After we turned virtual I/O on, we started getting weird symptoms in
TSM, like tapes that we would check back in to the library, but later
TSM could not find them.  So we decided that maybe virtual I/O changed
the element number map, and we should have redefined the library to TSM.
So we:

1) Deleted the drive paths, drives, library path, and library on the
library master instance, and all client instances.
2) From the Tape library Web interface, performed a complete library
inventory (just in case)
3) Defined the library, library path, drives, and drive paths on the
library master instance, and all client instances.
4) Checked back in the scratch tapes
5) Checked back in the private tapes
6) Did an Audit library on the library master and all library clients.

It was only a few days later that we started getting errors from TSM of
the form:

09/04/08 22:00:56     ANR8300E I/O error on library SUN2079
(OP=00006C03,
                       CC=314, KEY=05, ASC=3B, ASCQ=0E,
SENSE=70.00.05.00.00.00-
                       .00.0A.00.00.00.00.3B.0E.00.C0.00.04.,
Description=The
                       source slot or drive was empty in an attempt to
move a
                       volume).  Refer to Appendix C in the 'Messages'
manual
                       for recommended action. (SESSION: 395703,
PROCESS: 487)
09/04/08 22:00:56     ANR8312E Volume 101781L4 could not be located in
library
                       SUN2079. (SESSION: 395703, PROCESS: 487)

09/04/08 22:00:56     ANR8358E Audit operation is required for library
SUN2079.
                       (SESSION: 395703, PROCESS: 487)

09/04/08 22:00:56     ANR8381E NAS volume 101781L4 could not be mounted
in drive
                       LTO4_F2_D09 (c576t0l0). (SESSION: 395703,
PROCESS: 487)
09/04/08 22:00:56     ANR1402W Mount request denied for volume 101781L4
- volume
                       unavailable. (SESSION: 395703, PROCESS: 487)

09/04/08 22:00:56     ANR1410W Access mode for volume 101781L4 now set
to
                       "unavailable". (SESSION: 395703, PROCESS: 487)

It is different tapes every time, so we now have over a dozen tapes that
are missing on account of this.  Did we do something wrong with we
turned on virtual I/O for this library?  I found this technote, that
sounds like it is supported.  It also says we need to restart the TSM
server instance, which we have done now a couple of times since this
problem started.

With SAN Device Mapping implemented (since TSM520 for Windows and TSM530
for most other platforms), when hardware changes are done to the
library, the Tivoli Storage Manager server needs to be restarted. During
server initialization, Tivoli Storage Manager server will access the
library. If the library inventory has changed, it will be refreshed at
that time. If drives are added or deleted from the library or drive
element addresses are changed, this information will be refreshed at
server initialization also. If the library path has changed, then the
path needs to be updated with the new device name for the new library
path. The same holds true when dealing with a 3584 library with the ALMS
feature. This is discussed in the following document :

http://www-1.ibm.com/support/docview.wss?uid=swg21053638

That document will mention that "TSM supports the Advanced Library
Management System (ALMS) and Virtual I/O Slots (VIOS) features of IBM
3584. When changing the number of drive, storage, or import/export
elements for a logical library, the TSM server must be restarted."

Is there something else I needed to do in TSM?  Some option in the
options file, or setting in the library?

Best Regards,

John D. Schneider
Lead Systems Administrator - Storage
Sisters of Mercy Health Systems
3637 South Geyer Road
St. Louis, MO  63127
Phone: 314-364-3150
Cell: 314-750-8721
Email:  John.Schneider AT Mercy DOT net


This e-mail contains information which (a) may be PROPRIETARY IN NATURE OR
OTHERWISE PROTECTED BY LAW FROM DISCLOSURE, and (b) is intended only for
the
use of the addressee(s) named above. If you are not the addressee, or the
person responsible for delivering this to the addressee(s), you are
notified
that reading, copying or distributing this e-mail is prohibited. If you
have
received this e-mail in error, please contact the sender immediately.