ADSM-L

Re: [ADSM-L] Reading tape without TSM

2011-06-20 06:27:33
Subject: Re: [ADSM-L] Reading tape without TSM
From: "Meuleman, Ruud" <ruud.meuleman AT TATASTEEL DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Mon, 20 Jun 2011 12:24:08 +0200
When a tape in the copypool isn't readable anymore, I will delete the
tape and there will be another created when the daily backup of the
primary pool is done. Here we have the problem that the tape in the
primary pool isn't readable too.

The last write date of the tape is 9th of June 2010. That is too long
ago for the activity log of TSM.
q v 050399JA f=d
     Approx. Date Last Written: 09-06-2010 12:34:01

The Tape Alert is enbabled on the TSM Library Manager and TSM Library
Client. A "move data 050399JA" gives no output for "q tapealertmsg" and
in the activity log there is no more data. Also in the errpt are no
errors of the tapedrive where volume 050399JA was mounted.

Only the already known output of the activitylog.
20-06-2011 09:57:05      ANR8302E I/O error on drive DRIVE7 (/dev/rmt7)
with volume
                          050399JA (OP=LOCATE, Error Number=5, CC=0,
KEY=08,
                          ASC=00, ASCQ=05,
SENSE=70.00.08.00.00.00.00.58.00.00.00.-
 
00.00.05.FF.0A.50.70.00.00.00.09.01.91.00.02.50.00.00.00-
 
.2B.00.11.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0-
 
0.05.00.00.00.00.00.00.00.00.03.72.00.01.27.27.24.E0.00.-
 
00.00.00.00.00.37.30.38.20.20.20.20.00.C0.41.4A.30.35.30-
                          .33.39.39.00.00.00.00.00.00.01, Description=An
                          undetermined error has occurred).  Refer to
Appendix C in
                          the 'Messages' manual for recommended action.
(SESSION:
                          49345, PROCESS: 137)
20-06-2011 09:57:05      ANR1146W Move data process terminated for
volume 050399JA
                          - excessive read errors encountered. (SESSION:
49345,
                          PROCESS: 137)


The last time the firmware was upgraded is the 4th March 2011.
D3I1_EC7.fmr for the 3592 E05 drives
D3I2_708.fmr for the 3592 E06 drives

The upgrade of Atape and atldd is a year ago.
Library Client
%lslpp -h Atape.driver atldd.driver
  Fileset         Level     Action       Status       Date         Time
 
------------------------------------------------------------------------
----
  Atape.driver   11.6.0.0   COMMIT       COMPLETE     07/30/09
09:11:11
  atldd.driver    6.7.7.0   COMMIT       COMPLETE     10/23/08
09:28:28

Library Manager
# lslpp -h Atape.driver atldd.driver
  Fileset         Level     Action       Status       Date         Time
 
------------------------------------------------------------------------
----
  Atape.driver   11.6.0.0   COMMIT       COMPLETE     06/21/10
10:53:00
  atldd.driver    6.7.7.0   COMMIT       COMPLETE     06/21/10
10:55:20

The damage is 43 lost files. The volume in the primary pool is still in
the TSM database, but readonly. There were 2 volumes in the copypool.
Those are deleted and checked out and not overwritten. I will other
tools, like tsmtape, to read the volumes.

Kind Regards,
Ruud Meuleman 

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Richard Sims
Sent: Friday, June 17, 2011 5:01 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Reading tape without TSM

Ugh.  The inability to get files back from the copy pool may indicate
that Backup Stgpool operations have not been ongoing or failed for some
reason.

Given these circumstances, you may have a TSM system which is no longer
viable.  I would suspend operations until you figure out what's going
on: further sessions may be writing new storage pool data which would
not be usable later.  Do 'Query Volume 050399JA Format=Detailed" to get
the last write date, and then begin inspection of your Activity Log,
expanding around that time, looking for problem indications.  Your
operating system log may have more info.  Hopefully, this is not a
shared library where TSM tapes are being overwritten by some other
application.  If device drivers or drive firmware were updated recently,
review that.  Turning on TapeAlert may help, going forward.  At this
point it's a matter of finding out how much damage there is, and
isolating a cause...

     Richard Sims

On Jun 17, 2011, at 10:38 AM, Meuleman, Ruud wrote:

> "move data" in several tapedrives, but the same message.
> 06-06-2011 15:23:01      ANR8302E I/O error on drive DRIVE2
(/dev/rmt2)
> with volume 050399JA
> 06-06-2011 15:54:34      ANR8302E I/O error on drive DRIVE5
(/dev/rmt5)
> with volume 050399JA
> 06-06-2011 16:16:52      ANR8302E I/O error on drive DRIVE8
(/dev/rmt8)
> with volume 050399JA
> 07-06-2011 12:51:00      ANR8302E I/O error on drive DRIVE9
(/dev/rmt9)
> with volume 050399JA
> 17-06-2011 15:13:12      ANR8302E I/O error on drive DRIVE7
(/dev/rmt7)
> with volume 050399JA
> 17-06-2011 15:17:32      ANR8302E I/O error on drive DRIVE4
(/dev/rmt4)
> with volume 050399JA
> 
> "restore v 050399JA p=y" has no succes:
> 17-06-2011 15:29:01      ANR1241I Restore preview of volumes in
primary
> storage
>                          pool ATL1 has ended.  Files Restored: 0, 
> Bytes
> Restored:
>                          0. (SESSION: 24832)
> 17-06-2011 15:29:01      ANR1256W Volume 050399JA contains files that
> could not be
>                          restored. (SESSION: 24832)
> 
> because of the corrupt volumes in the copypool, I think:
> 01-06-2011 08:30:32      ANR9999D_1336436977
> EndMoveVolume(afrclm.c:3475)
>                          Thread<2424571>: Reclamation terminated for 
> volume 060458
>                          - unexpected result code (25).(SESSION:
> 857538, PROCESS:
>                          3262)
> 30-05-2011 16:11:15      ANR9999D_1336436977
> EndMoveVolume(afrclm.c:3475)
>                          Thread<2397675>: Reclamation terminated for 
> volume 060556
>                          - unexpected result code (25).(SESSION:
> 843097, PROCESS:
>                          3172)
> 
> tapeutil is on the system, what actions do you advice?
> 
> Kind Regards,
> Ruud Meuleman
> 
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of Richard Sims
> Sent: Friday, June 17, 2011 3:46 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Reading tape without TSM
> 
> The first three records on a TSM tape are the Standard Labels, which 
> are in EBCDIC.  (See "Tape labels" in 
> http://people.bu.edu/rbs/ADSM.QuickFacts).
> 
> I would try Move Data in TSM, repeating on multiple drives if 
> necessary, to copy as much data as can be copied; then employ Restore 
> Volume (presuming it is a primary pool tape) to recover the rest, then

> retire the bad tape or perform further evaluations on it using 
> tapeutil.  It may not be the tape that is bad, per se: one drive may 
> be acting up, which may have affected a number of tapes: such tapes 
> might be quite usable with drives that are in good condition.
> 
>    Richard Sims
> 
> On Jun 17, 2011, at 9:32 AM, Meuleman, Ruud wrote:
> 
>> The tape where the copy is, isn't also readable.
>> 
>> It is possible to use the dd command. Coping blocks to file is 
>> possible, but I still read it.
>> 
>> # dd bs=256k count=20 if=/dev/rmt10 of=/tmp/cpp.out
>> 0+3 records in
>> 0+3 records out
>> 
>> # more cpp.out
>> M-eM-VM-SM-qM-pM-vM-pM-tM-uM-x@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> @
>> @@
>> @@@@@@@@@@@@@@@@@@@@@@@@@@@@M-HM-DM-YM-qM-AM-DM-bM-TKM-BM-FM-bKM-eM-p
>> M
>> -p
>> M-qM-pM-rM-B
>> M-A@@@@@@M-pM-pM-pM-qM-pM-pM-pM-q@@@@@@M-pM-pM-xM-sM-qM-t@M-yM-yM-sM-
>> v
>> M-
>> uM-pM-pM-pM-pM-pM-pM-pM-AM-DM-bM-T@@@@@@@@@@@@@@@@M-HM-DM-YM-rM-dM-rM
>> -
>> vM
>> -rM-qM-tM-pM-p
>> M-pM-pM-p@M-p@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> @
>> @@
>> @@@@
>> 
>> Should we skip some blocks with TSMtape, before we get to the actual 
>> data?
>> 
>> Kind Regards,
>> Ruud Meuleman
> 
> **********************************************************************
> 
> 
> This transmission is confidential and must not be used or disclosed by
anyone other than the intended recipient. Neither Tata Steel Europe
Limited nor any of its subsidiaries can accept any responsibility for
any use or misuse of the transmission by anyone.
> 
> For address and company registration details of certain entities 
> within the Tata Steel Europe group of companies, please visit 
> http://www.tatasteeleurope.com/entities
> 
> 
> ********************************************************************** 
  
**********************************************************************
 
 
This transmission is confidential and must not be used or disclosed by anyone 
other than the intended recipient. Neither Tata Steel Europe Limited nor any of 
its subsidiaries can accept any responsibility for any use or misuse of the 
transmission by anyone.
 
For address and company registration details of certain entities within the 
Tata Steel Europe group of companies, please visit 
http://www.tatasteeleurope.com/entities
 
 
**********************************************************************