Tapes/Volumes Disappear

[mikeatkc]

My TSM environment recently ran out of scratch tapes quickly. Hence, I did a manual inventory, query the media for offsite, tapedata, etc. print a list and compares it with actual media, and found 24 offsite tapes that shouldn't be offsite. The tapes were brought back and check in as scratch.

However, I can only check in 4 tapes and TSM rejected the other 20 tapes with message that it cannot check those volumes in as scratch. I did a query, select, etc. on the volumes and they are no where in the storage pool.

- Why are these volumes disappear in the TSM storage pool?
- Do I need to have TSM checkin these volumes as scratch w/unlabel? to re-label the volumes.

TSM 5.5.2.0 on AIX (previously on 5.2.2.0 and wondering about any bug in 5.2 that causes volumes to disappear, ie. http://adsm.org/forum/showthread.php?t=2701). And no del volhist before DRM generated.

Thanks.


Mike

 

[JohanW]

Can you check them in as private? If so, they may still contain data. Or is the problem that you don't have more than 4 I/O ports? What was the error message? (And what was the command issued?)

Are the 20 volumes named anywhere in the volume history? Last entry should be STGDEL.

Q LIBV and Q DRM should tell you what TSM thinks is in the library and offsite. Everything else should be empty and should be scratchable.

 

[mikeatkc]

From the VOLHIST, it seems these volumes are BACKUPFULL (full db backup). However, the setting to keep full db backup is 5 days, so these tapes should have been updated to return from the offsite as scratch?

2009/03/31 13:23:50 BACKUPFULL 2324 0 1 LTO-GEN2 A00113 ...

I used the following command to check-in the tapes:
checkin libv ltolib1 status=private search=bulk

Do I just need to update these volumes in the volhist with status STGDELETE? to check them in as scratch?

update volhist VOLNAME ...

Shouldn't TSM done this a while back? (ie. 3/31 -> 4/23, less 5, 18 days ago).

 

[DanGiles]

Do a "q drmstatus" and see what the "DB backup series experation days" is set to.
In your scripts/admin schedules, do you do a "del volhist type=dbback todate=-XX"?
If the "XX" is less than the setting in the drmstatus, you will "lose" tapes, just as you are experiencing now. (I've seen this happen more than once.)

 

[mikeatkc]

The "db backup series expiration " is set at 5. I'm not seeing any "del volhist" from our daily script or schedule, etc. Could this be the cause, that I need to setup a "del volhist type=dbback todate=-6" somewhere on daily basis?

 

[DanGiles]

You don't NEED to do a "del volhist..." (though it's good to clean out this file every now and then). If you DO have "del volhist type=dbb todate=-X", where X<=5, that would explain what you're seeing. Look carefully, because it could be hidden anywhere! (Search your actlog for this command.)

If this is NOT the case, then it would need more investigation

 

[mikeatkc]

I searched the actlog for the past five days, no "del volhist" anywhere. I need to insert this into my daily script somewhere. But for now I think I should run the command manually to get these tapes updated and check in?

 

[mikeatkc]

tsm>delete volhist type=dbbackup todate=-5

Do you wish to proceed? (Yes (Y)/No (N)) Y
ANR2467I DELETE VOLHISTORY: 73 sequential volume history entries were successfully deleted.

Now what? How long until TSM reports/updates these volumes as scratch for retrieval?

 

[Jacob_6]

They normally return to scratch right after the command has run.

 

[mikeatkc]

tsm>q drmedia * wherestate=vaultretrieve
ANR2034E QUERY DRMEDIA: No match found using this criteria.
ANS8001I Return code 11.

 

[JohanW]

Correct.

If you use DRM, DO NOT del volhist. Use expiration. If you del volhist, you do not get any warning, and have no way to generate a list of which tapes to return from vault, short of printing out Q DRMEDIA WHERESTATE=VAULT and retrieving everything that's not on that list.

 

[mikeatkc]

OK, I think I sort of got it...

1) If I use DRM the expiration process should have taken care of this.

If I don't use DRM, do #2 on daily basis.

2) Daily command doing: DEL VOLHIST TYPE=DBBACKUP TODATE=-X


I am using DRM and expiration process ran on daily basis (q proc below). However, I'm not seeing any backup volumes expired, the below "0 DB backup volumes" portion.

915 Expiration
Examined 2594920 objects, deleting 41401 backup objects, 0 archive objects, 0 DB backup volumes, 0 recovery plan files; 0 errors encountered.


Below is from the "help expire" but my TSM doesn't seem to do it...

When you have the disaster recovery manager function for your Tivoli
Storage Manager server, the inventory expiration process also removes
eligible virtual volumes that are used for the following:
* Database backups of type BACKUPFULL, BACKUPINCR, and DBSNAPSHOT. The
SET DRMDBBACKUPEXPIREDAYS command controls when these volumes are
eligible for expiration.
* Recovery plan files of type RPFILE and RPFSNAPSHOT. The SET
DRMRPFEXPIREDAYS command controls when these volumes are eligible for
expiration.

 

[mikeatkc]

Here's my DRMSTATUS. DB expires set to 5 days, DRM file to 30 days and DRM files are not being remove either.

tsm>q drmstatus

Recovery Plan Prefix: /var/tsmdoc/TSM
Plan Instructions Prefix: /var/tsmdoc/TSM
Replacement Volume Postfix: @
Primary Storage Pools:
Copy Storage Pools:
Not Mountable Location Name:
Courier Name:
Vault Site Name:
DB Backup Series Expiration Days: 5 Day(s)
Recovery Plan File Expiration Days: 30 Day(s)
Check Label?: No
Process FILE Device Type?: No
Command File Name:

 

[Jacob_6]

If they the recovery plans are being stored locally, you have to manually remove them from the file system they are stored on. I go in every ten days or so to delete mine, to prevent the file system from filling up.

 

[mikeatkc]

I'm down to three scratch tapes. What can I do to immediately set the outdated volumes (> 5 days) with DBBACKUP (full/incremental) to scratch status? UPDATE VOL command?

 

[mikeatkc]

Yes, I setup the DB incremental to go to disk, due to limited scratch tapes. I already did the DEL VOLHIST TYPE=BACKUPDB TODATE=-5 earlier.

tsm:>q volhist type=dbb

       Date/Time: 04/20/09   16:53:35
     Volume Type: BACKUPINCR
   Backup Series: 2,343
Backup Operation: 2
      Volume Seq: 1
    Device Class: TEMPFILES
     Volume Name: /misc/40264415.DBB
 Volume Location:
         Command:

       Date/Time: 04/21/09   16:58:57
     Volume Type: BACKUPINCR
   Backup Series: 2,343
Backup Operation: 3
      Volume Seq: 1
    Device Class: TEMPFILES
     Volume Name: /misc/40351137.DBB
 Volume Location:
         Command:

       Date/Time: 04/22/09   16:54:03
     Volume Type: BACKUPINCR
   Backup Series: 2,343
Backup Operation: 4
      Volume Seq: 1
    Device Class: TEMPFILES
     Volume Name: /misc/40437243.DBB
 Volume Location:
         Command:

       Date/Time: 04/23/09   12:42:21
     Volume Type: BACKUPFULL
   Backup Series: 2,344
Backup Operation: 0
      Volume Seq: 1
    Device Class: LTO-OFF
     Volume Name: A00208
 Volume Location:

         Command:

       Date/Time: 04/23/09   16:58:36
     Volume Type: BACKUPINCR
   Backup Series: 2,344
Backup Operation: 1
      Volume Seq: 1
    Device Class: TEMPFILES
     Volume Name: /misc/40523916.DBB
 Volume Location:
         Command:

       Date/Time: 04/24/09   11:20:53
     Volume Type: BACKUPFULL
   Backup Series: 2,345
Backup Operation: 0
      Volume Seq: 1
    Device Class: LTO-OFF
     Volume Name: A00232
 Volume Location:
         Command:

 

[mikeatkc]

It works. I did the DELETE VOLHIST TYPE=DBBACKUP TODATE=-5 and then have to manually go through the expired volumes. I now have 16 scratch tapes. Anyhow, due to limited scratch tapes I set the DB incremental to file, resetting that now.

Thank you all for your help.


Mike

tsm>select count(*) from libvolumes where status='Scratch'

 Unnamed[1]
-----------
         16