Running out of scratch tapes, lots of database backup tapes.

[ananke]

It seems that we have some issues where for some odd reason tsm cannot get scratch tapes.



Few things first:



Storage Pool Name VBITAPEPOOL

Storage Pool Type PRIMARY

Device Class Name LTOCLASS

Estimated Capacity (MB) 20815381.1

Pct Util 21.7

Pct Migr 36.0

Pct Logical 98.3

High Mig Pct 90

Low Mig Pct 70

Collocate? YES

Reclamation Threshold 51

Maximum Scratch Volumes Allowed 100



Looking at the Pct Utilization, we have plenty of space.



Now, q libvol shows something which I cannot understand. out of 140 tapes, over 60 have 'DbBackup' as their last use, only 20 or so have 'Data'. There are a few tapes with no last use, and there are no scratch tapes.



I've been trying to figure out what could possibly cause this issue. q db shows pct util only at 28.1.



The only other things that may matter: few months ago we ran out of the tapes, 40 more were purchased. Database grew too large, and we had to expand. Other than that, there have been no changes to the configuration, except adding a few nodes. TSM version is 5.1.



I would greatly appreciate any hints/suggestions.

 

[custew]

Are you using DRM? If so, check the setting for database expiration days. Run "q drmstat" and look at your output. You can use the "SET DRMDBBACKUPEXPIREDAYS" command to change this to whatever you want.



If you are not using DRM, then you can use "delete volhist" command. If you wanted to get rid of all the DB backups older than 7 days you would run this command..



"del volhist t=dbb todate=today-7"



Check the output of help on both the set and del volhist commands closely before running them so you are sure of what you are doing.



curtis

 

[ananke]

Yes, we're using DRM, and the output seems to be correct:

...

DB Backup Series Expiration Days: 7 Day(s)

Recovery Plan File Expiration Days: 30 Day(s)



However, I just noticed something very interesting:



03/04/05 16:42:18 ANR8463E LTO volume VT0122 is write protected.

03/04/05 16:42:44 ANR8381E LTO volume VT0122 could not be mounted in drive

RMT1 (/dev/rmt1).



Yet I know for a fact that volume has never been altered to be write protected. What could possibly cause that? The q vol shows:



q vol vt0122 f=d



Volume Name: VT0122

Storage Pool Name: VBICOPYPOOL

Device Class Name: LTOCLASS

Estimated Capacity (MB): 190,734.0

Pct Util: 91.8

Volume Status: Filling

Access: Unavailable

Pct. Reclaimable Space: 0.7

Scratch Volume?: Yes

In Error State?: No

Number of Writable Sides: 1

Number of Times Mounted: 21

Write Pass Number: 1

Approx. Date Last Written: 02/23/05 01:12:37

Approx. Date Last Read: 02/22/05 19:09:58

Date Became Pending:

Number of Write Errors: 0

Number of Read Errors: 0

Volume Location:

Last Update by (administrator): ADMIN

Last Update Date/Time: 03/04/05 15:05:06



It seems TSM has changed the status to unavailable. What could be the reason for that?

 

[themadkiwi]

Which library are you using does it keep tape read/write error logs...?



As a quick fix...



perform a manual check of the tapes and ensure no write locks are set.



clean all drives using 'CLEAN DRIVE lib_name drv_name' for each drive.



and only then issue:

Update vol * whereacc=unav acc=readw



this will make the tapes available again but you will need to monitor the logs to ensure that I/O errors etc arn't playing a part in this.



Hope it helps.

 

[klee]

ananke



Are you still having problems?



We had a problem recently where the number of data volumes (non-scratch, non-dbbackup) did not match the number of volumes shown with the q vol command. For some reason TSM had marked many tapes as unavailable even though they had not been used (about 40 out of 100tapes were affected). We corrected this by checking them out of the library and checking them back in as scratch (if you do not use scratch tapes and assign them to the pool as private you could use the update vol command to change access to readw.



If you have many dbbackup tapes. Confirm it with the q volh type=dbb command. Use the del volhist t=dbb todate=today-n to delete backups older than n days. The tapes holding these older backup versions should be scratched automatically.



Careful if you take dbbackup tapes offsite. Using the del volhist command loses all information about the tape and you have to ensure the tapes are brought back onsite and checked back into the library.

 

[ananke]

I forgot to check the forums in a week or so, and ironically, today we just fixed our problem. The solution IBM has given us was exactly the same as yours: it seems that we had a ton of database backups that were never expired. Running:



delete volhist type=dbbackup todate=today-7



has yielded 68 scratch tapes. I'm still not sure what has happened to our schedules, it seems something is missing from our DRM. We'll be investigating it further this week.



Big thanks to you and everybody on the forums for all the help.