DEVCLASS and Tape Volume Issue - need assistance

[buster99]

Hello, we have stumbled across an issue regarding tape volumes that are in the wrong devclass and we have not been able to find a good solution so far. We were hoping that we could get assistance here. I will try to describe our issue as briefly as possible. Our tape library contains 5286 tapes and we have many more in onsite and offsite storage. While trying to determine which tapes I could remove from the library to add scratch tapes, I found that we have tapes that were linked to the wrong devclass both onsite and offsite.

We have 2 TSM instances. A library manager (TSMLM) and a client (TSM16). Most of our nodes (Windows/AIX) backup on TSM16. The TSMLM contains our NDMP/NAS backups.
I was finding tapes that were backups of Windows nodes on TSM16 that were linked to the NAS devclass on TSMLM. And there are a lot of them ! Over 2000 tapes.
It looks like this incorrect setting doesn't allow the tapes to be picked up as they normally would during our reclamation and migration processing. They are in limbo with most of them being untouched since late 2013. It looks like something happened in 10/2013 that caused this issue. I see a NAS devclass update at that time.

I tried simple updates on the volumes but nothing would change the devclass associated to the volumes. I have an open IBM PMR but we haven't gotten anywhere with that either. Their latest suggestion was to perform a "movedata" on the tapes. I would like to avoid doing a movedata on 2000+ tapes.

I will put samples below of our settings and a volume sample. Any recommendations would be appreciated. Please let me know if any more info is needed. Thank you

Storage Management Server for AIX - Version 7, Release 1, Level 4.0

TSMLM:
tsm: TSMLM>q devclass NAS

Device Device Storage Device Format Est/Max Mount
Class Access Pool Type Capacity Limit
Name Strategy Count (MB)
--------- ---------- ------- --------- ------ -------- ------
NAS Sequential 2 NAS DRIVE 819,200. DRIVES
0
tsm: TSMLM>q devclass NAS f=d
Device Class Name: NAS
Device Access Strategy: Sequential
Storage Pool Count: 2
Device Type: NAS
Format: DRIVE
Est/Max Capacity (MB): 819,200.0
Mount Limit: DRIVES
Mount Wait (min): 60
Mount Retention (min): 0
Label Prefix: ADSM
Library: LTO5LIB
Directory:
Server Name:
Retry Period:
Retry Interval:
Shared:
High-level Address:
Minimum Capacity:
WORM: No
Drive Encryption:
Scaled Capacity:
Primary Allocation (MB):
Secondary Allocation (MB):
Compression:
Retention:
Protection:
Expiration Date:
Unit:
Logical Block Protection:
Last Update by (administrator): FMALVEST
Last Update Date/Time: 10/09/13 13:08:54

tsm: TSMLM> select VOLUME_NAME, DATE_TIME, TYPE,DEVCLASS,VOLUME_NAME,LOCATION from VOLHISTORY where VOLUME_NAME='017581L4'

VOLUME_NAME: 017581L4
DATE_TIME: 2012-05-29 10:10:19.000000
TYPE: REMOTE
DEVCLASS: NAS
VOLUME_NAME: 017581L4
LOCATION: TSM16


TSM16

tsm: TSM16> select VOLUME_NAME, DATE_TIME, TYPE,DEVCLASS,VOLUME_NAME,LOCATION from VOLHISTORY where VOLUME_NAME='017581L4'

VOLUME_NAME: 017581L4
DATE_TIME: 2012-05-29 10:10:20.000000
TYPE: STGNEW
DEVCLASS: LTO4
VOLUME_NAME: 017581L4
LOCATION:
tsm: TSM16>q vol 017581L4 f=d
Volume Name: 017581L4
Storage Pool Name: ONSITE_TP
Device Class Name: LTO4
Estimated Capacity: 789.7 G
Scaled Capacity Applied:
Pct Util: 99.6
Volume Status: Full
Access: Read/Write
Pct. Reclaimable Space: 0.4
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 4
Write Pass Number: 1
Approx. Date Last Written: 06/26/12 09:29:31
Approx. Date Last Read: 06/27/12 17:28:55
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 05/29/12 10:10:20
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: Library
Logical Block Protected: No

 

[buster99]

Hi!

I was reading your post with interest, when I took a look at my own environment. We implemented an Isilon NAS a few months ago, and as part of that, we implemented NDMP backups on one of our TSM clients with a NAS devclass, and a corresponding NAS device class on the library manager. In the volume history, ALL OF THE VOLUMES IN THAT LIBRARY are shown on the library manager as being associated with my NAS devices class (even those data volumes that are members of client storage pools that are very clearly defined as LTO4 pools.

HORROR OF HORRORS!

I think this is just a case of TSM defaulting the device class to the one that has the highest writing format. You'll notice that your NAS device class (and mine!) has a format of "DRIVE". This means that TSM will use the highest format that is supported by the drive upon which a volume is mounted.

I don't think that this has anything at all to do with your ability to move volumes offsite or onsite. This is just volumehistory.

Can you prove that the volumes are not being migrated/reclaimed because of their identification in the library managers volume history as being a member of the NAS devclass?

Hello, Thank you for the reply. I will try to find and example of reclamation/migration not working because of the devclass setting. Thanks

 

[buster99]

Hello mikeymac, sorry for the delay getting back to you. Have been working on some other issues. I took a look at this when you replied but I could not find a specific example of one of my tapes being skipped by a reclaim/migrate process. I will try again soon...hopefully. I was leaning that way originally because I have so many tapes that have available space on them that are in the library but these tapes haven't been written to in years. I thought the "devclass" setting was preventing them from being called by newer processes.

Here is an tape example :
tsm: TSM16>q vol 017106L4 f=d

Volume Name: 017106L4
Storage Pool Name: ONSITE_TP
Device Class Name: LTO4
Estimated Capacity: 856.3 G
Scaled Capacity Applied:
Pct Util: 67.7
Volume Status: Full
Access: Read/Write
Pct. Reclaimable Space: 39.7
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 26
Write Pass Number: 1
Approx. Date Last Written: 05/13/12 12:20:28
Approx. Date Last Read: 08/19/14 15:27:29
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 04/28/12 10:55:39
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: Library
Logical Block Protected: No

 

[buster99]

Hello Mikeymac, how about this example...

I run this sql on my library manager, TSMLM...
tsm: TSMLM>select VOLUME_NAME, DATE_TIME, TYPE,DEVCLASS,VOLUME_NAME,LOCATION from VOLHISTORY where LOCATION='TSM16' and DEVCLASS='NAS'

VOLUME_NAME: 017170L4
DATE_TIME: 2012-05-04 06:27:13.000000
TYPE: REMOTE
DEVCLASS: NAS
VOLUME_NAME: 017170L4
LOCATION: TSM16 (TSMCLIENT)

On the TSM client, TSM16, here is the volume info and the results of running "run recinfo" against the ONSITE_TP stgpool...
Since the volume is only 2.6 % used, shouldn't I see it when I do "run recinfo" ? Doesn't seem to be available for me to reclaim.

tsm: TSM16>q vol 017170L4 f=d
Volume Name: 017170L4
Storage Pool Name: ONSITE_TP
Device Class Name: LTO4
Estimated Capacity: 1.5 T
Scaled Capacity Applied:
Pct Util: 2.6
Volume Status: Filling
Access: Read/Write
Pct. Reclaimable Space: 0.0
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 223
Write Pass Number: 1
Approx. Date Last Written: 01/06/14 07:33:39
Approx. Date Last Read: 04/17/16 02:06:54
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 1
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 06/19/14 15:41:56
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: Library
Logical Block Protected: No

tsm: TSM16>run recinfo
ONSITE_TP 025467L4 75.1
ONSITE_TP 020440L4 75.6
ONSITE_TP 026469L4 76.0
ONSITE_TP 018225L4 76.0
ONSITE_TP 022451L4 76.8
ONSITE_TP 024664L4 76.9
ONSITE_TP 025626L4 77.3
ONSITE_TP 025480L4 77.8
ONSITE_TP 026495L4 77.9
ONSITE_TP 014151L4 79.7
ONSITE_TP 020894L4 80.0
ONSITE_TP 020330L4 80.2
ONSITE_TP 019961L4 80.9
ONSITE_TP 018927L4 82.2

 

[buster99]

ok...make sense but...

This specific tape has been in the library since the last time it was written to (01/06/2014) and it hasn't been used...seems odd since it is only 2.6 % utilized...no ?
Shouldn't TSM be trying to fill it up instead of using other onsite tapes first ? I just get the feeling that I have tapes in the library that have plenty of available space on them, but because they have this incorrect devclass on them, they get overlooked and sit in the library taking up space and costing us $ in new tape purchases. We back up tons of data every night....why would this tape not be written to in over 2 years ?

Approx. Date Last Written: 01/06/14 07:33:39
Approx. Date Last Read: 04/17/16 02:06:54

Just an fyi...here is the run recinfo script details...we use it to determine the number of volumes each stgpool has available to reclaim and how we want to set the "rec pct" to get them processing...

tsm: TSM16>q script recinfo f=l

Name Line Command
Number
---------- ------ ------------------------------------------------------------
RECINFO 1 /* -------------------------------------------------*/
5 /* Script Name: recinfo */
10 /* Description: Display volume reclaimation for stg */
15 /* pools and list the volumes */
20 /* Parameters : None */
25 /* Example: run recinfo */
30 /* -------------------------------------------------*/
35 select substr(char(STGPOOL_NAME),1,16) as stgpool, -
40 count(*) as reclaim_volumes -
45 from volumes where PCT_RECLAIM>75 group by STGPOOL_NAME
50 select substr(char(STGPOOL_NAME),1,16) as stgpool, -
55 substr(char(RECLAIM),1,8) as REC_PCT from stgpools
60 select substr(char(STGPOOL_NAME),1,16) as stgpool, -
65 substr(char(VOLUME_NAME),1,10) AS VOLUME, -
70 PCT_RECLAIM from volumes where PCT_RECLAIM>75 order by -
75 STGPOOL_NAME,PCT_RECLAIM

more script results I didn't include earlier...
STGPOOL RECLAIM_VOLUMES

---------------- ----------------
DIRMCOFF_TP 16
OFFSITE_TP 270
OFFSIT_VMDK 13
OFFSIT_VMX 22
ONSITE_TP 14
ONSITE_VMDK 45

STGPOOL REC_PCT
----------------- ---------
DIRMCOFF_TP 100
DIRMCSTGP
OFFSITE_TP 100
OFFSIT_ARC 100
OFFSIT_VMDK 100
OFFSIT_VMX 100
ONSITE_ARC 90
ONSITE_DSK
ONSITE_DSK2
ONSITE_TP 100
ONSITE_VMDK 100
ONSITE_VMDK_DSK 100
ONSITE_VMX 60
ONSITE_VMX_DSK 80