Tape is filling but no content / files on the tape

[bdgree]

TSM version 5.4 level 3.1. It is not supported but we have AIX 4.3.3 nodes and the old NT Windows.
We have 11 LTO3 tape in the copystorage pool status filling 0% utilization with no files on it.
tsm: ZTMTSM1>q vol QW5704L3 f=d
Volume Name: QW5704L3
Storage Pool Name: BCP_ZTMTSM1_ZSTLIB1
Device Class Name: LTO3_ZSTLIB1
Estimated Capacity: 409,6 G
Scaled Capacity Applied:
Pct Util: 0,0
Volume Status: Filling
Access: Read/Write
Pct. Reclaimable Space: 0,0
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 3
Write Pass Number: 1
Approx. Date Last Written: 28-08-2013 15:21:08
Approx. Date Last Read: 30-08-2013 08:46:20
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): ADMIN
Last Update Date/Time: 30-08-2013 11:50:03
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager:

tsm: ZTMTSM1>q cont QW5704L3
ANR2034E QUERY CONTENT: No match found using this criteria.
ANS8001I Return code 11.

When we audit the tape there are files on it:

30-08-2013 08:46:04 ANR0984I Process 48 for AUDIT VOLUME (REPAIR) started in
the BACKGROUND at 08:46:04. (SESSION: 19594, PROCESS: 48)
30-08-2013 08:46:04 ANR2312I Audit Volume (Repair) process started for volume
QW5704L3 (process ID 48). (SESSION: 19594, PROCESS: 48)
30-08-2013 08:46:21 ANR4132I Audit volume process ended for volume QW5704L3;
840 files inspected, 0 damaged files deleted, 0 damaged
files marked as damaged, 0 objects updated. (SESSION:
19594, PROCESS: 48)

When we scratch the tape, (with discard contents) the next day with the houskeeping a new scratch tape in the copypool is used because of the difference between Primary and copy storage pool
So we have again a tape back with no contents on it with: q cont <volume> but with audit the there are files on it.
We have the same on about 11 tape.


How can we see what is on the tape?
How is this possible?
How can we get rid of it?

 

[LED888]

The LTO3 tapes, are these new tapes or have they been used by the TSM Server?
If they are new tapes, need to run the lable libv to table the tapes.
Remove these tapes from the library, from the TSM Server issue an audit library.
Place the tape in the I/O slot and run the label libv command with the overwrite=yes parameter.

Good Luck,
Sias

 

[TiborB]

You mean you have 11 FILLING tapes in that copy pool with no content?
If you have only 1 such tape, then I see no problem here. It seems there are some (uncompleted) chunks of files.

It is possible to find out where a specific file is saved (on which tapes - if spread on multiple tapes), but I dont know (and would like to learn) how can be the reverse found. That means if you have a tape and want to find out to what files chunks on the tape belongs.

 

[bdgree]

The tapes are not new, they are used a long time in this TSM server. Also the data is in the primary tapepool because if you discard the tape in de copy tapepool the command ba stg <primary tape pool> <copy tape pool> use a new scratch tape in the copy tape pool. for all the 11 tapes.

 

[TiborB]

so right now you have 11 (or more) filling tapes in your <copy tape pool>?
And every time you run 'ba stg' a new scratch tape is grabbed and put into the pool?


Did you try 'move data' on any of those 'empty' tapes?

 

[bdgree]

We have more tapes in the pool but 11 tapes have this "problem" . No not everytime I run ba stg but only if i scratch one "problem" tape with his contents in the copy tape pool.
If I try to move the data on the tape in the same pool to another tape he takes always a new scratch tape and the tape is filling 0% and no data on it again. the old tape comes in pending and after the reuse delay scratch. I have the problem tapes now excluded from the reclaiming because else every day the reclaime process is moving data from 11 tapes to 11 new scratch tapes en the old ones are again pending

 

[TiborB]

hm.....

can you post result of 'q stg BCP_ZTMTSM1_ZSTLIB1 f=d', perhaps there will be some hint....

 

[bdgree]

tsm: ZTMTSM1>q stg BCP_ZTMTSM1_ZSTLIB1 f=d
Storage Pool Name: BCP_ZTMTSM1_ZSTLIB1
Storage Pool Type: Copy
Device Class Name: LTO3_ZSTLIB1
Estimated Capacity: 220.881 G
Space Trigger Util:
Pct Util: 30,4
Pct Migr:
Pct Logical: 99,4
High Mig Pct:
Low Mig Pct:
Migration Delay:
Migration Continue: Yes
Migration Processes:
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold:
Access: Read/Write
Description: Copypool ZTMTSM1 in ZSTLIB1
Overflow Location:
Cache Migrated Files?:
Collocate?: Group
Reclamation Threshold: 69
Offsite Reclamation Limit: No Limit
Maximum Scratch Volumes Allowed: 300
Number of Scratch Volumes Used: 134
Delay Period for Volume Reuse: 1 Day(s)
Migration in Progress?:
Amount Migrated (MB):
Elapsed Migration Time (seconds):
Reclamation in Progress?: No
Last Update by (administrator): ADMIN
Last Update Date/Time: 02-09-2013 12:00:09
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?:
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:

 

[TiborB]

might the problem be here: Collocate?: Group
We do not use collocation and I have no practical experiences with this, but it might behave the way that data from a single node are put to a separate tape, and this might explain why the content is forcefully spread over multiple tapes... instead on single tape

 

[bdgree]

We are using collocation for a long time and never seen this problem before. I think if there are splitted files on the tape, then if we are using reclamation on the "problem" tapes the data must go to another tape in the collocation group and not always on a new scratch tape or am i wrong? On the moment of reclamation there are other filling tapes (read/write) for the collocation group available.
But if there are splitting files on the tape how can i see that? and bring them together

 

[TiborB]

Well, I googled a bit. To find what node has data on a specific folume use this:

select distinct NODE_NAME from volumeusage where VOLUME_NAME='ABCDEFG'
 

 NODE_NAME                                                         
 ------------------------------------------------------------------
 BLABLA

So check the content of these 11 tapes if the nodes differs.


Also, if a node is not in collocation group it is treated as a separate collocation group and is put into separate tape.


You can use this select to verify:

select NODE_NAME,COLLOCGROUP_NAME from nodes

Did you changed "membership" of collocation groups lately, or add new nodes, or increased number of maxscratch tapes lately?

 

[bdgree]

The output of the command is:
tsm: ZTMTSM1>select distinct NODE_NAME from volumeusage where VOLUME_NAME='VS6120L3'
ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.
All the " problem" tapes have the same output.

All the nodes have a collocation group
The last time nothing changed.

So it stay's curious!!!!

 

[TiborB]

well, I am going out of ideas, however this select

select distinct volume_name from volumeusage where volume_name in (select volume_name from volumes where status='FILLING' or status='FULL') and node_name=''

should list all volumes in FULL or FILLING states that have no nodedata on them. So it should list al least your eleven tapes. Please check it, maybe you notice anything suspicious - like more then 11 such tapes...

I was not able to properly test the select, it returns "No match found using this criteria" here - what is expected at the end...

 

[bdgree]

ThanX TiborB,

but the sql command gives nothing

tsm: ZTMTSM1>select distinct volume_name from volumeusage where volume_name in(select volume_name from volumes where status='FILLING' or status='FULL') and node_name=''
ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.

 

[TiborB]

Well, I dont know ....... but I would like to hear some solution...



I just found out that there is auditdb command, at least for 5.x versions, maybe it would recognize this issue as errors in DB. But I dont think they are errors.


One workaround would be to create a new stgpool with only one tape and move the data from these 11 volumes there...


And back to collocation - I presume that when you said all nodes backing up to the stgpool has a colllocation group, you meant they have the same collocation group....

 

[bdgree]

could this maybe the problem?
I have removed a time ago backupsets. The backupsets are created with a toc.
Could it be that this toc is not deleted but still on tape?
How can i find out if this is the problem?

 

[bdgree]

Yes , we have more collocationgroups but not for every node of course