TSM No scratch in the Library

[marclant]

I'm not going to go through your activity log for you. Check for messages prior to "SPACE RECLAMATION completed with a completion state of FAILURE." and look for errors (ANR####E) that are for (PROCESS: 3525).

 

[DazRaz]

The problem is here.

03/19/2017 09:48:36 ANR8341I End-of-volume reached for LTO volume 000134L5.
(PROCESS: 3857)
03/19/2017 09:48:36 ANR0515I Process 3857 closed volume 000134L5. (PROCESS:
3857)
03/19/2017 09:48:41 ANR8336I Verifying label of LTO volume 000134L5 in drive
DRIVE_4 (mt0.0.0.6). (PROCESS: 3857)
03/19/2017 09:48:42 ANR1405W Scratch volume mount request denied - no scratch
volume available. (PROCESS: 3857)
03/19/2017 09:48:54 ANR1626I The previous message (message number 1405) was
repeated 2 times.
03/19/2017 09:48:54 ANR1086E Space reclamation is complete for volume
000335L5. The storage pool has insufficient space.
(PROCESS: 3857)

Are there free scratch in the library? What is the maxscratch setting for your offsite storage pool? (q stg <storage_pool_name> f=d)

 

[chafik]

Its :300

tsm: q stg OFFSITE-DAILY f=d

Storage Pool Name: OFFSITE-DAILY
Storage Pool Type: Copy
Device Class Name: LTO_CLASS_OFF
Storage Type: DEVCLASS
Cloud Type:
Cloud URL:
Cloud Identity:
Cloud Location:
Estimated Capacity: 757,379 G
Space Trigger Util:
Pct Util: 17.8
Pct Migr:
Pct Logical: 99.6
High Mig Pct:
Low Mig Pct:
Migration Delay:
Migration Continue:
Migration Processes:
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold:
Access: Read/Write
Description:
Overflow Location:
Cache Migrated Files?:
Collocate?: No
Reclamation Threshold: 97
Offsite Reclamation Limit: No Limit
Maximum Scratch Volumes Allowed: 300
Number of Scratch Volumes Used: 125
Delay Period for Volume Reuse: 2 Day(s)
Migration in Progress?:
Amount Migrated (MB):
Elapsed Migration Time (seconds):
Reclamation in Progress?: No
Last Update by (administrator): ADMIN
Last Update Date/Time: 03/20/2017 20:34:21
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?:
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: No
Processes For Identifying Duplicates:
Compressed:
Deduplication Savings:
Compression Savings:
Total Space Saved:
Auto-copy Mode:
Contains Data Deduplicated by Client?: No
Maximum Simultaneous Writers:
Protection Storage Pool:
Date of Last Protection:
Deduplicate Requires Backup?:
Encrypted:
Space Utilized(MB):

 

[DazRaz]

Ok. What it looks like has happened is that you ran out of scratch tapes and reclaimation failed because of that. You then added scratch tapes but there was so much to catch up on that it ran out of scratch before actually completing any individual volumes. Assuming that your DRM is working ok and you are getting tapes back if they are reclaimed, this is what I would do.

Set the reclaimation level to 60 (from 97).
Limit the number of tapes the process tries to reclaim at any one time to 5.
Add one scratch tape and start reclaimation.

upd stg OFFSITE-DAILY rec=60 offsitereclaiml=5

<Add one scratch tape>

reclaim stg OFFSITE-DAILY

This should get the first 5 tapes reclaimed reasonably quickly and you should see the status of the tapes changing to PENDING state. I use the follow SQL query to checking pending tapes.

select volume_name, pending_date as "Pending Date " from volumes where status='PENDING' order by pending_date

You should start seeing tapes listed here. As you have a reuse delay of 2 days, any tapes fully reclaimed in the last 2 days will show with this query.

 

[marclant]

upd stg OFFSITE-DAILY rec=60 offsitereclaiml=5

<Add one scratch tape>

reclaim stg OFFSITE-DAILY

If you do that, reclamation can kick off anytime a volume is eligible for reclamation.

Best to keep the reclamation threshold for the pool at 100. Create an admin schedule that kicks off reclamation when YOU want, not when the server deems it necessary. Your admin schedule should have this command:
reclaim stgpool {poolname} threshold={number} duration={number}

Or you can kick off the above command manually

 

[chafik]

I followed the steps:
- updating the stg
- Add 3 new scratchs
- launch reclamation

No tape was reclaimed and no tape was in pending state


I went through the logs:

- I saw the following Error:

ANR1893E Process 5331 for SPACE RECLAMATION completed with a completion state of FAILURE

I checked the time and what was before, I saw the following warning:

Date/Time Message
-------------------- ----------------------------------------------------------
04/01/2017 00:36:00 ANR1040I Space reclamation started for volume 000206L5,
storage pool OFFSITE-DAILY (process number 5334).
(PROCESS: 5334)
04/01/2017 00:36:00 ANR1259W Files on volume 000313L5 needed for offsite
reclamation cannot be accessed - access mode is
"unavailable" or "offsite". (PROCESS: 5334)
04/01/2017 00:36:00 ANR1259W Files on volume 000331L5 needed for offsite
reclamation cannot be accessed - access mode is
"unavailable" or "offsite". (PROCESS: 5334)
04/01/2017 00:36:00 ANR1259W Files on volume 000314L5 needed for offsite
reclamation cannot be accessed - access mode is
"unavailable" or "offsite". (PROCESS: 5334)
04/01/2017 00:36:00 ANR1259W Files on volume 000176L5 needed for offsite
reclamation cannot be accessed - access mode is
"unavailable" or "offsite". (PROCESS: 5334)
04/01/2017 00:36:00 ANR1259W Files on volume 000116L5 needed for offsite
reclamation cannot be accessed - access mode is
"unavailable" or "offsite". (PROCESS: 5334)
04/01/2017 00:36:00 ANR1163W Offsite volume 000253L5 still contains files
which could not be moved. (PROCESS: 5334)
04/01/2017 00:36:00 ANR1163W Offsite volume 000345L5 still contains files
which could not be moved. (PROCESS: 5334)
04/01/2017 00:36:00 ANR1163W Offsite volume 000169L5 still contains files
which could not be moved. (PROCESS: 5334)
04/01/2017 00:36:00 ANR1163W Offsite volume 000206L5 still contains files
which could not be moved. (PROCESS: 5334)
04/01/2017 00:36:00 ANR1163W Offsite volume 000221L5 still contains files
which could not be moved. (PROCESS: 5334)
04/01/2017 00:36:00 ANR4932I Reclamation process 5334 ended for storage pool
OFFSITE-DAILY. (PROCESS: 5334)
04/01/2017 00:36:00 ANR0985I Process 5334 for SPACE RECLAMATION running in
the BACKGROUND completed with completion state FAILURE
at 00:36:00. (PROCESS: 5334)
04/01/2017 00:36:00 ANR1893E Process 5334 for SPACE RECLAMATION completed
with a completion state of FAILURE. (PROCESS: 5334)

==============================================================


I updated the volume to Read/Write and launch the reclamation again.
The reclamation stops after few minutes and the tapes become unvailable again

Really I`m lost, becose the number of the showreclain command in increasing day after day and I`m running of tapes

Thank you

 

[marclant]

I updated the volume to Read/Write and launch the reclamation again.

Are those primary volumes in the library? If not, check them in, then make sure they are read/write.

 

[chafik]

Hello,

I cancelled all the session on the TSM server.
I did an audit for the library.
I add one more tape
I launched the reclamation.
Now I see tapes in pending status
I changed the reuse delay to 1 day.

Is there a way to call these OFFSITE tape immediately ? without changing the delay reuse to 0

Thank you

 

[chad_small]

So are you saying after reclamation the offsite tapes don't come back onsite? Is there a reuse delay? If TSM is failing the offsite reclamation then there is an issue with the onsite copy. Since TSM uses the onsite copy to reclaim offsite tapes if there are any issues with the onsite copies you will fail reclamation. Are any of the tape in the onsite copy in an unavailable state? Are there no scratch available to execute the reclamation?