COPYPOOL Reclamation Acting Strangly

[DaJuRan]

Good Day All,

I am hoping i can get lots of input on this issue as its baffling me a little bit at the moment. Some background info, my TSM experience has been very limited however since our initial TSM administrator left the organization I have been learning quite a bit around it in the last few months.

I have just recently, about 8 days ago setup collocation groups and turned group collocation on for both our TAPEPOOL (onsite) and COPYPOOL (offsite). We are running a maintenance script containing a reclamation threshold of 50%, which causes most days for 3-4 tapes from the previous days backup/reclamation to fall within the threshold again for reclamation. This is where its being a little strange as from monitoring the reclamation on my COPYPOOL lately its not completing the reclamation of the single volume at the same time, it appears to be doing some of each tape and then on a second or sometimes third round through will finish the reclamation for the tape, therefore causing what a lot of unnecessary mount points and wait times for them.

Being the tapes were used the previous day, and with collocation on would be collocated data my thoughts would have been that it would complete the reclamation on one of the reclamable tapes in the copypool and then move to the next one, instead it does some of one and then some of another etc and for each case has to switch to a different input volume and an output volume because of the group collocation. It would make more sense to me if it had its output volume loaded for the tape its going to reclaim (will be the same output volume for the entire tape as the data being reclaimed is already collocated via the groups) and if it had to go through a couple of tapes to hit all the data in the TAPEPOOL to do the reclamation then it would do so, and so possibly a few mounts on the input volumes but none on the output until its starts reclaiming a second tape.

Seems like getting hit on both input and output when wouldn't have expected it to jump around this way at all. Any information would be greatly appreciated or suggestions to check or if I am not alone possibly in such an issue?

Some info that might be requested below:
q stgpool COPYPOOL f=d

Storage Pool Name: COPYPOOL
Storage Pool Type: Copy
Device Class Name: LTO4
Estimated Capacity: 1,502,176 G
Space Trigger Util:
Pct Util: 1.8
Pct Migr:
Pct Logical: 99.9
High Mig Pct:
Low Mig Pct:
Migration Delay:
Migration Continue:
Migration Processes:
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold:
Access: Read/Write
Description: Copy Pool
Overflow Location:
Cache Migrated Files?:
Collocate?: Group
Reclamation Threshold: 100
Offsite Reclamation Limit: No Limit
Maximum Scratch Volumes Allowed: 1,000
Number of Scratch Volumes Used: 33
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?:
Amount Migrated (MB):
Elapsed Migration Time (seconds):
Reclamation in Progress?: Yes
Last Update by (administrator): HELPDESK
Last Update Date/Time: 08/13/2012 18:56:01
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?:
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: No
Processes For Identifying Duplicates:
Duplicate Data Not Stored:
Auto-copy Mode:
Contains Data Deduplicated by Client?: No


q stgpool TAPEPOOL f=d

Storage Pool Name: TAPEPOOL
Storage Pool Type: Primary
Device Class Name: LTO4
Estimated Capacity: 1,450,854 G
Space Trigger Util:
Pct Util: 1.9
Pct Migr: 2.7
Pct Logical: 99.9
High Mig Pct: 90
Low Mig Pct: 70
Migration Delay: 0
Migration Continue: Yes
Migration Processes: 1
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold: No Limit
Access: Read/Write
Description: Onsite Tape Pool
Overflow Location:
Cache Migrated Files?:
Collocate?: Group
Reclamation Threshold: 100
Offsite Reclamation Limit:
Maximum Scratch Volumes Allowed: 1,000
Number of Scratch Volumes Used: 27
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?: No
Amount Migrated (MB): 0.00
Elapsed Migration Time (seconds): 0
Reclamation in Progress?: No
Last Update by (administrator): HELPDESK
Last Update Date/Time: 08/13/2012 18:56:09
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?: Yes
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: No
Processes For Identifying Duplicates:
Duplicate Data Not Stored:
Auto-copy Mode: Client
Contains Data Deduplicated by Client?: No

There are five groups containing in total 78 nodes, 38 of which are in its own group and then two groups of 6 nodes, a group of 8 and a group of 20.

Thanks,
DaJ

[Ramanbhatla]

However I feel you should try changing the 50% parameter to around 80% or 90%. Hopefully that should work.

[DaJuRan]

80 or 90% would make for very ineficient use of offsite storage, do believe 60% is recommended best practice and with some configs and tweaks lately I have been able to reduce our backup window to give a large window for reclamation so set to 50% to better utilize offsite storage space.

DaJ

[drivechip]

Do you use DRM and are the tapes which are qualified for reclamation offsite?

[DaJuRan]

yes and yes, some of the tapes that fall into reclamation contain the previous two or three days backups and possibly some older that fell into reclamation along the way, however both pools are collocated and almost all data on our onsite copypool has been collocated so its not making logical sense for it to not finish the tape it is reclaiming first before moving to another when to move to another it has to load both a new output and input tape, whereas finishing the initial tape it might have had to load a new input but with plenty of space on the output that would have been ok.

Any settings etc anyone can think on that might correct? I know collocation does cause more mount points because of its grouping and use of tapes per group but I didn't expect it to have mounts without an actual purpose.

DaJ

[drivechip]

Maybe I don't get it. But the behavior could be possible when a not yet collocated tape in a offsite pool is reclaimed. This could lead to several tape mounts for source and destination volumes.

[DaJuRan]

That it could, however all tapes involved have already been collocated, thats what is so odd about it. Had it ocurr today where it left .1% utilized on the tape moved onto other tapes for reclamation and then will go back to that later in its processing and load the same two tapes it was working with before and finish reclaiming. Very odd, guess its not something anyone has come accross or possibly never noticed, or taken any issue with it. I just hate anything that causes my reclamation to process slower then it should as it can then cause someone to have to make a trip in to pick up tapes afterhours when its finished.

Thanks,
DaJ

[sjl]

The only thing I can think of is that TSM does generate an internal list of the tapes it will need for the reclamation run, based upon which tapes hold data for the tapes to be reclaimed. I would have expected it to mount the tapes for a given collocation group in order, rather than skipping between groups, though. Are you absolutely certain that all of the data on all of your tapes is collocated? Because if they aren't, it might explain the behaviour.

I'm just guessing, though. QUERY NODEDATA on the source volumes that it's churning through might help you figure it out (along with QUERY NODEDATA on the copygroup volumes that don't seem to be fully reclaimed each time - check the nodes that come up and see which groups they're in, see whether it's all as expected.)

[DaJuRan]

I will have to take a peek on that side of it.

I did notice yesterday however after making a change in my maintenance script to have multiple copypool reclaims but limited to do a single tape at a time, essentially a way around having it jump all over the place, that it was taking forever for the last couple of percent on some of the tapes and so when I looked at it further it was going against a number of tapes for small files and because of the mounts and winding of the tape took about 40 minutes in one case for 0.1% of a tape. The tapes it was going against have not yet been collocated, however they are a few weeks old now and not sure why it would have to go against these other tapes.

Is it possible that something would be stored on these tapes from the full backup that was done initially which is required on each of the collocated tapes, and because its stored on non collocated tapes that it has to jump through a few of them to grab this small amount of data for the new collocated tapes?

If this is the case hopefully within the next couple of weeks as these other tapes are collocated it will smooth itself out.

Thanks,
DaJ

[DaJuRan]

Figure would conclude up this thread. I still havn't found time to investigate the finding I had posted in my last posting however setting multiple reclamation commands in my maintenance script with a limit of a single tape for each has gained me the processing I was hoping for in tape by tape being completed.

Thanks,
DaJ