How to make a copy of a specific copy pool volume ?

[ldmwndletsm]

Okay, the answer to this question is most likely a resounding "NO". After hunting around for a while on this forum and elsewhere, it appears that there is no support in this product for this, but in case I've overlooked something, or there might could be some ingenious workaround, let me state the background and problem.

[ BACKGROUND ]
We have a storage pool wherein the data must be restorable for perpetuity. Therefore, we do not use reclamation on this pool. Even if we set a very low threshold, it would never occur because data is seldom deleted from the affected file spaces on the client, certainly never enough to even satisfy a 1% reclamation threshold. We have 'No Limit' set on the copy group parameters. We have a copy pool for off-site storage. This data comprises a single node, the sole member of its own collocation group. No, that probably wasn't necessary since it's the only node writing to that storage pool, but for consistency, we created a collocation group just for it. Data is first written to disk pool then copied ("backup stgpool") to copy pool tapes, migrated (nextstgpool; lowmig/highmig) from disk to primary pool, and later another "backup stgpool" command is run to copy any data on primary pool tape to copy pool tape that might not have made it there before it was deleted from disk.

[ PROBLEM ]
The problem that we've run into is that it's taking longer and longer for the tapes to become FULL than it used to. Obviously, the primary pool tapes are not a concern as they have forever to fill. Yes, we can use "move data" to consolidate the copy pool volumes to the one filling tape with the least space (e.g. forcing this by marking the others readonly). But this still will not push the remaining copy pool tape close enough to 100% usage. So we have a choice between waiting for it to eventually fill up or taking a low usage tape off site. This is okay from time to time but it will eventually take its toll on our tape supply if we do this ad infinitum. And we cannot simply return it from off site later to append to it since that places us at risk.

With other storage pools, wherein reclamation is used, this is not a concern since on a weekly basis, we get back a fair number of the off-site tapes for reclaims, never mind the ones that never come back, but this is not possible with the aforementioned pool.

[ FANTASY SOLUTION ]
If would be nice if we could simply make a copy of a specific copy pool tape. Then that could be taken off site as a surrogate, and we then repeat that each week until the original tape is full, and then it goes off site, and all the temporary surrogates would then be brought back and recycled. We used to do this with another backup product that supported this since it allowed more granular control on a per tape basis -- very flexible and worked like a champ. We never ate through too many of these temporary copy tapes before the original filled up, so no big deal in terms of temporary tape consumption, and that allowed us to space out the deposits of the original copy tapes or minimally allow them to reach higher usage while still offering proper protection.

1. A backupset could be created, I suppose, but it doesn't look like it allows you to specify the input volumes, and it would capture all the data in the named file spaces, right ? I checked, and there are a number of file spaces on the current filling copy pool tape with the least remaining space. Having to run a full of all of those file spaces would take a long time. 2. Creating a second copy storage pool, and running a second "backup stgpool" command would end up trying to copy all the data in the primary pool all over again to the second copy pool as none of it would exist there yet.

We really need to be able to just make a copy of the incremental from any one tape.

[ ACTUAL SOLUTION ]
Would it be possible to modify our modus operandi to automate the process of creating a second copy pool volume for each "original" copy pool volume ? -- BUT without having to re-copy all the data in the primary pool as that would take a year !

Perhaps, we should have set this up from the beginning, but now it's too late ? Okay then, what would be the closest thing in TSM that we could do now to mitigate this problem going forward ? I guess that's the real question.

 

[ldmwndletsm]

[ Idea 1 ]
One possible idea would be to create a new primary storage pool (no reclamation), generate a list (contents table) of the file path names on the copy volume, reduce that down to a unique list (sort -u) and then run that through a script to back up all those files to a different storage pool. They all live on the same node so fairly straightforward. That would create a different primary pool tape with the same data, and although possibly a superset, it would not be appreciably more than what the copy volume holds. Then, rather than creating a copy of that, you simply take that second "new" primary pool tape off site to act as your temporary copy pool volume and so on and so forth until the copy volume is full at which point you then delete those temporary primary pool volumes and recycle them.

Of course, creating these would take longer than if you could instead just read and write from the copy volume to another volume since you'd have to back up the data across the network and then migrate it from disk to tape. That's also going to consume resources on the node, possibly overlapping with the nightly backups. But there is also the distinct possibility that some of the files might not exists or might have changed since the original copy volume was made, so you might not end up with all the same versions in which case you would not have a duplicate of the original copy volume. You could check before backing the files up, but what are you going to do if some of the files have changed or no longer exist ? But even if no changes occurred, you'd still have to check again after to be certain nothing changed during backups.

Also, this is going to be cumulative, so each time you make an off-site visit, you'd have to re-back up all the same files again since the filling copy volume would have since grown -- unless, of course, you generate another list and compare that against the previous list and only re-back up the delta. That might work, but dates are not included in the contents table (too convenient) and are instead squirreled away in some other table far off in Neverland.

But even if all that passed muster, it's unclear to me if TSM would permit you to restore, say, a subdirectory wherein part of the files live on the original copy volume, and the others live on the temporary volume. I may not be remembering correctly here, but I think one time that I restored data as a test wherein at least one of the primary pool volumes was available, but not all of them, but the copy volumes that had that corresponding data were available. As a result, it was a mixed recovery wherein TSM did not use solely primary or solely copy pool volumes but some of each. However, I don't know if this would work in this case since you're crossing pools, and they're not related (primary/copy counterparts).

Regardless, this would clearly be ridiculous.

 

[ldmwndletsm]

[ Idea 2 ]
You could simply take the copy volumes off site prematurely, before they're full, and then at a later date, once you have enough off-site filling volumes to add up to 100% usage you then return them and consolidate them using "move data" and then take the final "new" volume off site the next business day, and the original ones that were brought back would be returned to scratch. This is a risk versus gain scenario but could probably be completed in one day, and best done M-Th so you don't face risk for more than one day.

[ Idea 3 ]
Alternatively, you could leave the filling volumes off site and just run the "move data" in which case TSM would use the primary pool volumes and create a "new" copy pool volume from that data. Then you'd take that tape off site and bring the others back. This seems safer in obvious ways, but it still exposes you to the same risk until you get that "new" volume off site because once the "move data" completes, TSM is no longer tracking any data on those off-site tapes. Only a database restore would fix that, but that's a mess and would impact production operations unless you used some test server that could access a tape drive. Moreover, you're hedging all your bets on the primary pool copy in the sense that regardless of whether the original copy was made from the same data on a disk volume or by reading the primary pool tape, it somehow seems risky in that the primary pool volume and copy pool volume are no longer both in the picture, and you instead have the primary pool copy twice. So if anything went bad on that primary, but could otherwise still be read, and the original copy was good, then you've now forfeited that. Somehow, making a copy from the copy (if you could), rather than another copy from the primary (that will now replace the original copy), seems less risky in that regard. So idea 2 has this advantage over 3.

 

[ldmwndletsm]

In my original post where I said:

[ ACTUAL SOLUTION ]
Would it be possible to modify our modus operandi to automate the process of creating a second copy pool volume for each "original" copy pool volume ? -- BUT without having to re-copy all the data in the primary pool as that would take a year !

Perhaps, we should have set this up from the beginning, but now it's too late ?

Setting up an additional copy job from the get-go, using a second "backup stgpool" command/admin schedule, to make a second copy to a another copy storage pool would not have helped because as soon as you delete the temporary copy volumes, TSM would want to re-create those copies the next time the "backup stgpool" command was run. It always wants to keep the copy storage pool up to date with whatever is on the primary storage pool so if my understanding is correct then this seems a fool's errand.

It would be nice if TSM allowed more granular control of specific volumes and the data on those, but that does not appear to be the case as it is with other backup product(s). So my best guess at this point is that one would have to use something along the idea of what I stated in "[Idea 3]" in my last post.