Long term retention suggestions?

RecoveryOne

ADSM.ORG Senior Member
Joined
Mar 15, 2017
Messages
347
Reaction score
81
Points
0
PREDATAR Control23

So, I am now faced to a similar issue from this post way back when: https://adsm.org/forum/index.php?threads/long-term-retention-of-data-and-tsm.28157/

Customer wants to keep 10 years worth of data within TSM. They only have enough disk space for two years on their storage appliance. The data in question is fairly large video/image files. Estimates are 10 - 25G a day. If they require data from year 4, a ticket would be entered to restore specific date. Good news, folder structure will be setup in a form such as:
|-YYYY | |-MM | | |-DD | | | |-ID Value
And that structure matches their database so they can easily say March 30 2024, ID 6. Makes my life easier.

I was thinking HSM would be perfect for this, however the backing device for customer's data is a filer, and HSM doesn't support working over UNC paths. So that's out the window.

Archives? Monthly archives, weekly archives are a thought. I was told that the customer will be manually managing their storage space, as in going to their mount point and manually deleting files when their quota is reached!!!! I brought up archives can automatically delete data after the archive process ran. That was shot down by them. They want to be in control of the deletion processes but cannot commit to a date/time when such process would run. Right now, it's my understanding that it will be a human to go in and purge these files by hand, no sort of automation. :eek: So, if an archive processes is running building its file list, they go in and nuke those files, I would have nothing.

As the post I referenced above, the "cheatin' it" method.
Set up a RetOnly policy of 3650 days for their unc path. Run an incremental once, twice a day depending on their needs and forget about it. Backup to filepool and migrate to tape pool. The thought of doing a month end, or year end won't work in my mind as there's a very real chance I could miss data as I have no commitment from the customer as to when/what time they will purge. Same issue with the archive idea.

I am unable to change anything about their workflow, so the best I've been able to come up with is an incremental forever with a RetOnly of 10 years and just bite the bullet.

I mentioned that the storage device is a filer, but I try to stay way from filer based snapshots. Just worried if that filer is destroyed, we'd have to have an identical one to restore data back to. Given that we run our units till they should be in the scrap heap, and then ask a few more years out of them that really worries me. So, I need file level backup that I can restore to any node at a whim.

What am I missing? What am I forgetting? I know this will be a strain on my DB, but shouldn't be too bad as I will not be performing any sort of deduplication or compression. Copies will still be in place. Gotta have my copies! Dedicated primary tape pool, shared copy tape pool. Am a bit concerned about bit rot on the tapes, especially as they reach that 10 year mark, but they are left in my library and not touched by human hands.
 
PREDATAR Control23

I am unable to change anything about their workflow, so the best I've been able to come up with is an incremental forever with a RetOnly of 10 years and just bite the bullet.
Yup, that's much better than archives or weekly/monthly. Make sure you set verdeleted to 1 or greater if they intend on deleting the files from the filesystem.
 
PREDATAR Control23

Yup, that's much better than archives or weekly/monthly. Make sure you set verdeleted to 1 or greater if they intend on deleting the files from the filesystem.

Cool, good to know I'm not far off the wall. I will admit, I had some reservation of hope that I was looking at it from the wrong point of view and someone might have something.

I was actually thinking about having VerDataExists and VerDataDel set to no limit and control the expire via RetExtra and RetOnly. Since this will in all likely hood be a once or twice a day backup, if I kept RetExtra at say 32 days, that should cover most 'hey we changed this file want to go back to it'. Yes, more discussions with the customer needs to happen to narrow that down but at least from my initial thoughs might be the way to go.
 
PREDATAR Control23

I hope they will do their own restore if they need old data they deleted. Otherwise, whoever is the backup admin will be busy. An HSM type solution would have been better, but like you said, not possible with the current setup.
 
Top