• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Long term retention suggestions?

RecoveryOne

ADSM.ORG Senior Member
#1
So, I am now faced to a similar issue from this post way back when: https://adsm.org/forum/index.php?threads/long-term-retention-of-data-and-tsm.28157/

Customer wants to keep 10 years worth of data within TSM. They only have enough disk space for two years on their storage appliance. The data in question is fairly large video/image files. Estimates are 10 - 25G a day. If they require data from year 4, a ticket would be entered to restore specific date. Good news, folder structure will be setup in a form such as:

|-YYYY
| |-MM
| | |-DD
| | | |-ID Value

And that structure matches their database so they can easily say March 30 2024, ID 6. Makes my life easier.

I was thinking HSM would be perfect for this, however the backing device for customer's data is a filer, and HSM doesn't support working over UNC paths. So that's out the window.

Archives? Monthly archives, weekly archives are a thought. I was told that the customer will be manually managing their storage space, as in going to their mount point and manually deleting files when their quota is reached!!!! I brought up archives can automatically delete data after the archive process ran. That was shot down by them. They want to be in control of the deletion processes but cannot commit to a date/time when such process would run. Right now, it's my understanding that it will be a human to go in and purge these files by hand, no sort of automation. :eek: So, if an archive processes is running building its file list, they go in and nuke those files, I would have nothing.

As the post I referenced above, the "cheatin' it" method.
Set up a RetOnly policy of 3650 days for their unc path. Run an incremental once, twice a day depending on their needs and forget about it. Backup to filepool and migrate to tape pool. The thought of doing a month end, or year end won't work in my mind as there's a very real chance I could miss data as I have no commitment from the customer as to when/what time they will purge. Same issue with the archive idea.

I am unable to change anything about their workflow, so the best I've been able to come up with is an incremental forever with a RetOnly of 10 years and just bite the bullet.

I mentioned that the storage device is a filer, but I try to stay way from filer based snapshots. Just worried if that filer is destroyed, we'd have to have an identical one to restore data back to. Given that we run our units till they should be in the scrap heap, and then ask a few more years out of them that really worries me. So, I need file level backup that I can restore to any node at a whim.

What am I missing? What am I forgetting? I know this will be a strain on my DB, but shouldn't be too bad as I will not be performing any sort of deduplication or compression. Copies will still be in place. Gotta have my copies! Dedicated primary tape pool, shared copy tape pool. Am a bit concerned about bit rot on the tapes, especially as they reach that 10 year mark, but they are left in my library and not touched by human hands.
 

marclant

ADSM.ORG Moderator
#2
I am unable to change anything about their workflow, so the best I've been able to come up with is an incremental forever with a RetOnly of 10 years and just bite the bullet.
Yup, that's much better than archives or weekly/monthly. Make sure you set verdeleted to 1 or greater if they intend on deleting the files from the filesystem.
 

RecoveryOne

ADSM.ORG Senior Member
#3
Yup, that's much better than archives or weekly/monthly. Make sure you set verdeleted to 1 or greater if they intend on deleting the files from the filesystem.
Cool, good to know I'm not far off the wall. I will admit, I had some reservation of hope that I was looking at it from the wrong point of view and someone might have something.

I was actually thinking about having VerDataExists and VerDataDel set to no limit and control the expire via RetExtra and RetOnly. Since this will in all likely hood be a once or twice a day backup, if I kept RetExtra at say 32 days, that should cover most 'hey we changed this file want to go back to it'. Yes, more discussions with the customer needs to happen to narrow that down but at least from my initial thoughs might be the way to go.
 

marclant

ADSM.ORG Moderator
#4
I hope they will do their own restore if they need old data they deleted. Otherwise, whoever is the backup admin will be busy. An HSM type solution would have been better, but like you said, not possible with the current setup.
 

Advertise at ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

UpCloud high performance VPS at $5/month

Get started with $25 in credits on Cloud Servers. You must use link below to receive the credit. Use the promo to get upto 5 month of FREE Linux VPS.

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 14 17.3%
  • Keep using TSM for Spectrum Protect.

    Votes: 51 63.0%
  • Let's be formal and just say Spectrum Protect

    Votes: 9 11.1%
  • Other (please comement)

    Votes: 7 8.6%

Forum statistics

Threads
31,380
Messages
133,653
Members
21,492
Latest member
CamoVan
Top