Advice needed on a major backup snafu

[jambraun]

Ok, I got a doozy for you all to wrap your minds around...or you might have an answer Here are the particulars; I have a storage pool 1.9TB large which I've managed to fill to 100% capacity effectively hindering the backup of any clients. When I began this process, I started with snapshot images of each server that may, or may not have been compressed. From then on out I did incremental backups, at first not compressed, but compresed had been added to the procedure later a few days. Backup and Archive retention had been set to 365 days which I have now lowered to 5 days respectively.



I know for a fact that only 1TB of data exists on all of our servers, but we've managed to store twice that amount in backup. My initial thought is that inactive, archived files created during the incremental processes have added up. Another thought is that not all the files on the server are compressed as they should be.



What I was thinking of doing right now was expiring and removing the inactive files more than 5 days old and possibly compressing what is already on the server. Is there a command that I can use to do either of these things or does anyone have another solution that might work? The only other alternative that I can think of is deleting the filespaces from the storagepool and starting from scratch, which I REALLY don't want to do for obvious reasons.

 

[heada]

Once the data has been sent to the TSM server, the client can no longer compress it (obviously) The only other way to compress data is to write it out to tape in a compressed format. I assume you either don't have tape or don't want to use it since that is the easiest answer to your problem. (just migate the diskpool to tape)



Once you lowered the retention period, did you run client backups and then expire the data? Until the data is marked expired and you run an "expire inventory" it wont be removed from TSM.



You say that the clients only have 1TB of data, but if 500GB changes every night, you'll have 2.5TB of changed data over the 5 day retention. I would look to see how much data is being backed up every night to size the diskpool to that (125% of needed is a good plan)



-Aaron

 

[jambraun]

Thanks for replying Aaron.



I figure I'd add some additional info for clarrification. As you've discerned, we're not a tape house. Everything we do is on Harddrives...including backupsets and data rentention. So yes, option 1 is definately out.



As for running client backups again, I assumed they would fail since the storage pool is at max capacity, but if it expires and removes prior to backing up data, that may work. A lot more will be removed than backed up so it's worth a shot.



Oh, it took 3 weeks reach capacity, without inital compression so hopefully this will be a short term solution until I can smooth-talk the budgeting people in to additional storage



On a side note, I did find an option within the Delete Filespace command to remove expired files if all else fails, but that'll be the second to last resort.

 

[jambraun]

It was a no go. Running client backups resulted in a server out of storage error. Is there a command that I can run across all nodes that will remove inactive files across the board? Apparently the command I tried was of no use:



delete filespace apollo * type=archive



Then again, it's completely possible that there is another command that I can try that will accomplish this. Something with the "Type=Inactive" in the name. The search continues.

 

[heada]

Have you tried to expire all the old data?



expire inv quiet=yes



-Aaron

 

[jambraun]

Not yet...although that is probably the command I'm looking for.



Incidentally, I'm quite frustrated I was unable to query my filespaces for inactive files. Apparently this option is only available when using the QUERY BACKUP command, which is NOT available on windows servers. I know i can browse for these files using the GUI but the command line would be a lot more efficient and actually easier.



I'll give the expiration idea a go and see what happens. Thanks.

 

[jambraun]

Alright, I want to say that running that command flagged all the files for deletion that should be deleted. Was the EXPIRE command supposed to also remove these files from the filespaces or just mark them for deletion at the next backup?

 

[jambraun]

Alright, I think we're almost in good shape. The problem is that I need these files to be actually deleted from the objects before EXPIRE INVENTORY can remove them from the pool. What seems to be working (and this has GOT to be the most INSANE way of doing this so please someone chime in) is granting clients backupset deletion rights and from client of each node, run a DELETE BACKUP command on all files that are INACTIVE. Once that completes, they are flagged for removal by way of EXPIRE INVENTORY from the server console.



As of now, I have to do this by hand from each client....can someone please clue me in to the correct way to do this? Preferably from a single location?

 

[Robert_Cody]

I'd first check to see how many files are found in the default managment class using:



select node_name, count(*) as "Files" from backups where class_name='DEFAULT' and node_name in (select node_name from nodes where domain_name='SERVER') group by node_name



If there are any, I suspect you have a problem with your incl/excl statements being properly applied.



Next, I'd look to see many volumes are reclaimable using:



select count(*) from volumes where stgpool_name='DISKFILE' and upper(status)='FULL' and pct_utilized < 40



If there are none, run "expire inventory" to see this adds any.



If expire inventory doesn't free up some volumes, then you will need to update your copypools to remove all versions except the most recent. Example:

--------------------------------------------------------------------------------------------------

update copygroup server server data standard type=backup verexists=1 verdeleted=0 retextra=0 retonly=0



validate policyset server server

activate policyset server server



expire inventory

--------------------------------------------------------------------------------------------------

Recheck free volumes:



select count(*) from volumes where stgpool_name='DISKFILE' and upper(status)='FULL' and pct_utilized < 40



If you now have relaimable space, run your defrags. Example:



update stgpool diskfile reclaim=60