How to remove tape with long term archived datas from a crowded library?

[DiegoC]

Hi all!



I work for a movie post production facility, the problem with our businness is that when we finish a prodution we have to archive for a long time big amounts of datas, 0.5-2 Tb when a movie get ready and 0.1-0,5 Tb for a commercial.



I am new to tivoli administration but willing to learn



For the pourpose of archiving this projects I setted up a very long retention time for archived datas.



I use:

dsmc archive /projectdir/ -des="Projectname_{TIMESTAMP}" >/ArchiveLogs/Projectname_{TIMESTAMP}.log 2>&1



to archive finished works and, after verifing the log, I delete the datas from disk.



Our library is reaching his limit of 140 LTO2 Tapes quite fast and I need some suggestion on how to put this data offline, possibly getting a standalone set of tapes for every archived project. Can any one help me?



Reading the forums I found about overflowing and it seem like a possible path to follow for me, in this case, can I "overflow" tapes containig specifica projects only? and how?



Thanks for any help!

[heada]

I would first start out by making sure that the tapes you have are used efficently. Check the reclamation threshold for the stgpools and check your tapes to make sure they are within those limits.



As for removing tapes from a library, you can define an overflow location and checkout the tapes to that location. TSM will ask you to check the tapes back in if a restore is needed.



When you are archiving the data, you can archive and delete in one motion with TSM. There is an "archive-delete" option that will write the data to tape and then remove it from the filesystem.



When you archive the data and remove it from the system, do you also clean up the backup history of those files?



-Aaron

[DiegoC]

Hi Aaron!

First of all thanks for your answer.



I cheked tapes are used efficently, most of the tape are full (120), just 3 or 4 are filing...



I'm not using the archive-delete funztion becouse I want the data runner to do e test restore before deletind the datas by hand.



I don't know how to clean the data history of archived files, our data runner moves all data related to the same project to be archived in a special foder (this folder is excluded from backup) than a script archives the files and move them to a "done" dir (this folder is excluded form backup too), after we are sure the datas can be restored we delete them from the done folder.



So, we are going for the overflow location solution, the problem now is that the overflow location is defined for an entire storage pool; we are using the same storage pool for backup and archive, I'd like the overflow location to be used for archived datas only. Do I have to create e new storage pool and move arrchives in there? Is it possible to move archives from one stg pool to another? Is there a bettere solution? I'm looking for best pratices....



Thanks!

[chad_small]

Well the first thing you want to do is create a dedicated Archive pool. This way when the tapes within the pool are full you can easily move them to the overflow location. NEVER share backup storage pools with archives. Its a mess when it comes to reclamation and is not easily managed as you are finding out. That can free up alot of space when capacity is an issue. The issue now is how to move the data already archived to the new storage pool. Use the MOVE NODEDATA command to facilitate this by using it on the clients that do the archiving and specifying data type of archive to be moved. Easily done. Can do it for the whole client or for individual filespaces.

[DiegoC]

Hi Chad,



Ok I agree, I created a new disk pool called spd_arch and a new tape pool called spt_arch. spd_arch uses spt_arch as "next storage pool". I added "update stg spd_arch hi=0 lo=0" to my migration_on script and "update stg spd_arch hi=90 lo=2" to my migration_off script in administrative schedules. This should migrate data from disk storage to tape storage during tivoli's "freetime".



Than in the "policy set"/"management class"/"archive copy group" associated with the node I use to archive I changed "copy destination from "spd_back" (the backup pool) to spd_arch (the new archive pool).



From now on every archive I do on the node sould go to spd_arch and they will be migrated to spt_arch.



Did I miss sometihing? Is everything right?



Now I sould try something like "move nodedata archive fromstgpool=spt_arch tostgpool=spt_arch" (archive is the name of the node I use to archive datas and it does archive only, no backup at all for this node) right? I'm a bit afraid in issuing this command! I have more than 22Tb of datas to move ("select node_name,sum(physical_mb) from occupancy group by node_name order by 2 desc" gives me 23095650.56 for node archive!), will tivoli move all this datas on different tapes? This is going to take ages! Moreover I'm quite sure they won't fit my scratches (!



Please forgive me I'm still a newbie in tivoli adminitration!



Thanks

Diego.

[dlaw51]

Another option you may want to consider is making a backup set.

This will take a "snapshot" of the filesystems at the point of the last backup and

write it out to tape with a retention that you set. Once written, the tapes

can be removed from the library. The advantage of this method is that the

backup sets are self-describing, which means that the db is not involved, so this

can prevent db bloat, especially with the amount of data that you are exporting.



The disadvantage is that, to the best of my knowledge, you have to restore

everything if you want to restore anything. Another disadvantage is that only

active files are written to the backup set, so you could only restore the filesystem

to the point of the last backup, but not to any point before that.



If you want to save _everything_, you can use export node to tape with filedata=all

which results in everything being written out to self-describing tapes as well.



I tend to use these methods for long-term archives with a low probability of requiring

restores.

[DiegoC]

Just to close this thread, I solved the problem, Thank you guys!



Here what I did:

- created a new tape pool called spt_arch with an overflow location I called "tapestorage"

- created a new disk pool called spd_arch with spt_arch as next storage pool

- created a new managemente class called mc_archive with an archive copy gorup going to spd_arch

- edited my archive cron script as:

archive /archivedir/ -des="archivename" -optfile=/var/TivoliArchive/dsm.opt -archmc=MC_ARCHIVE >log.txt

- moved all old archive datas form the "normal" storage pool to the archive pool with:

move nodedata NODENAME fromstgpool=SPT_BACK tostgpool=SPT_ARCH filespace=/archivedir type=archive

- Used move:

media * stgpool=SPT_ARCH

to move all archived tape aout of the library



Now, when I need to restore something I get a request on the tivoli console for the tapes it needs.



And I have plenty of space for new scratch tapes!