Library getting full / Remove archive data from a storage pool?

[Brinky]

I have a problem and I would like to know what a good solution would be.

Problem: We are running out of tape storage space in our 3584 library and it already has an extension on it. What can I do to make more space inside the tape library?

Background/Environment Setup:
-5 TSM servers (about 400 client nodes) back up to a diskpool (SAN disk on each TSM server). Then a backup of the diskpool data is made to a copypool (offsite tape). Then the data in the diskpool is migrated to the tapepool (onsite tape within the 3584 tape library).
-The library is not partitioned in any way
-All data (incrementals and archives) is mixed in the same pools, meaning we don't have an archivepool separate for archive data.
-Basically our tapepool and copypool are identical, 1 is onsite tape, and 1 is offsite tape (both have all data).
-Have done collocation to save space within library about a year or so ago/

What are my options? This is how I see it, please help.
-Add another library expansion (I don't think mgmt will approve, but it's an option). This is possible, right?
-And I am am not sure what my other options are??? It is unnecessary for our archive data to be both the tapepool and copypool. Is there a way to delete archive data from the tapeool and still keep it in the copypool?

Please help.....I need some suggestions

 

[cheekmp]

It's looking like I'm going to be facing the same problem in the next month or two with my 3584. I doubt I'll be able to add another frame. I'm starting to think about just checking out my archivepool volumes to make room for more tapepool volumes.

I'm curious to hear what other folks do in this situation.

 

[Brinky]

cheekmp - I was thinking of doing the same thing. The problem with just checking out archivepool tapes for me is that I don't have an archivepool. All of my data is mixed together (progressive incrementals and archives).

 

[cheekmp]

Could you create an distinct archivepool and then move the archive data to the new archivepool? Then you'd be able to checkout those archivepool tapes. Reclamation of the tapepool would then presumably reduce the size of your tapepool. Just a thought.

I'm probably just going to checkout my entire archivepool and put them in a safe. I need slots!

 

[Brinky]

That's possible. Is there a way to move just archive data from one pool to another pool?

 

[cheekmp]

I've never done it, but it looks like you might be able to move your nodes' archive data from one stgpool to another stgpool using the "MOVE NODEDATA" command. Since you have 400 nodes, it might take a while (and be tedious). You should be able to script it. Take a look at the COLLOCGroup option so you don't have to specify each node explicitly.

Matt

 

[rore]

Hi,
Yes, depending on your 3584 model you can add another expansion.

Yes, you can separate your archive data to another pool:
First create another primary pool (and copy pool) for you archives and modify your policy so all new archives (and copy) go to those pools. And when your TSM and library have free time (I am presuming not during the night) move your archive data using 'move nodedata'.
Finally re-evaluate your policy retentions and exclusions. This could be hard for 400 client nodes but a bad initial setup will always bring problems.

Rudy

 

[Brinky]

Thanks for the suggestions rore and cheekmp.

rore - I understand creating a new primary pool for archive data, but is it really necessary to create another copypool? Couldn't I just use my current copypool?


Also, I am open to any other suggestions. I am talking with mgmt about this early next week, so i would like as many options as possible.

 

[rore]

Thanks for the suggestions rore and cheekmp.

rore - I understand creating a new primary pool for archive data, but is it really necessary to create another copypool? Couldn't I just use my current copypool?

No, it is not necessary but you might want to do it to separate it as well. Specially (but not limited to) if you manage different retention policys for backups and archives. Trying to follow best practices.

Rudy

 

[Jacob_6]

Take a look at the move media command in the reference guide, this allows you to move volumes in and out of the library according to specific criteria. Support used to have a pretty good documen on it, however, I could not locate it on the IBM Website.

http://publib.boulder.ibm.com/infoc...oc/b_adminref_aix230.htm#dqx2r_cmd_media_move

But it basically allows tsm to track volumes that have been removed and stored in an overflow location.

 

[Brinky]

Thanks rore. Sometimes I wish I knew best practice. I am the lone TSM Admin/Engineer here with knowledge of TSM only by research and implementation over the past year. The previous 2 persons in charge of this environment have left the company and this was put in my lap. Unfortunately TSM is not the direction my company wants to move forward with in the future, so my eagerness to become certified/more knowledgable about the product has diminished. I only get about 5 hours a week to really work on TSM, unless something is wrong, then it becomes top priority. This will probabaly be a case where it is top priority.

Another question rore - What if I did a move nodedata of archive datat from my tapepool to my copypool, even though my copypool already has the archive data there. Will this remove the archive data from the tapepool? It would be really awesome if it did.

 

[rore]

Thanks rore. Sometimes I wish I knew best practice. I am the lone TSM Admin/Engineer here with knowledge of TSM only by research and implementation over the past year. The previous 2 persons in charge of this environment have left the company and this was put in my lap. Unfortunately TSM is not the direction my company wants to move forward with in the future, so my eagerness to become certified/more knowledgable about the product has diminished. I only get about 5 hours a week to really work on TSM, unless something is wrong, then it becomes top priority. This will probabaly be a case where it is top priority.

Another question rore - What if I did a move nodedata of archive datat from my tapepool to my copypool, even though my copypool already has the archive data there. Will this remove the archive data from the tapepool? It would be really awesome if it did.

Well, the direction of your company should understand that critical data loss cannot be recovered magically. A well planned and maintaned storage/backup environment addresses the challenge.

You cannot move data from primary pool to a copy pool. Instead you do a 'backup storage pool'. I think you want to have your archive data just in your copy pool. Although it is possible (using the update vol access=destroyed) it is not a 'recommended state' for your data. A copy pool is that, a 'copy' of something that (should) exists (the primary pool).

Rudy

 

[Brinky]

I agree with you rore. We are looking to phase out use of tape and only use disk with an open source backup solution such as Amanda or Bacula to suit our backup need in the future is the direction I am told.

That makes sense that I am unable to completely move data from a primary pool to the copy pool. The destroyed option for me wouldn't work since the data on my primary storage pool is mixed (incrementals and archive). I don't think I would even feel comfortable doing that. As of now, it looks like my only option is to separate the data with archive pools. I'm thinking a primary archive pool (SAN disk) to catch the archives each night (if any). Then a copy pool that is really just offsite storage of what was backed up the previous night. Then I could move the data from my current tapepool to the archive copy pool. Sound good? Or am I missing something.

 

[Canuck]

You said you're doing collocation? Are you doing collocation by node or by group? If it's by node then that's why you're running out of space -- each node gets its own tape and is probably only using 10 - 20% of the tape space.

What does your tape utilization look like when you run this statement (enter your stgpool name between the quotes):

select volume_name, pct_utilized from volumes where stgpool_name='YOUR_PRIMARY_STORAGE_POOL_NAME' order by pct_utilized desc

And how many nodes per tape do you have if running collocation by group:

select count(distinct node_name) as "Number of Nodes", volume_name from volumeusage where stgpool_name='YOUR_PRIMARY_STORAGE_POOL_NAME' group by volume_name order by "Number of Nodes" desc

If collocation by group isn't working properly either then you'll have quite a few tapes with only one node on them.

 

[Brinky]

Thanks for the info Canuck!!

I am doing collocation by group. Based on the sql queries I have several tapes that only have 1 on them. But there are also some that only have 2 and 3 nodes per tape.

I went back and checked if there were any nodes that weren't set up in collocation groups, and it turns out there are. So I will need to create some more collocation groups to free up some space.

Also, on one of my TSM servers there are several 1 node tapes, but every node is being collocated by group. If collocation isnt working correctly...how do you suggest I fix it? Possibly by doing some move nodedata commands by collocation groups?

 

[Canuck]

In general I would put about 20 -30 nodes per collocgroup (if using LTO3/LTO4 media). It saves in restore times but uses up more tapes. After you place all nodes in groups then you can run move datas on the volumes or move node datas:

move data C012121L4 recons=yes

move nodedata nodea from=primarypool recons=yes

I prefer move datas and use a script posted earlier from mateinone:

http://adsm.org/forum/showthread.php?p=43138

But just change the script to order by pct_utilized.

Also, double check which nodes aren't in groups using this statement:

select node_name, collocgroup_name from nodes where length(collocgroup_name) is null

If your storage pool was setup for collocation but nodes did their first backup uncollated then that tape would only have one node's data on it. Easiest thing to do is run move datas on those tapes to reconsolidate the data back into the storage pool. And if you run move data or move nodedata, add the recons=yes statement to eliminate wasted space due to expiration.

 

[mateinone]

Like Jacob suggest, I use overflo locations where the library is filling up, it is a good way to manage it, but it can be a little painful. I generally move media based on the day criteria as it is less likely to be needed for a restore.

 

[wim]

Another possibility to get or gain space in the library is to review the retention policies of the backups and archives.
Also reviewing the include/exclude statements might gain some place.
Another thing I come across was an issue with TPD for Oracle where some data was not expired correctly and taking up large amount of space.

Regards. Wim.

 

[Jacob_6]

When you do collocation by group, you have to make sure every node is assigned to a group. Otherwise, the unassigned nodes default to collocation by node:

"
If you specify COLLOCATE=GROUP but do not define any collocation groups or if you specify COLLOCATE=GROUP but do not add nodes to a collocation group, data is collocated by node. Be sure to consider tape usage when organizing client nodes into collocation groups. For example, if a tape-based storage pool consists of data from grouped and ungrouped nodes and you specify COLLOCATE=GROUP, the server does the following:

• Collocates by group the data for grouped nodes only. Whenever possible, the server collocates data belonging to a group of nodes on a single tape or on as few tapes as possible. Data for a single node can also be spread across several tapes associated with a group.
• Collocates by node the data for ungrouped nodes. Whenever possible, the server stores the data for a single node on a single tape. All available tapes that already have data for the node are used before available space on any other tape is used."

Assigning the nodes to collocation groups and them moving the node data should free up some space.

 

[rwhtmv]

Wait until 6.x comes out with data deduplication builtin. Chances are, a large portion of your data in the diskpool and on tape are redundant files. This will make a huge difference. I hear that 6.x is due out by Summer 09.