Slow backup General question

[kongfranon]

I was hoping for some general advice on what things I look for if backups all of a sudden this week are really slow, where a backup normally takes an hour takes 12 hours +

All I see on the log on the clients is "waiting for tsm server" messages.

Are there any first steps I should look at?


Thanks!



EDIT: Just to add I know very little about TSM, the guy who normally handles this just quit..and now I am totally responsible for it

 

[moon-buddy]

There are so many things to look at and look for.

One indication why backups are getting slow is that there is contention with backup storage space. Normally, TSM is setup to backup clients to disk first before moving data to tape. If the disk is full and is moving data to tape while clients starts backing up, things slows down. This may happen if no disk to tape data moves have been done the previous day or days.

Check storage pool space: 'q stgp' and see the PCT UTIL values

 

[kongfranon]

Thanks! yes I know it goes to disk first and then eventually goes to tape.

So I ran it, but is this something that I need to check really in middle of night when backups are happening? This looks ok except for maybe Disk Pool2?



Storage Device Estimated Pct Pct High Low Next
Pool Name Class Name Capacity Util Migr Mig Mig Storage
Pct Pct Pool
----------- ---------- ---------- ----- ----- ---- --- -----------
DISASTERCR- LTO4 77,336 G 25.8
YPT1
DISKPOOL1 DISK 2,950 G 43.6 43.6 75 50 DISKPOOL2
DISKPOOL2 DISK 2,940 G 90.5 90.4 99 99 DISKPOOL3
DISKPOOL3 DISK 1,670 G 26.4 25.8 95 90 SPILLCRYPT-
ION1
MAILPOOL1 DISK 320 G 12.7 12.7 90 70
SPILLCRYPT- LTO4 31,038 G 51.9 83.3 90 70
ION1

 

[moon-buddy]

From what you sent me, the disk pools seems to be cascading from one to the other: DISKPOOL1 overflows to DISKPOOL2 then to DISKPOOL3.

I don't understand why this was setup this way.

The normal way is to have a next storage pool (after disk) as q sequential type storage like TAPE.

And, yes, you have to monitor this IF you don't have monitoring or automation in place.

Welcome to the life of a TSM Admin!

 

[kongfranon]

Thanks for the help!

From what I gathered, they had multiple storage disk pools, because they ran out of room on the first storage system, so then each time ran out of room just added another set of storage completely independent from the first. Each storage system is a separate mount point on the backup server. Each storage system is then created into a Disk Pool.

Are you saying the best way to do this is to only have one storage pool which would be one disk , and then that would go to Tape?

Or am I misunderstanding?


Yes there is no monitoring, well we do get a report once a day around mid afternoon that shows how many backups were missed, completed, and some general info about it from the night before.

 

[moon-buddy]

Yes what I am saying is to have one DISK storage pool with multiple volumes and then have the next storage pool as tape.

I try to make my disk pool as large or larger than the expected data to be backed up. I set it to be, as much as possible, 25% larger than the total expected data to be backed up.

 

[kongfranon]

I would like to change that and in fact I can do whatever I want, it pretty much my domain now.

But I guess the best way to do that is buy a much bigger storage that would give me room for growth and add it and then I assume do some sort of migration to the new Storage Pool.


One thing and I am probably not understanding the terminology yet, when you say one Storage Pool with multiple volumes. What are volumes in TSM? I just looked at the Storage Pool in the webgui and I see each of them, but don't see anything about a volume?

I know what volumes are in a IBM SVC it pretty much a separate LUN attached to each host but maybe not understanding in terms of TSM



EDIT: ok i found something that said query storage pool and it listed all of the DSM files. There are about 100 of them each about 10GB each.

 

[moon-buddy]

To see the volumes that belong to a storage pool:

'q vol stgp=<stgpool_name>'

To se storage pools:

'q stgp'

 

[kongfranon]

So i was doing some research and see there were migrations on the diskpool happening several times throughout the night.

So that can definitely cause a disk contention correct and possible slowdown of backups?




One question i always see Migration proccess ended in Actlog, but never see where it starts? I was curious if it was happening around a certain backup for example:

4/20/2012 03:04:11 ANR1001I Migration process 5875 ended for storage pool
AM DISKPOOL3. (PROCESS: 5875)
04/20/2012 03:04:11 ANR0986I Process 5875 for MIGRATION running in the
AM BACKGROUND processed 272 items for a total of 313,028,78-
8,224 bytes with a completion state of SUCCESS at
03:04:11 AM. (PROCESS: 5875)


I did a search in the log and can never find the start time does it record that?

 

[moon-buddy]

There should be all sorts of activity record in the ACTLOG. search for the storage pool name.

You can avoid arbitrary diskpool migration by increasing your disk pool space (addding volumes to it - see 'help define vol') so the total space is greater than the incoming data (see my reply earlier).

With the backup data held on disk, you now backup and migrate to tape during the 'OFF' backup hours (after all nodes have he backups done).

 

[kongfranon]

Thanks!


If increasing disk spool space is not physically possible at the moment would changing the thresholds on the migration disk pool help?


I am thinking if i lower it would migrate faster during the day.

 

[moon-buddy]

Lowering threshold would always trigger migration every now and then. Generally you should not have any major backups during the day except for database LOG backups (as the majority backups). If you really have very low disk space, and run frequent backups diring the day, yes go ahead and lower the threshold.

What you can also do is combine the DISKPOOL2 and DISKPOOL3 with DISKPOOL1 to have a bigger pool! Of course, this is assuming that 2 and 3 are the next storage pools.

 

[kongfranon]

Yes DISKPOOL2 and DISKPOOL3 would be the next.

I just don't know how to combine them physically without destroying whatever data is on there first? is there some sort of command that can do that easily?

 

[moon-buddy]

To combine DISKPOOL2 and 3, you have to migrate all data out of DISKPOOL2 and 3.

You can update the storage pool (DISKPOOL2 and 3) such that the next storage pool is the tape. See help update storage pool. This also means that you have to update DISKPOOL1 such that the next storage pool is also tape.

After doing this, and no backup or restore is happening, issue the command migrate stgpool DISKPOOL2 lo=0 and this should move all data to tape. Do the same for DISKPOOL3 (see help migrate stgp). If cache is ON for this pools, you will see residual data. For this not to happen, set cache migrated data to NO (see help update stgp)

When all data has moved, delete DISPOOL2 and 3 and use the volumes on this disk pools to be used for DISKPOOL1 - see help define vol.