Single Node Backupset takes more than a week

[mosiac]

We have a backupset schedule that runs on Friday for 1 node. This node contains a lot of data 9tb and it's taking over a week to complete. The schedule would in theory start over on itself if it doesn't finish by 7pm tonight. I know that's not what will happen, but I don't know how to troubleshoot a fix.

This is a backupset that gets dumped to LTO tape and usually all other backupsets that contain multiple nodes finish in 1 to 2 days instead of taking over a week.

Any where I should start looking?

 

[marclant]

My guess is that since other smaller backupsets work fine, it's not a performance problem with the actual data movement. Because this node is really large, it's potentially a DB performance issue.

I'd use this checklist to make sure the database is configured for best performance:
http://www-01.ibm.com/support/knowl...com.ibm.itsm.perf.doc/r_chklist_srvdbdsk.html

Can also use this to try to find the bottleneck:
http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.0/com.ibm.itsm.perf.doc/t_perf_pd.html

 

[mosiac]

Yeah I've recently made another post about my database utilization being too high and following what I've read to fix it hasn't cleared up the issue yet. We've been sitting at 96 percent utilization for over two weeks now, I've added more space but it doesn't appear the DB wants to use it yet.

 

[marclant]

You might need to rebalance the database manually:
http://www-01.ibm.com/support/knowl...m.srv.doc/t_srv_rebalance_manual.html?lang=en

Helps too if all the directories/filesystems are the same size.

 

[mosiac]

I see what you're saying but it seems odd that a database issue would affect 1 backupset but not others. I'm looking at bottlenecks as well. This node is a file server used by a lot of people so it's bandwidth may be somewhat hindered at times.

 

[marclant]

Is the number of objects considerably larger for this particular backupset compared to others? Larger number of objects, means a larger number of database transactions.

Even if it's used by a lot of people, they are accessing the fileserver, the backupset is only accessing the TSM Server database and the storage pools, as well as output tape.

 

[mosiac]

Yes there are much more objects here we're currently over 24million objects. It's just strange that number hasn't grown that much in a month I wouldn't think.

 

[marclant]

Some problems like that creep up. Did you start noticing the performance degradation at the same pace the database started filling up?

 

[mosiac]

Yes. We also noticed that the backupset hadn't finished before but our reports not telling us there were any errors either. So we're in a really strange situation now for sure.