Oracle Backup Cleanup Taxing TSM Server

[GregE]

TSM server 5.5

Trying to wrap my hands around why our Oracle cleanup routine takes so long. An Oracle DB backup that takes 4-6 hours, has a cleanup script that then takes 12-15 more hours.

#
crosscheck backup;
delete expired backup;
delete obsolete;
#

We only run it once a week. Could this be the problem? Does the crosscheck "mark" records in Oracle so that they are not looked at during the next crosscheck? If that is true, then the DBAs should run this more often. If that is not true, then it will run many hours multiple times a week and doesn't help me. It's destroying (well, its part of a larger list of things) my TSM server ability to complete other backups and do other administrative processing.

Can someone provide some thoughts on this?

 

[moon-buddy]

TSM server 5.5

Trying to wrap my hands around why our Oracle cleanup routine takes so long. An Oracle DB backup that takes 4-6 hours, has a cleanup script that then takes 12-15 more hours.

#
crosscheck backup;
delete expired backup;
delete obsolete;
#

We only run it once a week. Could this be the problem? Does the crosscheck "mark" records in Oracle so that they are not looked at during the next crosscheck? If that is true, then the DBAs should run this more often. If that is not true, then it will run many hours multiple times a week and doesn't help me. It's destroying my TSM server ability to complete other backups and do other administrative processing.

Can someone provide some thoughts on this?

How big are the databases in question?

I had a TSM 5.5.6 environment with about 5T of Oracle backup and no way does the cleanup take this long.

On another thought, why is the built-in RMAN retention policy not used to do automatic cleanups?

 

[gimlet]

Crosscheck should verify that all of the backups in the controlfiles and catalog still exist. I believe you can use a "COMPLETED AFTER" clause to limit what gets checked. Not sure if it will just query TSM to see if the files are there, or whether it will actually read the files. If it does, and you have a long retention, and the files are on tape...

Can you tell which step(s) is taking a long time?

 

[GregE]

How big are the databases in question?

Approximately 2-3 TB.

On another thought, why is the built-in RMAN retention policy not used to do "automatic cleanups?"

Now you've intrigued me. What do you mean the automatic cleanups?

"delete obsolete" is doing that. Are you referring to something different?

Output example. Completion of crosscheck then the start of delete obsolete:

...
backup piece handle=DB1_pqoat0kk_2_1 RECID=633611 STAMP=816743166
crosschecked backup piece: found to be 'AVAILABLE'
backup piece handle=DB1_pqoat0kk_3_1 RECID=633612 STAMP=816743288
Crosschecked 13581 objects


specification does not match any backup in the repository

RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 14 days
Deleting the following obsolete backups and copies:
Type                 Key    Completion Time    Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set           30741883 May 08 2013 21:24:15
  Backup Piece       30741928 May 08 2013 21:24:15 bk_140366_29_814914411
Backup Set           30741883 May 08 2013 21:23:36
  Backup Piece       30741927 May 08 2013 21:23:36 bk_140366_28_814914411
...etc

 

[moon-buddy]

By automatic cleanups, I mean scripted RMAN task that is triggered by retention policies. Cleanup is not done on all databases at the same time.

 

[GregE]

Ok I got it. "delete obsolete" is doing that part.

So thinking more and looking it up, it sounds like crosscheck and delete obsolete are not related. Crosscheck though, IS related to the "delete expired" command as it's crosscheck that is marking the files as expired.