RMAN not deleting old backups from TSM

[putukas]

Hello,

We have problem where RMAN is not deleting old backups from TSM db.

TSM server version 6.1.3
TDP client 5.3

Here's the RMAN script which is runned every day. Also we have archive logs backup running once per hour.

run {
allocate channel t1 type 'sbt_tape' PARMS 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';

backup
format 'df_%t_%s_%p_%u_%c'
(database);
backup archivelog all delete all input format 'af_%t_%s_%p';
backup current controlfile format 'cf_%d_%u';
backup spfile;

release channel t1;
}
allocate channel for maintenance device type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';

crosscheck backup;
delete noprompt expired backup;
delete noprompt obsolete;
crosscheck archivelog all;
delete noprompt expired archivelog all;

release channel;

TSM info:
vere=no limit, verd=0 rete=7 ret=0 backupdelete=yes

we want to keep RMAN backups for 7 day.

Anyone seeing anything wrong about this? Strange thing is that oracle dba is saying that he can see 7 day backups from RMAN. But i can see alot of old backups not deleted in tsm db. So DBA is saying theres something wrong with TSM and im saying theres something wrong with RMAN

oracle dba is using "allocate channel for maintenance" to delete backups but IBM manual says "allocate channel for delete" oracle dba is saying it doesnt matter and if it does hes demanding explanation from me.

Thank you!

 

[Efim]

Hi

You set wrong TSM policy. The policy must be:
vere=1, verd=0 rete=0 ret=0
All RMAN objects are ACTIVE object so it never expired from TSM.
When you run delete noprompt expired backup and delete noprompt obsolete; RMAN delete ACTIVE object from TSM DB.

Efim

 

[putukas]

ok i have set it to
vere=1 verd=0 rete=0 ret=0 backupdelete=yes

lets see how it goes tonight

 

[putukas]

still no luck

TSM db still has all the old backups. How can i get it deleting older then 7 day backups as rman is deleting?

 

[moon-buddy]

Has "Expire Inventory" ran yet?

 

[putukas]

yep its running every morning

 

[mikeatkc]

Did you setup your TDP node to allow delete of archive and backup? Do a query node <NODE-TDP> f=d and view the full setting.

 

[GregE]

This is what Efim is referring to:
http://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/topic/com.ibm.itsmfd.doc_5.4/ab5w000250.htm

What is your RMAN retention setting? (in RMAN, not in TSM)

RMAN> show all;

RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 35 DAYS;

 

[TonyB]

The only data RMan knows about is that which is in its catalog/control file. If RMan has already expired the objects from its catalog, but the inactivate failed (at the time) with TSM, then the two are out of sync.

There is a utility called tdposync which can resync the two repositories - might be your best bet.

T

 

[putukas]

All oracle backups are in active_version state. RMAN retention policy is set to 7 days and if DBA lists backups from RMAN he can see only 7 day backups but if i do
"select * from backups where node_name='NODENAME' order by backup_date desc" in tsm i can see all backups still in ACTIVE_VERSION state.
tdposync is not in use because oracle rman is not using catalog but controlfile. I'll talk about start using tdposync with dba.

i understand that old backups wont go to inactive when they are already gone from RMAN controlfile but why new backups are not going to inactive state with the rman script above and tsm settings i have posted.

 

[Efim]

Hi

This is just points to check:

1. Run dsmadmc -console
2. Ask you Oracle admin to delete (oldest) backupset using command:
change backupset 19 delete;
During delete he must use the same environment as during backup ('ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';
3. During delete you can see connection to TSM (p.1)
ANR0406I Session 91 started for node ORACLE_DB (TDPO Linux86) (Tcp/Ip 192.168.75.128(58460)).
ANR0403I Session 91 ended for node ORACLE_DB (TDPO Linux86).
4. After that use command:
select * from backups where NODE_NAME='<nodename>'
for check that state changed to INACTIVE_VERSION.
5. You can see in RMAN log:
allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: sid=14 devtype=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: Data Protection for Oracle: version 5.5.2.0

deleted backup piece
backup piece handle=DB.20100121_21_1_0ll414gs_1_1.bkp recid=19 stamp=708874782
Deleted 1 objects

P.S In your script you save control files to TSM.
During disaster you can't restore those files from TSM without those files
you can use script like this:

sqlplus '/as sysdba' <<EOF >> //<path>/backup_log.log
alter database backup controlfile to '//<path>/ctlfile.ctl';
exit
EOF

and after that backup or archive control file + spfile using BA Client

Efim

 

[crooked]

When you registered the node did you setup backdel=yes ?

This is required to enable rman to remove the data.

 

[putukas]

if we do it manually like Efim said i can see backups going to inactive_version state but it is not working with script running at night.

 

[Efim]

if we do it manually like Efim said i can see backups going to inactive_version state but it is not working with script running at night.

It means that you have correct TSM configuration.
Now you have guide how to troubleshoot TDP for Oracle.

Efim

 

[n9hmg]

tdposync is the correct way to hunt down extra backups.
BTW:rete won't keep them from being deleted by RMAN. They're not being marked expired - they're being deleted.
If your DBAs dig in their heels and refuse to learn to use tdposync, get a printout of all the RMAN objects for all the nodes, then select NODE_NAME, FILESPACE_NAME, HL_NAME, LL_NAME, and OBJECT_ID from BACKUPS for just the nodes and filespaces in question, and remove the objects the DBAs say should be there, recheck the list for oddities, and use delete object on the object ids that should not be there. I have to do that every few months... deleted 17TB last time around.

 

[putukas]

rman is still not deleting it's backups from TSM db and im kinda clueless. I'll paste the RMAN scripts we are using atm. Maybe someone of you can find bugs in it.

Full backup, this is command runned by tivoli every night.

$ cat /usr/tivoli/tsm/client/oracle/sched/schedbkdb.scr
#!/bin/ksh
su oracle -c /opt/app/oracle_php/oracle/scripts/backup_sched.scr

$ cat /opt/app/oracle_php/oracle/scripts/backup_sched.scr
#!/bin/ksh
. /home/oracle/.oraprofile_php
rman target rmanbackup/rmanbackup@php_dedicated cmdfile /opt/app/oracle_php/oracle/scripts/backup.scr msglog /opt/app/oracle_php/oracle/scripts/logs/backup.log

$ cat backup.scr
run {
allocate channel t1 type 'sbt_tape' PARMS 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';

backup
format 'df_%t_%s_%p_%u_%c'
(database);
crosscheck archivelog all;
backup archivelog all delete all input format 'af_%t_%s_%p'; sql "alter database backup controlfile to ''/php_flash/oradata/php/backupset/controlfile_backup_binary.ctl'' reuse"; sql "alter database backup controlfile to trace as ''/php_flash/oradata/php/backupset/controlfile_backup_human.ctl'' reuse"; backup current controlfile format 'cf_%d_%u'; backup spfile format 'spf_%d_%u';

release channel t1;
}
allocate channel for delete device type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';

crosscheck backup;
delete noprompt obsolete;

release channel;

And this is for archive logs runned every hour.

$ cat /usr/tivoli/tsm/client/oracle/sched/schedbkarch.scr
#!/bin/ksh
su oracle -c /opt/app/oracle_php/oracle/scripts/backup_archsched.scr

$ cat /opt/app/oracle_php/oracle/scripts/backup_archsched.scr
#!/bin/ksh
. /home/oracle/.oraprofile_php
rman target rmanbackup/rmanbackup@php_dedicated cmdfile /opt/app/oracle_php/oracle/scripts/backup_arch.scr msglog /opt/app/oracle_php/oracle/scripts/logs/backup_arch.log

$ cat /opt/app/oracle_php/oracle/scripts/backup_arch.scr
run {
allocate channel t1 type 'sbt_tape' PARMS  'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';

backup archivelog all not backed up format 'af_%t_%s_%p'; sql "alter database backup controlfile to ''/php_flash/oradata/php/backupset/controlfile_backup_binary.ctl'' reuse"; sql "alter database backup controlfile to trace as ''/php_flash/oradata/php/backupset/controlfile_backup_human.ctl'' reuse"; backup current controlfile format 'cf_%d_%u';

release channel t1;
}

 

[itcsge]

i do this as a manual process... but will be scripted soon.

1. allocate channel for maint.
2. resync catalog
3. delete force noprompt obsolete

 

[putukas]

my backups are now deleting. Im not sure what dba did. Only thing he told he changed is that he separated backup deleting script from main rman script.

 

[rowl]

Hello,
Oracle dba is using "allocate channel for maintenance" to delete backups but IBM manual says "allocate channel for delete" oracle dba is saying it doesnt matter and if it does hes demanding explanation from me.

I was wondering the same thing.

I am looking at a similar problem. Many Oracle TDP clients that should have 30 day backup retention but when I search through the backup table I find backups going back years. It looks like rman is deleting any reference to backups older than 30 days in it's catalog, but this is never communicated to TSM.

The last two lines of the rman script are:

allocate channel for maintenance device type 'SBT_TAPE' parms 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/tdpo.opt)';
delete noprompt obsolete recovery window of 31 days device type 'SBT_TAPE';
exit;

 

[zack]

my backups are now deleting. Im not sure what dba did. Only thing he told he changed is that he separated backup deleting script from main rman script.

Hello,

I am encounter the same problem. RMAN still unable to delete all the old backup set. TSM still detect Active state for the old RMAN Backup set.
Any advice on this?

Thanks,
zack