Re: [ADSM-L] TDPO problem - can't delete old tdpo backups

We finally resolved this problem.

I worked with IBM support trying to identify why TSM/TDPO objects would not
go away when we would run TDPOSYNC. We traced a TDPOSYNC run and clearly
saw that the objects in TSM _were_ in rman - TDPOSYNC was working
correctly! Somewhere/somehow the objects were still in rman, but not
easily identified or located. Once we had hard evidence as to where the
problem was, the DBA dug into the RMAN catalog and found the objects. I do
not know what he deleted or how he deleted the data out or RMAN, but he did
it.

Summary: We preformed a RMAN restore which included a Oracle DBID change.
This in effect became a NEW Oracle db in RMAN which was backed up to the
same node in TDPO/TSM. The old backups became stranded in RMAN and TSM.
To TDPOSYNC everything was in-sync since these old objects were still in
RMAN. I personally have not used/run RMAN, so I do not know what this
looked like, but apparently it was not obvious that the old DB objects were
still in RMAN. After the DBA found and deleted these old objects, TDPOSYNC
deleted the TDPO/TSM objects.

Here are the IBM instructions for tracing TDPOSYNC, and further down is a
explanation of just how TSPOSYNC works.

#There is a way we can check the RMAN and the TDPOSYNC outputs.
#The background is in the following technote:
#
http://www-01.ibm.com/support/docview.wss?rs=663&q1=1142277&uid=swg21142277&loc=en_US&cs=utf-8&lang=en

# I want to gather the outputs from this tdposync processing.
# There are some temporary files as well as a trace that will help
#to see the problem. If we trace the tdposync processing, this will
#give us the path that the SQLxxxx files will be placed in and the
#trace will also cause the SQLxxx files to not be deleted at the end
#of the TDPOSYNC SYNCDB processing. This will prevent the
#temporary files from being deleted after the TDPOSYNC completes
#and we can gather them at leisure.

#To trace the TDP Oracle client:
#In the TPDO_OPTFILE: /opt/tivoli/tsm/client/oracle/bin/tdpo.opt you will
need to place the trace variables:
# TDPO_TRACE_FILE /directory/and/Filename
# TDPO_TRACE_FLAGS orclevel2

#Be sure to specify a directory and filename that has writeable permissions
for the Oracle user.
#Then run the "TDPOSYNC SYNCDB"

And, here is a very good explanation from lvl2 support sent of how
TDPOSYNC works.

#During the TDPOSYNC processing, it will initially connect to
#SQLPLUS in order to query the RMAN catalog and pull the entire
#list of backuppieces stored there (placing the list within a temporary
#file in the temp directory). There is no specification for the target
#database. The query list will be for all the objects stored in the RMAN
#catalog. The TDPOSYNC will then connect to the TSM Server and
#verify the list of objects from there based on the NodeName and Filespace.
#The comparison is only one direction, which checks if the object that is
on
#the TSM Server exists in the RMAN catalog. A list is output of those items
#that are on TSM but not within the RMAN catalog (additional objects may
#exist in the RMAN catalog).

Rick

Richard L.
Rhodes/FirstEnerg
y To
"ADSM: Dist Stor Manager"
02/10/2009 03:17 <ADSM-L AT VM.MARIST DOT EDU>
PM cc

Subject
TDPO problem - can't delete old
tdpo backups(Document link: Richard
L. Rhodes)

We've been using rman/tdpo backups for over a year
now, and are still trying to get our heads fully
around it. I've been working with TSM support
on a problem and not getting very far, so I thought
I'd ask the list.

We have several TDPO nodes in a TSM server that have
old backups we cannot get rid of. In all other ways
TDP/TSM is working great. When we run TDPOSYNC it
says the RMAN catalog and TDPO is INSYNC - nothing to
cleanup. But, a list of objects form the backups table
in TSM for the node shows old backups. we keep backups
for 50 days - these old files are well back into last year.

What we think happened:

Back in Oct/Nov/Dec time frame, we did RESTORES on
several databases. According to the DBA's, part
of these restore was Oracle creating a new DBID (Database ID).
In essence, after the restore we had a completely NEW DATABASE.
>From RMAN's perspective, these were NEW DATABASES.
The old objects in TSM that we can't get rid of appear
to be from PRIOR TO THE RESTORE. It's as if both RMAN
and TDPO have no knowledge of these objects. The DBA
who handles rman/tdpo (who is very good!) says that
these old objects are NOT in RMAN. RMAN/TDPO/TSM is
CORRECTLY backing up and deleting backups SINCE the
restore. Also, we setup rman/tdpo so that every
Oracle db has it's own TSM node.

IBM suggested

1) delete them with the BA client. It doesn't work. It seems the files in
TSM have no owner, and the ba client won't/can't see them.

2) rename the node and start over. Yes, I can do this, but it seems to be
running around the problem.

or 3) User error (the dba and myself) - we don't understand something
about RMAN/TDPO/TSM.

It seems to me that I have either a RMAN bug, or a TDPOSYNC bug.
what I wish is that tdposync would list out the full set
of files that it is comparing from rman and tsm.

Yes, I'm aware of the "delete object" command . . . I really don't want to
go there!

=================

Here is an example of a the files in one of the problem nodes:

==> tsmsap2 TDPO-ONSITE SAPEQ1D1_DB
==> q node: SAPEQ1D1_DB AIX TDPO-ONSITE 1 1 No

i0jo7odu_1_1 ACTIVE_VERSION 2008-08-17
i8jo7tuc_1_1 ACTIVE_VERSION 2008-08-17
<deleted lines>
u7jrej66_1_1 ACTIVE_VERSION 2008-09-25
u8jrejb3_1_1 ACTIVE_VERSION 2008-09-25
<deleted lines>
ucjrejk0_1_1 ACTIVE_VERSION 2008-09-25
udjrejlo_1_1 ACTIVE_VERSION 2008-09-25

4mk2gkvp_1_1 ACTIVE_VERSION 2008-12-17
4tk2gv5l_1_1 ACTIVE_VERSION 2008-12-17
<deleted lines>
o0k3bpge_1_1 ACTIVE_VERSION 2008-12-28
o5k3bvvt_1_1 ACTIVE_VERSION 2008-12-28
<delted lines>
6pk410jh_1_1 ACTIVE_VERSION 2009-01-05
6uk4173b_1_1 ACTIVE_VERSION 2009-01-05
<deleted lines>
btk4iavp_1_1 ACTIVE_VERSION 2009-01-11
buk4ib01_1_1 ACTIVE_VERSION 2009-01-11

Thanks

Rick

-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.

Re: [ADSM-L] TDPO problem - can't delete old tdpo backups - RESOLVED