Oracle Backup - Mountpoint Issue

[GregE]

Solaris 10, client and TSM server
TSM server 5.4.5.0
TSM BA client 5.4.1.5
Oracle TDP 5.4

I have some Oracle RMAN backups for which I have set the node to "MAXNUMMP=1." We only want these backups using one tape drive each. This has been working fine.

Today I see this in one of the Oracle RMAN output scripts......

ORA-19502: write error on file "bk_82918_1_705114015", blockno 3585 (blocksize=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
ANS0326E (RC41) This node has exceeded its maximum number of mount points.
channel t1 disabled, job failed on it will be run on another channel
channel t2: finished piece 1 at 09-DEC-09

In the script, the backup appears to continue. The only error is on that particular file, yet the error points to something regarding the media manager and mount points. What could be causing this? Should I set my node "KEEPMP=YES" to keep the mount point for the duration of the backup? I'm not sure why this would make a difference, but someone else might know more.

 

[Harry_Redl]

Hi,

how many channels did you allocate? Can we see the RMAN script?


Harry

 

[GregE]

Well, the RMAN script allocates 2 channels right now. I Know I have to reduce that to match the mountpoints. That's something we've lived with for a few weeks and we do get one error on those backups because of the channel allocation being 2, with mountpoints being one. We'll correct that later though. This is the "normal" error we're ok with because we know why it exists...

channel t2: starting piece 1 at 08-DEC-09
RMAN-03009: failure of backup command on t2 channel at 12/08/2009 02:01:19
RMAN-10038: database session for channel t2 terminated unexpectedly
channel t2 disabled, job failed on it will be run on another channel

But the error in my initial post is something different that we've not seen, so I am wondering why that is. Aside from that one file, that backup seems to have completed everything else.

 

[GregE]

It did this again on a database last night. One file did not backup due to this error, but this database backs up fine all the time otherwise. I set KEEPMP=YES on all of the Oracle nodes last week, so this is set on this one, but didn't seem to make any difference. It looks like it actually does not backup the file it errors on. It doesn't try again. I can't change my scripts right now as we are in an end-of-year change freeze.

I could understand if this happened everytime this database backs up, but it doesn't. It's only occasionally, which I don't understand the reason for.

 

[Mikey D]

Greg I've actually seen something weird like this on my SUN backup servers. For some reason when Oracle is finishing a file piece and getting ready to send the next one in it doesn't seem to release the tape drive in time for the new session and causes a maximum number of mount points exceeded message. Not sure if the same thing is happening to you but you'd probably have to watch the backup job to confirm.

We started seeing it since the 5.4 version of TSM and have never been able to get an answer on why it's happening from IBM.

 

[GregE]

Thanks for the reply. Interesting you get it too. So, when I'm able to change my script to only open one channel (to match my mountpoints), it won't stop this from occasionally happening. That's ugly because as I look down thru the RMAN log that is generated, the file that it doesn't backup, it doesn't try again later, so apparently it misses it altogether. Here's the RMAN output, just fyi....

allocated channel: t1
channel t1: sid=541 devtype=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.4.1.0


allocated channel: t2
channel t2: sid=272 devtype=SBT_TAPE
channel t2: Data Protection for Oracle: version 5.4.1.0


Starting backup at 16-DEC-09
channel t1: starting incremental level 0 datafile backupset
channel t1: specifying datafile(s) in backupset
input datafile fno=00136 name=/oracle/data/DB1/dmdata.bdf
input datafile fno=00072 name=/oracle/data/DB1/tax_archive/txdata_08.dbf
input datafile fno=00073 name=/oracle/data/DB1/tax_archive/txdata_07.dbf
input datafile fno=00071 name=/oracle/data/DB1/tax_archive/txdata_09.dbf
input datafile fno=00021 name=/oracle/data/DB1/sharch_itemdiv.bdbf
channel t1: starting piece 1 at 16-DEC-09
channel t2: starting incremental level 0 datafile backupset
channel t2: specifying datafile(s) in backupset
input datafile fno=00070 name=/oracle/data/DB1/shdata_03.dbf
input datafile fno=00084 name=/oracle/data/DB1/snapshots_04.dbf
input datafile fno=00150 name=/oracle/data/DB1/shindx_ft_06.dbf
input datafile fno=00077 name=/oracle/data/DB1/tax_archive/txdata_03.dbf
input datafile fno=00099 name=/oracle/data/DB1/shdata_04.dbf
channel t2: starting piece 1 at 16-DEC-09
RMAN-03009: failure of backup command on t1 channel at 12/16/2009 01:00:38 <--------------- problem
ORA-19502: write error on file "bk_83330_1_705718831", blockno 3585 (blocksize=512) <-------- problem
ORA-27030: skgfwrt: sbtwrite2 returned error <----------------------------------------------- problem
ORA-19511: Error received from media manager layer, error text: <----------------------------- problem
   ANS0326E (RC41)   This node has exceeded its maximum number of mount points. <-----------problem
channel t1 disabled, job failed on it will be run on another channel   <------------------------ this one I already can fix later - unrelated
channel t2: finished piece 1 at 16-DEC-09
piece handle=bk_83331_1_705718831 tag=DB1_HOT comment=API Version 2.0,MMS Version 5.4.1.0
channel t2: starting piece 2 at 16-DEC-09
channel t2: finished piece 2 at 16-DEC-09

 

[Mikey D]

Yeah not sure if yours is the same thing but it already sounds like it from the messages and descriptions. We ended up just bumping the maxnummp for the Oracle nodes one higher to compensate for it.