Problem solved!!  If anybody cares it turns out that the issue had to do 
with session time outs on our firewall.  FYI - our master NB server and 
RMAN db server are one in the same and they sit on the opposite side of 
a firewall from the target Oracle db.  At the beginning of a hot backup 
RMAN initiates communication with the target db , then that connection 
sits idle for the duration of the data backup which in our case was 
about 1.5 hours.  This was proved be simple snoops on said port.  After 
the data is written to tape the RMAN script then tries to reestablish 
communication across that idle session to do the archive log portion of 
the backup.  Our firewall had dropped that session due to the idle time 
out and that caused the failures.

We proved/solved this issue by upping the time out on the port to allow 
for that session to remain active and we now have succesfull hot db 
backups.  Our firewall guys rolled their eyes, shook their heads and 
complained that Oracle doesn't included some very simple keep alive code 
for that connection - I have to agree with them.

Thanks for all the suggestions....
-jeremy




> Anybody seen this before?
> 
> This problem is driving me crazy.  I've ruled out both firewall issues 
> and db version compatibility between the RMAN and target db.  Regular NB 
> cold backups work fine.  Im not a DBA so I am struggling through this. 
> The entire instance backs up just fine over an hour plus then this error 
> is generated just prior to deleting archive logs, I think, and NB errors 
> out with a 23 error.  thanks in advance for any pointers!
> 
> ---snip---
> RMAN-08525: backup set complete, elapsed time: 00:00:50
> RMAN-03026: error recovery releasing channel resources
> RMAN-08031: released channel: ch00
> RMAN-08031: released channel: ch01
> RMAN-00571: ===========================================================
> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
> RMAN-00571: ===========================================================
> RMAN-03002: failure during compilation of command
> RMAN-03013: command type: partial resync
> RMAN-06004: ORACLE error from recovery catalog database: ORA-03113: 
> end-of-file
> on communication channel
> RMAN-06097: text of failing SQL statement: begin dbms_rcvman . resetAll 
> ; :succe
> ss := dbms_rcvman . SUCCESS ; :av
> RMAN-06099: error occurred in source file: krmk.pc, line: 9433
> 
> RMAN>
> 
> Recovery Manager complete.
> 
> Script /export/home/oraprod/scripts/hot_database_backup.sh
> ==== ended in error on Thu May 1 13:32:30 EDT 2003 ====
> 
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu