Oops, the filesystem that contains db/images (and also logs) filled for
about 10 minutes the other day.  I received several errors about bad
headers in the images.  Some of them (3) are associated with oracle rman
full backups.  The client side logs suggest that all the backups ran
successfully, but of course without the image headers, they would be
difficult to restore.

I've removed the zero-length image header files and their
corresponding file files, and restarted netbackup.

Now, I think I need to find any tapes responsible and do a bpimport.
According to available_media, I only have 7 tapes set for infinite
retention (used by the full backups) and in the oracle pool.  I've run
every one of those tapes through a bpimport -create_db_info -id $tape -L
> log/$tape.  According to the output I have, only one of my "bad"
images is on those tapes and available for reimport.  That image is the
final image that appears on the tape.

So, is there anything else I can do to narrow down where these images
might be?  Is there any chance that there are other images on that one
tape that bpimport isn't finding?  Could anything have been overwritten
or destroyed?  I'm guessing not because only the netbackup image
directory is on this filesystem and not anything associated with the volume
management stuff.

These are 3 of the bad headers...

Bad image header: ora01_oraclehot_full_1020481630_FULL
Bad image header: ora01_oraclehot_full_1020521749_UBAK
Bad image header: ora01_oraclehot_full_1020521933_UBAK

The first time (4th around 20:00) was several hours before the problem.
I'm guessing it's some sort of "master" image for the backup as a whole,
and failed when the filesystem was full at the end.  The _UBAK ones seem
to be 2GB chunks from rman.  The only one I found with bpimport was the
middle one.  I didn't find the last one or the first one.

These are DLT7000 tapes, so it's taking about 30 to 45 minutes or so to
run through a bpimport run.  I've got enough tapes that I can't do that
for everything in the library.

Thanks for any additional information.   
-- 
Darren Dunham                                           ddunham AT taos DOT com
Unix System Administrator                    Taos - The SysAdmin Company
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >