I have been looking in the bpdbm log and have found a record that has the following;
10:26:25.419 [23129] <16> delete_expired_backups: Bad image header: sfd-BV2105-linux-dd_1306900839_INCR, error: file read failed (13)
10:26:25.419 [23129] <2> IsCatalogCleanupTerminated: Terminated = 1
Looks to me like that is the problem image. Now what I can’t find is a doc going through how to remove just one image. Anyone have or know of a document that has instructions to remove one image?
Thanks
Wayne BeDour
Unix System Administrator
PH: 248-447-1739
Internet: wbedour AT lear DOT com
Sounds to me like image cleanup doesn't like something with a particular client's image. As you know, when /usr/openv fills up, nasty things can happen, including corrupt/incomplete catalog entries. I couldn't find a specific technote that addresses this in my quick search, but this one may help you.
http://www.symantec.com/business/support/index?page=content&id=TECH55691&key=15143&actp=LIST
It deals with bpdbm core dumping when it encounters the bad image, which would be a good thing to verify (though I think you would already know that). Basically, take a look through BPDBM log to see if you can find the specific image that is causing issues. You can also run image cleanup for a specific client by running bpimage -cleanup -client <offending client name>, which may help the log searching if nothing else.
Once you find the image you will have to clean it up manually. Be careful when cleaning it up, as you don't want to remove any good entries.
Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693
Keeping People and Information Connected® ▪ http://availability.sungard.com/
P Think before you print
CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system.
All:
Our environment, HP-UX 11-31 currently running one master / media server on NetBackup 6.5.2. We are running mostly unix / Linux backups and a couple windows b/u’s thrown in for good measure.
Currently, our image cleanup job hangs at the same place and will not complete without canceling it. The Detail Status shows it processing the same client when it hangs.
Here is what got us to this point. Last Thursday one of our admins kicked off a restore and didn’t notice that the window was for a few years. He went home with the search still running. Sometime overnight the /usr/openv filesystem filled up. The admin that kicked off the job didn’t work Friday so the rest of us didn’t know what was going on. (We have since flogged him and he is currently tied up..) NetBackup was taken down and brought back up. When NetBackup came up, we couldn’t submit jobs because the nbperm daemon wouldn’t stay up. We found a document that said “nbjm and nbpem processes are not synchronized when this occurs, stale data is left both in the pempersist file and the cached schedule information. As a result, nbpem never runs the job again, since its data indicates that the job is still active”. We followed the instructions and cleaned up the /usr/openv/netbackup/db/jobs/pempersist file and also had to clear lock files and the activity log database and deleted the following files: 063008.bpjobd.db , bpjobd.act.db , bpjobd.db , jobid , jobid.lock pempersist2. The bad news was that it cleared out the activity log for the 2 weeks which we can live with.
So, that’s where we are, anyone know how to take care of the image cleanup hang?
Thanks in advance..
Wayne BeDour
Unix System Administrator
PH: 248-447-1739
Internet: wbedour AT lear DOT com
|
**********************
** LEGAL DISCLAIMER **
**********************
This E-mail message and any attachments may contain
legally privileged, confidential or proprietary
information. If you are not the intended recipient(s),
or the employee or agent responsible for delivery of
this message to the intended recipient(s), you are
hereby notified that any dissemination, distribution
or copying of this E-mail message is strictly
prohibited. If you have received this message in
error, please immediately notify the sender and
delete this E-mail message from your computer. |
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
