Veritas-bu

[Veritas-bu] [Solution]: Backup jobs not kicking off any more....

2007-03-13 23:36:09
Subject: [Veritas-bu] [Solution]: Backup jobs not kicking off any more....
From: cjmanders at lbl.gov (Christopher Jay Manders)
Date: Tue, 13 Mar 2007 20:36:09 -0700
Hi,

I thought I would pass this along.

6.0MP4 has one issue with the PEM:

- *Any* media server that is down can/will hang the entire enterprise.

By removing the STU for the down media server all jobs are now working 
as they were prior to this weekend.

Gotta love it. I guess that is why they call it the 'Enterprise' 
version, and why you pay extra for it! ;-) (Just poking fun...a little ;)

Anyway, all backups are again working after I did a 'studel -label' on 
the STU with the down host.

30 hours in 2 days. Fun! :-)

Thanks again for all of the input and help.

Cheers!
--Chris



> Chris,
>  
> Did the support folks look at the media end of things? the resource 
> broker could be the problem. That is the piece that sits between the 
> scheduler and the media you are going to write on. If that get's 
> messed up netbackup may believe that it never has anything to write to.
>  
> I do not remember the command but there is an EMM command to suspend 
> backups (like the old bpconfig -tries 0). Is it possible that this 
> command was issued?
>  
> len
>  
>  
>
> ------------------------------------------------------------------------
> *From:* Christopher Jay Manders [mailto:cjmanders at lbl.gov]
> *Sent:* Tuesday, March 13, 2007 3:32 PM
> *To:* veritas-bu at mailman.eng.auburn.edu
> *Cc:* Preston, Douglas L; Len Boyle
> *Subject:* Re: [Veritas-bu] Backup jobs not kicking off any more....
>
> Hi,
>
> Thanks for the responses so far. Doug, yours comes close to what 
> support has suggested, and we have tried it. More on that below... 
> Still no go... :(
>
> A couple of new facts:
> 1 - We are on Solaris, but it is similar enuf for the below.
> 2 - We are at 6.0 MP4
> 3 - We did the following steps, which add to the step you mention below:
>
> Stop NetBackup by exiting any open NetBackup consoles and issuing the 
>following command:
>  
>
># /usr/openv/netbackup/bin/goodies/netbackup stop
>  
>
>- After NetBackup has stopped, run the following command to check for 
>hung NetBackup processes:
>  
>
># usr/openv/netbackup/bin/bpps -a
>  
>
>- Using kill -9 command, stop any remaining hung processes that are 
>listed by the bpps command.
>  
>
>- Verify that all Netbackup processes have been properly terminated.
>  
>
>- Remove the pempersist file: (Note: I would recommend just renaming the 
>original file, or make a copy of this file before proceeding)
>  
>
>The pempersist file location is as follows:
>  
>
>/usr/openv/netbackup/bin/bpsched.d/
>  
>
>- Remove the retirepersist file: (Note: I would recommend just renaming 
>the original file, or make a copy of this file before proceeding)
>  
>
>The retirepersist file location is as follows:
>  
>
>/usr/openv/netbackup/bin/bpsched.d/
>  
>
>- Remove the bpjobd.act.db file: (Note: I would recommend just renaming 
>the original file, or make a copy of this file before proceeding)
>  
>
>The bpjobd.act.db file location is as follows:
>  
>
>/usr/openv/netbackup/db/jobs/bpjobd.act.db
>  
>
>- Start NetBackup again:
>  
>
># /usr/openv/netbackup/bin/goodies/netbackup start
>  
>
>
> So far nothing seems to be working at all. Still no jobs get Queued or 
> Active. Nothing at all appears in the Activity Monitor.
>
> Any other ideas?
>
> I am almost ready to start pulling my hair out.
>
> Tx!!!
>
> --Chris
>
>>Under your C:\Program Files\VERITAS\NetBackup\db\jobs folder there is a
>>file called permpersist stop all your services and delete this file and
>>restart your services.  This should fix the problem
>>
>>
>>
>>Doug Preston
>>Systems Engineer
>>Land America Tax and Flood Services
>>Phone 626-339-5221 Ext 104
>>Email  dlpreston at landam.com
>>
>>
>>------------------------------------------------------------------------
>>------------
>>NOTICE: This electronic mail transmission may constitute a communication
>>that is legally privileged. It is not intended for transmission to, or
>>receipt by, any unauthorized persons. If you have received this
>>electronic mail transmission in error, please delete it from your system
>>without copying it, and notify the sender by reply e-mail, so that our
>>address record can be corrected.
>>------------------------------------------------------------------------
>>------------
>>
>>
>>-----Original Message-----
>>From: veritas-bu-bounces at mailman.eng.auburn.edu
>>[mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of
>>Christopher Jay Manders
>>Sent: Tuesday, March 13, 2007 10:05 AM
>>To: veritas-bu at mailman.eng.auburn.edu
>>Subject: [Veritas-bu] Backup jobs not kicking off any more....
>>
>>Hi,
>>
>>So, we have an 6.0 environment that has three times now over the course
>>of the last 2 months stopped accepting or kicking jobs off. The solution
>>we found previously, before this last Sunday the 11th, was to reboot the
>>master server.
>>
>>Unfortunately, now rebooting does not seem to help.
>>bpbackup -i -p <policy> does nothing.
>>No jobid is assigned and no jobs are showing up in the Activity Monitor.
>>In fact, bperror -U -backstat -by_statcode -hoursago 48 shows that no
>>jobs have gone off for the last 2 days.
>>
>>We are at 6.0 MP4 on all of our systems, and the DST patches were
>>applied awhile back. All system time is correct.
>>
>>Since this issue has happened before the DST thing, I am thinking this
>>is something else altogether, but no logs show anything of any use that
>>I can see, and the front-line support folks have not had any clues at
>>all.
>>
>>I see from the Reports->Problems that the consistent error appears to be
>>connected to the issue:
>>
>>get_string() failed - premature end of file encountered (5) could not
>>process request
>>
>>The error is repeated for each job that would normally fire off. So, the
>>logs are filled with thousands of the repeated error message.
>>
>>Anyone else seen anything like this or have any ideas?
>>
>>TIA!
>>
>>--Chris
>>_______________________________________________
>>Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>
>>_______________________________________________
>>Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>  
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20070313/200dd0d0/attachment.htm