Veritas-bu

[Veritas-bu] Backup jobs not kicking off any more....

2007-03-13 16:09:38
Subject: [Veritas-bu] Backup jobs not kicking off any more....
From: cjmanders at lbl.gov (Christopher Jay Manders)
Date: Tue, 13 Mar 2007 13:09:38 -0700
Hi Len,

Yes, I know what you are talking about. And, it makes sense that it 
would be something like that.

I doubt that anything was tun on the system like that.

The support folks are focusing on the nbpem right now.
nbpemreq -tables
nbpemreq -predict -date 3/13/2007

For awhile they showed a weird error: "Could not narrow object"

Now, after an hour from daemon restart, they show correct output. But, 
everyone was stranged by the fact that it returned the 'Could not narrow 
object' error msg for awhile...

I still get such a message intermittently, BTW.

Other ideas/questions are welcome.

TX!
--Chris



> Chris,
>  
> Did the support folks look at the media end of things? the resource 
> broker could be the problem. That is the piece that sits between the 
> scheduler and the media you are going to write on. If that get's 
> messed up netbackup may believe that it never has anything to write to.
>  
> I do not remember the command but there is an EMM command to suspend 
> backups (like the old bpconfig -tries 0). Is it possible that this 
> command was issued?
>  
> len
>  
>  
>
> ------------------------------------------------------------------------
> *From:* Christopher Jay Manders [mailto:cjmanders at lbl.gov]
> *Sent:* Tuesday, March 13, 2007 3:32 PM
> *To:* veritas-bu at mailman.eng.auburn.edu
> *Cc:* Preston, Douglas L; Len Boyle
> *Subject:* Re: [Veritas-bu] Backup jobs not kicking off any more....
>
> Hi,
>
> Thanks for the responses so far. Doug, yours comes close to what 
> support has suggested, and we have tried it. More on that below... 
> Still no go... :(
>
> A couple of new facts:
> 1 - We are on Solaris, but it is similar enuf for the below.
> 2 - We are at 6.0 MP4
> 3 - We did the following steps, which add to the step you mention below:
>
> Stop NetBackup by exiting any open NetBackup consoles and issuing the 
>following command:
>  
>
># /usr/openv/netbackup/bin/goodies/netbackup stop
>  
>
>- After NetBackup has stopped, run the following command to check for 
>hung NetBackup processes:
>  
>
># usr/openv/netbackup/bin/bpps -a
>  
>
>- Using kill -9 command, stop any remaining hung processes that are 
>listed by the bpps command.
>  
>
>- Verify that all Netbackup processes have been properly terminated.
>  
>
>- Remove the pempersist file: (Note: I would recommend just renaming the 
>original file, or make a copy of this file before proceeding)
>  
>
>The pempersist file location is as follows:
>  
>
>/usr/openv/netbackup/bin/bpsched.d/
>  
>
>- Remove the retirepersist file: (Note: I would recommend just renaming 
>the original file, or make a copy of this file before proceeding)
>  
>
>The retirepersist file location is as follows:
>  
>
>/usr/openv/netbackup/bin/bpsched.d/
>  
>
>- Remove the bpjobd.act.db file: (Note: I would recommend just renaming 
>the original file, or make a copy of this file before proceeding)
>  
>
>The bpjobd.act.db file location is as follows:
>  
>
>/usr/openv/netbackup/db/jobs/bpjobd.act.db
>  
>
>- Start NetBackup again:
>  
>
># /usr/openv/netbackup/bin/goodies/netbackup start
>  
>
>
> So far nothing seems to be working at all. Still no jobs get Queued or 
> Active. Nothing at all appears in the Activity Monitor.
>
> Any other ideas?
>
> I am almost ready to start pulling my hair out.
>
> Tx!!!
>
> --Chris
>
>>Under your C:\Program Files\VERITAS\NetBackup\db\jobs folder there is a
>>file called permpersist stop all your services and delete this file and
>>restart your services.  This should fix the problem
>>
>>
>>
>>Doug Preston
>>Systems Engineer
>>Land America Tax and Flood Services
>>Phone 626-339-5221 Ext 104
>>Email  dlpreston at landam.com
>>
>>
>>------------------------------------------------------------------------
>>------------
>>NOTICE: This electronic mail transmission may constitute a communication
>>that is legally privileged. It is not intended for transmission to, or
>>receipt by, any unauthorized persons. If you have received this
>>electronic mail transmission in error, please delete it from your system
>>without copying it, and notify the sender by reply e-mail, so that our
>>address record can be corrected.
>>------------------------------------------------------------------------
>>------------
>>
>>
>>-----Original Message-----
>>From: veritas-bu-bounces at mailman.eng.auburn.edu
>>[mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of
>>Christopher Jay Manders
>>Sent: Tuesday, March 13, 2007 10:05 AM
>>To: veritas-bu at mailman.eng.auburn.edu
>>Subject: [Veritas-bu] Backup jobs not kicking off any more....
>>
>>Hi,
>>
>>So, we have an 6.0 environment that has three times now over the course
>>of the last 2 months stopped accepting or kicking jobs off. The solution
>>we found previously, before this last Sunday the 11th, was to reboot the
>>master server.
>>
>>Unfortunately, now rebooting does not seem to help.
>>bpbackup -i -p <policy> does nothing.
>>No jobid is assigned and no jobs are showing up in the Activity Monitor.
>>In fact, bperror -U -backstat -by_statcode -hoursago 48 shows that no
>>jobs have gone off for the last 2 days.
>>
>>We are at 6.0 MP4 on all of our systems, and the DST patches were
>>applied awhile back. All system time is correct.
>>
>>Since this issue has happened before the DST thing, I am thinking this
>>is something else altogether, but no logs show anything of any use that
>>I can see, and the front-line support folks have not had any clues at
>>all.
>>
>>I see from the Reports->Problems that the consistent error appears to be
>>connected to the issue:
>>
>>get_string() failed - premature end of file encountered (5) could not
>>process request
>>
>>The error is repeated for each job that would normally fire off. So, the
>>logs are filled with thousands of the repeated error message.
>>
>>Anyone else seen anything like this or have any ideas?
>>
>>TIA!
>>
>>--Chris
>>_______________________________________________
>>Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>
>>_______________________________________________
>>Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>  
>>
>