Veritas-bu

Re: [Veritas-bu] Real time failure notification

2009-02-04 17:04:06
Subject: Re: [Veritas-bu] Real time failure notification
From: "Sweeney, Patrick" <Patrick.Sweeney AT axcelis DOT com>
To: "Sweeney, Patrick" <Patrick.Sweeney AT axcelis DOT com>, Travis Kelley <rhatguy AT gmail DOT com>, "judy_hinchcliffe AT administaff DOT com" <judy_hinchcliffe AT administaff DOT com>, "veritas-bu AT mailman.eng.auburn DOT edu" <veritas-bu AT mailman.eng.auburn DOT edu>
Date: Wed, 4 Feb 2009 16:38:04 -0500
I guess I only had two possible solutions.  :)

(Was thinking of using bperror in a fashion similar to bpdbjobs - but I don't 
think there is enough info in the output to filter since last run.)

Patrick Sweeney
 (978) 787-4553
 patrick.sweeney AT axcelis DOT com
 I.T. Systems/Networks

For issues requiring immediate attention please contact the Solution Center
IT Solution Center
 (978) 787-7777
Beverly.Helpdesk AT axcelis DOT com
Axcelis Technologies

Have you searched here and here?


-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Sweeney, 
Patrick
Sent: Wednesday, February 04, 2009 3:55 PM
To: Travis Kelley; judy_hinchcliffe AT administaff DOT com; veritas-bu AT 
mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Real time failure notification

Travis,
I have three possible solutions for you - I think.  I haven't tested the 
example I am providing here, so you'll probably need to do a little debugging.

1. Run BPDBJOBS at a regular interval, and parse for jobs of type backup, whose 
parentid is equal to its own id, with a status greater than 1 and an end time 
greater than the last time you checked.  The batch file below uses gawk 3.1.0 - 
hopefully there are no typos.
You could set this to run as a scheduled task at some appropriate interval - 
like once an hour maybe.
(Careful -lines may wrap)  You could actually do something simpler here if you 
just wrote the whole thing as an awk script - but this should work.

REM DETERMINE THE LAST Time this was run
REM We will use this as the cutoff time and ignore older errors
FOR /F %%i in (LASTRUN.TXT) do (
 SET LASTRUN=%%i
)

REM Now record the current time for the next run
ECHO. | gawk "{print systime()}" >LASTRUN.TXT

REM Now run bpdbjobs and find backup errors since the last job, 
REM but only if the parentid is the same as the jobid
REM for more info on the fields returned by bpdbjobs take a look at 
REM the "Veritas NetBackup Commands; Windows; Release 6.5"

bpdbjobs -most_columns |gawk -F "," "{if ($34 == $1 && $2 == 0 && $3 == 3 && 
$11 > %LASTRUN%) print $1,$4,$5,$6,$7,$11}" >NewAlerts.txt

For /f %%i in (NewAlerts.txt) do (
 ECHO JOBID = %%i > %%i.error
 ECHO EXIT STATUS = %%j >> %%i.error
 ECHO POLICY = %%k >> %%i.error
 ECHO SCHEDULE = %%l >> %%i.error
 ECHO CLIENT = %%m >> %%i.error
  call %NB_MAIL_SCRIPT% whoever AT wherever DOT tld "Backup on %%m Exited with 
Status %%j" %%i.error
 DEL %%i.error
)

2. I have posted a set of bpstart_notify.bat and bpend_notify.bat scripts 
intended to perform a set of actions only once.  These are at 
http://www.netbackupscripts.com/index.php?topic=4.0
My production environment isn't as intense as some, so your mileage may vary 
with these.

I think these could be modified as follows to satisfy your requirement:
In the bpend_notify.bat file
Echo the exit status of each stream to an individual file - and overwrite that 
file if it exits with a different status, i.e.
After the line
@DEL NBSTREAM_%STREAM_NUMBER%.%SPECIFICITY%LOCK
Add the following line
@ECHO %5 >NBSTREAM-%STREAM_NUMBER%.%SPECIFICITY%RESULT

Now, in the section of actions to be performed after the last stream, read 
through those results, and if any are greater than 1 - alert.
(Then clean up the results files)
So, after the line
@REM Perform the actions you want after backup

Add the following block.  (My apologies for the apparent clumsiness of this - 
specifying a fileset, never seems to work as I would like.)
(That and ENABLEDELAYEDEXPANSION doesn't seem to work as I would like in a 
nested FOR, but does when called.)

@SETLOCAL ENABLEDELAYEDEXPANSION
@SET HIGHSTATUS=0
@FOR /F %%i in ('dir NBSTREAM_*.%SPECIFICITY%RESULT /b') do (
 @FOR /F %%j in (%%i) do (
  @CALL :CHECKHIGHSTATUS %%j
 )
)

@IF %HIGHSTATUS gtr 1 (
  @call %NB_MAIL_SCRIPT% whoever AT wherever DOT tld "Backup on %1 - %5" 
)

@REM Now clean up the Result Files
@DEL NBSTREAM_*%SPECIFICITY%RESULT /Q

@GOTO :eof

:CHECKHIGHSTATUS
@IF /I %1 gtr !HIGHSTATUS! SET HIGHSTATUS=%1
@GOTO :eof









Patrick Sweeney
 (978) 787-4553
 patrick.sweeney AT axcelis DOT com
 I.T. Systems/Networks

For issues requiring immediate attention please contact the Solution Center
IT Solution Center
 (978) 787-7777
Beverly.Helpdesk AT axcelis DOT com
Axcelis Technologies

Have you searched here and here?

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Travis 
Kelley
Sent: Wednesday, February 04, 2009 11:39 AM
To: judy_hinchcliffe AT administaff DOT com; veritas-bu AT mailman.eng.auburn 
DOT edu
Subject: Re: [Veritas-bu] Real time failure notification

Looks like parent_end_notify does what I need for multistreamed jobs.
It doesn't get called for jobs without multistreaming though.  I guess
having an environment with policies that have multistreaming on and
off makes this harder.

For the other questions that were asked...this is NB6.5.2 and its.a
solaris master, so no blat.exe.

On 2/4/09, judy_hinchcliffe AT administaff DOT com
<judy_hinchcliffe AT administaff DOT com> wrote:
> Try the parent_end_notify
>
> This only runs when the parent job ends.
>
> If any of the children die the parent gets that error.
> But if all the children are good the parent gets a 0.
>
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Travis
> Kelley
> Sent: Wednesday, February 04, 2009 9:36 AM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] Real time failure notification
>
> Hi all.  I'm trying to find a solution to a monitoring problem we
> have.  I would like to create a mechanism to alert when a backup fails
> but to only send one alert if multiple streams from a backup fail.
> For instance if c: and d: both fail for a particular box, I only want
> 1 alert.  Also if a job fails twice but is successful on the third
> attempt I don't want an alert at all.  I only want to be alerted once
> when netbackup "gives up" on retrying a backup and fails the job.
> I've looked at backup_exit_notify but haven't been able to find a good
> way to implement this here.  Any ideas?
>
> --
> Sent from my mobile device
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>

-- 
Sent from my mobile device
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu