Veritas-bu

Re: [Veritas-bu] Process which runs while a job is running

2008-09-19 12:03:33
Subject: Re: [Veritas-bu] Process which runs while a job is running
From: "Mark Glazerman" <Mark.Glazerman AT spartech DOT com>
To: "Jeff Lightner" <jlightner AT water DOT com>, <veritas-bu AT mailman.eng.auburn DOT edu>
Date: Fri, 19 Sep 2008 10:40:58 -0500

Jeff,

 

When I tested the script on our master server it didn’t send out the notification that the services had been stopped until all the NBU services had been brought down.  I know that when our server gets hung up it can take a while to get the services stopped and sometimes I have to use bp.kill_all to get everything to shut down.  However, as the script appears to wait until everything is down before starting everything up again, it looks like I don’t have to overcomplicate my script (or overtax my brain on a Friday !!).

 

Thanks

 

Mark Glazerman

Desk: 314-889-8282

Cell: 618-520-3401

P please don't print this e-mail unless you really need to

 

From: Jeff Lightner [mailto:jlightner AT water DOT com]
Sent: Friday, September 19, 2008 10:33 AM
To: Mark Glazerman; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Process which runs while a job is running

 

You might want to put in a loop after the stop that checks to see if all processes are actually stopped (e.g. check “bpps –a” output) and run <installdir>/ bin/goodies/bp.kill_all to kill any remaining if it finds them and repeat until it does (with sleeps between to give all processes time to die).  You may need to put in a routine to make it page you (send email to your cell phone if you don’t have a pager) if all processes haven’t died after a certain number of attempts.

 


From: veritas-bu-bounces AT mailman.eng.auburn DOT edu [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Mark Glazerman
Sent: Friday, September 19, 2008 11:15 AM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Process which runs while a job is running

 

Thanks for all your responses.

 

After a bit of help from our resident scripting expert I’ve come up with the following and thought I would share in case anyone else needs to restart NBU on a regular basis.

 

#!/usr/bin/ksh

##  This script restarts NetBackup Once A Week To Try And Avoid The Master Server Crashing ##

#

nbpemreq -suspend_scheduling

t_p=`bpdbjobs -summary | egrep nbumaster01 | grep -v grep | awk '{print $4}'`

while [t_p -ne "0"]

do

t_p=`bpdbjobs -summary | egrep nbumaster01 | grep -v grep | awk '{print $4}'`

sleep 10

done

/etc/init.d/netbackup stop

echo "NetBackup services have been stopped" | mailx -s "NetBackup services have been stopped" email AT email DOT com

sleep 120

/etc/init.d/netbackup start

echo "NetBackup services have been restarted" | mailx -s "NetBackup services have been restarted" email AT email DOT com

 

I’m going to run it as a cron job at 2pm every Sunday.

 

Thanks again,

 

Mark Glazerman

Desk: 314-889-8282

Cell: 618-520-3401

P please don't print this e-mail unless you really need to

 

From: veritas-bu-bounces AT mailman.eng.auburn DOT edu [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Mark Glazerman
Sent: Friday, September 19, 2008 8:20 AM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] Process which runs while a job is running

 

I am wanting to write a script to place in the crontab of our master server which will restart the NBU services once a week.  I plan to have the script issue nbpemreq –suspend_scheduling but then need a way for it to check for any backups still running before issuing /etc/init.d/netbackup stop.

 

Does anyone know if there is a process I can have the script grep for which won’t be there once all the backups have completed ?  I don’t want to stop the NBU processes if there are backups still running.  We do plan to have the script run during a period when no backups are running but we all know how backups can randomly overrun their expected windows !!

 

In case anyone is curious, our master server has some kind of semi regular memory issue which arises about once every 10 days or so and can be resolved by restarting the NBU services.  My plan is to proactively do this on  a weekly basis in the hope that it will stop the master server rejecting connections due to a lack of swap (like it did last night during my daughters softball game !!).

 

Thanks

 

Mark Glazerman

Enterprise Storage Administrator

Spartech Corporation

Desk: 314-889-8282

Fax: 314-854-8282

Cell: 618-520-3401

mark.glazerman AT spartech DOT com

http://www.spartech.com

P please don't print this e-mail unless you really need to

This e-mail and any files transmitted with it are confidential, are intended solely for the use of the addressee, and may be legally privileged. If you have received this email in error please notify the sender immediately, and do not copy or forward it.

 

----------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
----------------------------------

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu