Veritas-bu

[Veritas-bu] Monitoring for Locked jobs

2006-09-27 11:27:58
Subject: [Veritas-bu] Monitoring for Locked jobs
From: stumpb at michigan.gov (Bob Stump)
Date: Wed, 27 Sep 2006 11:27:58 -0400
Why not run it on a UNIX media server? 
Perhaps even a LINUX? I dunno. 
I haven't tried it but there is a 
bpdbjobs -M <masterservername>

>>> "Dyck, Jonathan" <Jonathan.Dyck at cognos.com> 9/27/2006 11:21 AM >>>

Bah!
 
See?  All you Windows users should bail, and get a real OS ;-)
 
 
(this coming from a guy who's just put in a new Windows master and Windows 
media server)
 
Sorry Phil.   Guess I was just wasting your time.  On a different thread,  
there are some definite benefits of having mixed masters in an environment with 
mixed clients...  Don't really have the time to speak to them, but there ya go.
 
Cheers,
Jon
 

From: veritas-bu-bounces at mailman.eng.auburn.edu [mailto:veritas-bu-bounces 
at mailman.eng.auburn.edu] On Behalf Of Bob Stump
Sent: Wednesday, September 27, 2006 10:58 AM
To: Phil Koster; Dyck, Jonathan; veritas-bu
Subject: Re: [Veritas-bu] Monitoring for Locked jobs



Ain't no windows equivalent.

>>> "Dyck, Jonathan" <Jonathan.Dyck at cognos.com> 9/27/2006 10:47 AM >>>
I hear ya Phil.  Can't count the number of times @ 2AM I've cursed doing my 
"due diligence" on data collection in the event of a failure, but sometimes it 
even helps :)
 
Here's the quote straight from my support (I asked them the same question 
awhile ago) on the bpdbjobs command, you can customize to your own needs with 
BPDBJOB_COLDEFS...  
 
I run all my scripts on a Solaris box,  maybe someone else on this DL can fill 
you in  on the Windows equivalent...
 
Cheers,
Jon
 
==================
Yes, that?s correct. If you want to have the ?Operation? column displayed when 
executing ?bpdbjobs? command then you can add the setting ?BPDBJOB_COLDEFS = 
Operation 12 true? in the bp.conf. However, you will only get the output for 
Operation column when you execute the ?bpdbjobs? command. To have specific 
columns output for ?bpdbjobs?, you?d need to add multiple BPDBJOB_COLDEFS 
entries in the bp.conf.For detailed use of BPDBJOB_COLDEFS, please refer to the 
technote http://support.veritas.com/docs/266314If you don?t want to use the 
BPDBJOB_COLDEFS settings in the bp.conf you can reference the text file 
attached, which deciphering all the fields/columns for different outputs of 
?bpdbjobs? command. You can then use it in the script to spit out whatever 
fields/columns you want. ==================

From: "Koster, Phil" [mailto:pkoster at ci.grand-rapids.mi.us] 
Sent: Wednesday, September 27, 2006 10:12 AM
To: Dyck, Jonathan; veritas-bu
Subject: RE: [Veritas-bu] Monitoring for Locked jobs



According to the report logs we got a "bad image header" and a "could not build 
host list" error right around the same time it locked up.  Not to say it wasn't 
a mounting problem but our primary interest was getting things running again so 
we did not take enough care in documenting the state we found the server in.  
 
What status's will a bpdbjobs show?  When I look in the Windows Command 
reference I only see: field3 = state (0=queued, 1=active, 2=waiting for retry, 
3=done)The rest don't provide definitions with the potential responses (like a 
-all_columns or a -most_columns).  I suppose we could do a bpdbjobs 
-all_columns and look for a state (field 3) of 1 or 2 and then look at the 
elapsed time (field 10).  But is there a better field then the "state" field or 
is state just as useful for this as any other field?In 20/20 hindsight we would 
have been better off with a forensic quality investigation instead of a rapid 
recovery style response.  It was one of those split second decisions based on 
the fear of the status of our back ups.  (Back ups are high profile around 
here.  Last year we had a NAS device double fault on a RAID 5 and the back ups 
were bad, one week and $36,000 later we had only recovered about 60% of the 
~500 GB of data.  Ever since then....).  Thanks.
Phil Koster
Network Administrator
City of Grand Rapids
Direct: 616-456-3136
Helpdesk: 456-3999


From: Dyck, Jonathan [mailto:Jonathan.Dyck at cognos.com] 
Sent: Wednesday, September 27, 2006 9:08 AM
To: Koster, Phil; veritas-bu
Subject: RE: [Veritas-bu] Monitoring for Locked jobs




Phil,
I run into this kind of thing once in awhile, although I typically see hung 
jobs in a "mounting" state if there's a problem, and for some reason my media 
mount timeout doesn't kill the job.  Did you experience something similar?
 
One way to go about scripting a job would be to "bpdbjobs -report" and grep on 
"Mounting".  You can get a elapsed time from the report too, and using that 
time, you should be able to page out based on your threshold.
 
Just my thoughts...
 
Cheers,
Jon
 


From: veritas-bu-bounces at mailman.eng.auburn.edu [mailto:veritas-bu-bounces 
at mailman.eng.auburn.edu] On Behalf Of "Koster, Phil"
Sent: Tuesday, September 26, 2006 7:55 AM
To: veritas-bu
Subject: [Veritas-bu] Monitoring for Locked jobs



NBU 6 MP2 on Win2K Srvr.


Over the weekend, all our jobs locked up on the back up server essentially 
stopping all backups beginning around 9 AM Saturday. All NBU processes 
(Windows) continued to run normally which did not allow our Nagios Monitoring 
system to notify us (it all seemed just fine). So my question is, is there 
anything we can monitor, even if by manual script, to let us know when jobs get 
hung?


Can we do like a bpdbjobs and see how long the jobs have been running and use a 
script to evaluate that information? Anyone doing something like that already?


What I am thinking is at the very least just get something that can monitor the 
jobs automatically (like a once per hour check) and if the jobs get over x 
hours active then it sends some e-mails out to our cell phones and inboxes. I 
tried to play around with bpdbjobs last night but ran out of time before our 
back ups finished. (Took to long getting the kids to bed ;-)


Thanks.


Phil Koster
Network Administrator
City of Grand Rapids
Direct: 616-456-3136
Helpdesk: 456-3999


*****************************************************************************************************
 This message has been prepared on resources owned by the City of Grand Rapids, 
MI. It is subject to the Acceptable Use Policy and Procedures of the City of 
Grand Rapids. The information contained herein is confidential and is intended 
solely for the addressee. Access by any other party is unauthorized without the 
express written permission of the sender. If you are not the intended 
recipient, please contact the sender and delete this message. 
*****************************************************************************************************
 
 
     This message may contain privileged and/or confidential information.  If 
you have received this e-mail in error or are not the intended recipient, you 
may not use, copy, disseminate or distribute it; do not open any attachments, 
delete it immediately from your system and notify the sender promptly by e-mail 
that you have done so.  Thank you. 

*****************************************************************************************************
 This message has been prepared on resources owned by the City of Grand Rapids, 
MI. It is subject to the Acceptable Use Policy and Procedures of the City of 
Grand Rapids. The information contained herein is confidential and is intended 
solely for the addressee. Access by any other party is unauthorized without the 
express written permission of the sender. If you are not the intended 
recipient, please contact the sender and delete this message. 
*****************************************************************************************************
 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20060927/7da9c029/attachment.html