Veritas-bu

Re: [Veritas-bu] Long time between "requesting resources" and "granting resources"

2008-07-10 13:24:21
Subject: Re: [Veritas-bu] Long time between "requesting resources" and "granting resources"
From: <Mark.Donaldson AT cexp DOT com>
To: <Veritas-bu AT mailman.eng.auburn DOT edu>
Date: Thu, 10 Jul 2008 10:47:19 -0600
Nope - not waiting on the max policy jobs, that's set to unlimited.
Storage unit is also set for 99 jobs and isn't near that heavily used.
Max jobs per client is also high at 99 jobs/client (and nowhere near
that).

FWIW, some command-line stuff is running very slowly, too.  "bpplinfo
-modify" can take a good minute to return.  Further FWIW: this server is
on the other side of a firewall.

======== 
> sudo bppllist AP-01_RM_Arch_iprd -U
Policy Name:       AP-01_RM_Arch_iprd
  Policy Type:         Oracle
  Active:              yes
<...>
  Policy Priority:     12
  Max Jobs/Policy:     Unlimited   
<...>
========
> sudo bpclient -client ap-01 -L
Client Name: ap-01
<...>
 Max Jobs This Client:  99   
<...>
 Connect options:       0 1 3
========

..../etc/system is set for a large system, at least for v5.x when we
last probably visited this file, and hasn't been altered in quite some
time.  Once it was dead-on to the tuning guide...

* Message Queue Parameters
set msgsys:msginfo_msgmap=1024
set msgsys:msginfo_msgmax=65536
set msgsys:msginfo_msgmnb=131072
set msgsys:msginfo_msgmni=1024
set msgsys:msginfo_msgseg=16384
set msgsys:msginfo_msgssz=16
set msgsys:msginfo_msgtql=1024

* Semaphore Parameters
set semsys:seminfo_semmap=128
set semsys:seminfo_semmni=2048
set semsys:seminfo_semmns=2048
set semsys:seminfo_semmnu=2048
set semsys:seminfo_semmsl=600
set semsys:seminfo_semopm=128
set semsys:seminfo_semume=128

* Shared Memory Parameters
set shmsys:shminfo_shmmax=0x100000000
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=512
set shmsys:shminfo_shmseg=200
 
________________________________

From: Ed Wilts [mailto:ewilts AT ewilts DOT org] 
Sent: Wednesday, July 09, 2008 6:34 PM
To: Donaldson, Mark - Broomfield, CO
Cc: Veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Long time between "requesting resources" and
"granting resources"


On Wed, Jul 9, 2008 at 5:40 PM, <Mark.Donaldson AT cexp DOT com> wrote:



        The master server is an 8 CPU, 16G Solaris 8 v880 box.  It's got
        NBv6.5.1.
        
        It's running with an average of 50% idle or so but the load
average is
        kinda high:
        
        
        last pid:  8851;  load averages:  4.01,  3.66,  3.60
        11:50:46
        170 processes: 164 sleeping, 1 zombie, 1 stopped, 4 on cpu
        CPU states: 53.2% idle, 13.0% user, 28.8% kernel,  5.0% iowait,
0.0%
        swap
        Memory: 16G real, 8702M free, 2372M swap in use, 24G swap free
        


FWIW, that load average is not "kinda high".  You've got 8 cpus and only
4 jobs on a cpu - that means you've got 4 cpus doing nothing which also
explains your 50% idle time.  You are not CPU bound.  You also don't
seem to be memory bound.



        Here's the details of a job that ran slower than I'd like.  36
minutes
        from "request" to "granted".  It's a high-priority backup,
highest of
        all of them, FWIW.  It, and many of my policies now, are backing
up to
        storage unit groups.   The AP-01 client below is an AIX media
server
        with NBv6.
        
        07/09/2008 08:36:45 - requesting resource AP01_Pri
        07/09/2008 08:36:45 - requesting resource
ug-50.NBU_CLIENT.MAXJOBS.ap-01
        07/09/2008 08:36:45 - requesting resource
        ug-50.NBU_POLICY.MAXJOBS.AP-01_RM_Arch_iprd
        07/09/2008 09:12:21 - granted resource
ug-50.NBU_CLIENT.MAXJOBS.ap-01


Note the resource it's waiting for - it's the maximum jobs per policy
(NBU_POLICY.MAXJOBS).  You have it set to some number and that number
has been reached.  No other jobs will start for that client until a job
finishes.  If you check your logs, you should see that a job for that
policy completed shortly before the resource was granted.
 
   .../Ed


-- 
Ed Wilts, Mounds View, MN, USA
RHCE, BCFP, BCSD, SCSP, SCSE
mailto:ewilts AT ewilts DOT org

If I've helped you, please make a donation to my favorite charity at
http://firstgiving.com/edwilts 

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu