Veritas-bu

[Veritas-bu] NB51MP3 high cpu loads caused by bpdm?

2005-07-07 09:26:18
Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
From: kfhemness AT ucdavis DOT edu (Kathryn Hemness)
Date: Thu, 7 Jul 2005 06:26:18 -0700 (PDT)
Good Morning, Len -

Is the o/s shared memory set in the /etc/system file?  Here are
my current settings:

set msgsys:msginfo_msgmap=512
set msgsys:msginfo_msgmax=8192
set msgsys:msginfo_msgmnb=65536
set msgsys:msginfo_msgmni=256
set msgsys:msginfo_msgssz=16
set msgsys:msginfo_msgtql=512
set msgsys:msginfo_msgseg=8192

 set semsys:seminfo_semmap=64
 set semsys:seminfo_semmni=1024
 set semsys:seminfo_semmns=1024
 set semsys:seminfo_semmnu=1024
 set semsys:seminfo_semmsl=300
 set semsys:seminfo_semopm=32
 set semsys:seminfo_semume=64

  set shmsys:shminfo_shmmax=16777216
  set shmsys:shminfo_shmmin=1
  set shmsys:shminfo_shmmni=220
  set shmsys:shminfo_shmseg=100

On Thu, 7 Jul 2005, len boyle wrote:

> Date: Thu, 7 Jul 2005 09:16:18 -0400
> From: len boyle
> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> Subject: Re: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
>  Kathryn,
>
> Did you look at the usage of the o/s shared memory. It sounds like you will
> have to increase this resource.
>
> len
> ----- Original Message -----
> From: "Kathryn Hemness" <kfhemness AT ucdavis DOT edu>
> To: "Geyer, Gregory" <Gregory.Geyer AT avnet DOT com>
> Cc: <veritas-bu AT mailman.eng.auburn DOT edu>
> Sent: Thursday, July 07, 2005 8:54 AM
> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
>
> > Good Morning, Gregory --
> >
> > I had to remove the SIZE_DATA_BUFFERS_DISK and NUMBERS_DATA_BUFFERS_DISK
> > files last night after all of my evening scheduled backups failed in an
> > Incomplete state with an
> > EXIT STATUS 89 (problems encountered during setup of shared memory).
> >
> > I had left the tape buffer files intact and had just added these
> > two DISK files to the directory.  According to the documentation, no
> > special daemon restarts or reboots were necessary.
> >
> > I'm not giving up on these disk settings yet.  When I added the
> > files to the /usr/openv/netbackup/db/config directory, there
> > weren't any active backups, but there were disk images on the current
> > backup storage unit.  I'm also wondering if the storage unit fragment
> > size needs to be adjusted, too; it's currently set to the 2000 KB
> > maximum.
> >
> > I'll  google for more information on the EXIT 89 status and let
> > you know what I come up with.
> >
> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >
> >> Date: Wed, 6 Jul 2005 15:52:41 -0700
> >> From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >>
> >> Kathy,
> >>
> >> Not sure, I've only experimented with them lately but I think it's a
> >> case of trying a setting and watching and then perhaps tweaking them
> >> until you've found a good spot.  I'm in your boat as well: still looking
> >> for the best settings.
> >>
> >> G.
> >>
> >> -----Original Message-----
> >> From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> >> DOT edu] On
> >> Behalf Of Kathryn Hemness
> >> Sent: Wednesday, July 06, 2005 3:45 PM
> >> To: Geyer, Gregory
> >> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >>
> >> Thanks, Gregory -
> >>
> >> Is there a good rule of thumb for the number of disk data buffers based
> >> on the size of the storage unit?  I've set the number to 16 for my
> >> backups tonight.  Is the number of disk data buffers supposed to be a
> >> power of 2 like the non-disk NUMBER_DATA_BUFFERS?
> >>
> >> On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >>
> >> > Date: Wed, 6 Jul 2005 15:31:17 -0700
> >> > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >
> >> > A trick a Veritas instructor gave me last year was this:
> >> >
> >> > grep out "waited for full buffer" and "waited for empty buffer" out of
> >>
> >> > your bpdm/bptm logs.
> >> >
> >> > If you see many "waited for full buffer" in your bpdm/bptm log then
> >> > the storage unit disk/tape is waiting for the data buffer to fill.
> >> > Tune down the size of the data buffer so it fills quicker.
> >> >
> >> > If you see many "waited for empty buffer" then the server is waiting
> >> > on the storage unit.  Increase the number of buffers.
> >> >
> >> > G.
> >> >
> >> > -----Original Message-----
> >> > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> >> > DOT edu] On
> >> > Behalf Of Kathryn Hemness
> >> > Sent: Wednesday, July 06, 2005 3:18 PM
> >> > To: Geyer, Gregory
> >> > Cc: Kathryn Hemness; veritas-bu AT mailman.eng.auburn DOT edu
> >> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >
> >> > Thanks.  I'll give the DISK BUFFER settings a try.  I was kind of
> >> > wondering if a smaller buffer size might be better.
> >> >
> >> >
> >> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >> >
> >> > > Date: Wed, 6 Jul 2005 15:00:09 -0700
> >> > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>,
> >> > >      veritas-bu AT mailman.eng.auburn DOT edu
> >> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> > >
> >> > > Kathy,
> >> > >
> >> > > I believe SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS are for bptm
> >> only.
> >> > > I've only just discovered "SIZE_DATA_BUFFERS_DISK" and
> >> > > "NUMBER_DATA_BUFFERS_DISK" for use by bpdm.
> >> > >
> >> > > However I also experienced shared memory issues with these as we're
> >> > > a heavy disk-staging shop.  These might help your speeds, though I
> >> > > don't
> >> >
> >> > > know about offloading your CPU.
> >> > >
> >> > > Reference Document ID 273532 for the above info.
> >> > >
> >> > > G.
> >> > >
> >> > > -----Original Message-----
> >> > > From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> >> > > [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of
> >> > > Kathryn Hemness
> >> > > Sent: Wednesday, July 06, 2005 2:38 PM
> >> > > To: veritas-bu AT mailman.eng.auburn DOT edu
> >> > > Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> > >
> >> > > Greetings --
> >> > >
> >> > > I am running NetBackup 5.1 MP3 on a Sun V240 and using Sun StorEdge
> >> > > 3511 storage arrays with 6TB capacity for backups to disk (not using
> >>
> >> > > the disk-staging configuration).  The OS is Solaris 9.
> >> > > The backups are first written to the storage array and then staged
> >> > > to tape using cron-script managed duplications so that I can use
> >> > > backupid-files containing backupids having the same retentions to
> >> > > minimize tape mounts.  My tape library is a Qualstar 88264 LTO2
> >> > > library with 3 LTO2 drives.
> >> > >
> >> > > To optimize read and write speeds the storage arrays, I alternate
> >> > > directing all active backups to one storage array while staging
> >> > > diskimages to tape from the other storage array.  I'm able to
> >> > > alternate the active backup storage unit daily.
> >> > >
> >> > > In order to complete all of the backups during typical non-workday
> >> > > hours, I've set NUMBER_DATA_BUFFERS to 24 and SIZE_DATA_BUFFERS to
> >> > > 262144.
> >> > >
> >> > > The backups are finishing in their configured windows, but the CPU
> >> > > load is extremely high. The uptime commands shows a load of 25-40
> >> > > all during my backup windows and top shows the bpdm processes as
> >> > > using
> >> > > 1.0-1.4 percent of the CPU.
> >> > >
> >> > > I'm curious to see if there are others seeing high loads caused by
> >> > > bpdm whis is used for both disk and disk-staging storage units.
> >> > > I'd like to hear about how others are configuring NetBackup for
> >> > > backups to disk, mostly regarding the size of the disk storage
> >> > > units, NUMBER_DATA_BUFFERS, SIZE_DATA_BUFFERS, and number of
> >> concurrent jobs.
> >> > >
> >> > > Thanks for any information you can give me.
> >> > >
> >> > > --kathy
> >> > >
> >> > > _______________________________________________
> >> > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> > >
> >> >
> >> > --kathy
> >> >
> >>
> >> --kathy
> >>
> >
> > --kathy
> >
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >
>
>

--kathy

===============================================================================
Kathryn Hemness                        kfhemness AT ucdavis DOT edu
System Administrator                   phone: 530.752.6547
Campus Data Center & Client Services   fax:   530.752.9154