Veritas-bu

[Veritas-bu] NB51MP3 high cpu loads caused by bpdm?

2005-07-07 11:40:11
Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
From: kfhemness AT ucdavis DOT edu (Kathryn Hemness)
Date: Thu, 7 Jul 2005 08:40:11 -0700 (PDT)
Thanks, Len -

I have 2GB of RAM so I should be able to increase my shmmax from
16777216 to about 51200000.  I'll have to look around for the
other /etc/system values which may need adjusting.



On Thu, 7 Jul 2005, len boyle wrote:

> Date: Thu, 7 Jul 2005 10:11:44 -0400
> From: len boyle
> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> Subject: Re: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
> Yes it is the shmmax setting.
>
> You might also have to change some of the other settings also. There are
> several tech notes on what to change. But they do not have all the i's
> dotted and t's crossed in the veritas (symantic) doc. Some do to the best
> values being subjective and that sun is removing the need to set some of the
> varibles in the /etc/system file.
> At the min you have to find other applications, like oracle, and find a
> value for their usage, and then come up with an estimate for the netbackup
> usage. At least you will have to add up the number of buffers*the size of
> buffers*the number of active backup jobs. I am not 100% sure, but I believe
> that the values are in bytes. So with a max of 16 meg of memory being used
> by 1/4 meg buffers it would appear to be easy to overrun the shared memory.
>
> regards len
>
>
>
> ----- Original Message -----
> From: "Kathryn Hemness" <kfhemness AT ucdavis DOT edu>
> To: <veritas-bu AT mailman.eng.auburn DOT edu>
> Sent: Thursday, July 07, 2005 9:26 AM
> Subject: Re: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
>
> > Good Morning, Len -
> >
> > Is the o/s shared memory set in the /etc/system file?  Here are
> > my current settings:
> >
> > set msgsys:msginfo_msgmap=512
> > set msgsys:msginfo_msgmax=8192
> > set msgsys:msginfo_msgmnb=65536
> > set msgsys:msginfo_msgmni=256
> > set msgsys:msginfo_msgssz=16
> > set msgsys:msginfo_msgtql=512
> > set msgsys:msginfo_msgseg=8192
> >
> > set semsys:seminfo_semmap=64
> > set semsys:seminfo_semmni=1024
> > set semsys:seminfo_semmns=1024
> > set semsys:seminfo_semmnu=1024
> > set semsys:seminfo_semmsl=300
> > set semsys:seminfo_semopm=32
> > set semsys:seminfo_semume=64
> >
> >  set shmsys:shminfo_shmmax=16777216
> >  set shmsys:shminfo_shmmin=1
> >  set shmsys:shminfo_shmmni=220
> >  set shmsys:shminfo_shmseg=100
> >
> > On Thu, 7 Jul 2005, len boyle wrote:
> >
> >> Date: Thu, 7 Jul 2005 09:16:18 -0400
> >> From: len boyle
> >> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> Subject: Re: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >>
> >>  Kathryn,
> >>
> >> Did you look at the usage of the o/s shared memory. It sounds like you
> >> will
> >> have to increase this resource.
> >>
> >> len
> >> ----- Original Message -----
> >> From: "Kathryn Hemness" <kfhemness AT ucdavis DOT edu>
> >> To: "Geyer, Gregory" <Gregory.Geyer AT avnet DOT com>
> >> Cc: <veritas-bu AT mailman.eng.auburn DOT edu>
> >> Sent: Thursday, July 07, 2005 8:54 AM
> >> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >>
> >>
> >> > Good Morning, Gregory --
> >> >
> >> > I had to remove the SIZE_DATA_BUFFERS_DISK and
> >> > NUMBERS_DATA_BUFFERS_DISK
> >> > files last night after all of my evening scheduled backups failed in an
> >> > Incomplete state with an
> >> > EXIT STATUS 89 (problems encountered during setup of shared memory).
> >> >
> >> > I had left the tape buffer files intact and had just added these
> >> > two DISK files to the directory.  According to the documentation, no
> >> > special daemon restarts or reboots were necessary.
> >> >
> >> > I'm not giving up on these disk settings yet.  When I added the
> >> > files to the /usr/openv/netbackup/db/config directory, there
> >> > weren't any active backups, but there were disk images on the current
> >> > backup storage unit.  I'm also wondering if the storage unit fragment
> >> > size needs to be adjusted, too; it's currently set to the 2000 KB
> >> > maximum.
> >> >
> >> > I'll  google for more information on the EXIT 89 status and let
> >> > you know what I come up with.
> >> >
> >> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >> >
> >> >> Date: Wed, 6 Jul 2005 15:52:41 -0700
> >> >> From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> >> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> >> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> >> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >>
> >> >> Kathy,
> >> >>
> >> >> Not sure, I've only experimented with them lately but I think it's a
> >> >> case of trying a setting and watching and then perhaps tweaking them
> >> >> until you've found a good spot.  I'm in your boat as well: still
> >> >> looking
> >> >> for the best settings.
> >> >>
> >> >> G.
> >> >>
> >> >> -----Original Message-----
> >> >> From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> >> >> DOT edu] On
> >> >> Behalf Of Kathryn Hemness
> >> >> Sent: Wednesday, July 06, 2005 3:45 PM
> >> >> To: Geyer, Gregory
> >> >> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> >> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >>
> >> >> Thanks, Gregory -
> >> >>
> >> >> Is there a good rule of thumb for the number of disk data buffers
> >> >> based
> >> >> on the size of the storage unit?  I've set the number to 16 for my
> >> >> backups tonight.  Is the number of disk data buffers supposed to be a
> >> >> power of 2 like the non-disk NUMBER_DATA_BUFFERS?
> >> >>
> >> >> On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >> >>
> >> >> > Date: Wed, 6 Jul 2005 15:31:17 -0700
> >> >> > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> >> > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> >> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> >> >> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >> >
> >> >> > A trick a Veritas instructor gave me last year was this:
> >> >> >
> >> >> > grep out "waited for full buffer" and "waited for empty buffer" out
> >> >> > of
> >> >>
> >> >> > your bpdm/bptm logs.
> >> >> >
> >> >> > If you see many "waited for full buffer" in your bpdm/bptm log then
> >> >> > the storage unit disk/tape is waiting for the data buffer to fill.
> >> >> > Tune down the size of the data buffer so it fills quicker.
> >> >> >
> >> >> > If you see many "waited for empty buffer" then the server is waiting
> >> >> > on the storage unit.  Increase the number of buffers.
> >> >> >
> >> >> > G.
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT 
> >> >> > reality.ucdavis DOT edu]
> >> >> > On
> >> >> > Behalf Of Kathryn Hemness
> >> >> > Sent: Wednesday, July 06, 2005 3:18 PM
> >> >> > To: Geyer, Gregory
> >> >> > Cc: Kathryn Hemness; veritas-bu AT mailman.eng.auburn DOT edu
> >> >> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >> >
> >> >> > Thanks.  I'll give the DISK BUFFER settings a try.  I was kind of
> >> >> > wondering if a smaller buffer size might be better.
> >> >> >
> >> >> >
> >> >> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >> >> >
> >> >> > > Date: Wed, 6 Jul 2005 15:00:09 -0700
> >> >> > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> >> >> > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>,
> >> >> > >      veritas-bu AT mailman.eng.auburn DOT edu
> >> >> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >> > >
> >> >> > > Kathy,
> >> >> > >
> >> >> > > I believe SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS are for bptm
> >> >> only.
> >> >> > > I've only just discovered "SIZE_DATA_BUFFERS_DISK" and
> >> >> > > "NUMBER_DATA_BUFFERS_DISK" for use by bpdm.
> >> >> > >
> >> >> > > However I also experienced shared memory issues with these as
> >> >> > > we're
> >> >> > > a heavy disk-staging shop.  These might help your speeds, though I
> >> >> > > don't
> >> >> >
> >> >> > > know about offloading your CPU.
> >> >> > >
> >> >> > > Reference Document ID 273532 for the above info.
> >> >> > >
> >> >> > > G.
> >> >> > >
> >> >> > > -----Original Message-----
> >> >> > > From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> >> >> > > [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of
> >> >> > > Kathryn Hemness
> >> >> > > Sent: Wednesday, July 06, 2005 2:38 PM
> >> >> > > To: veritas-bu AT mailman.eng.auburn DOT edu
> >> >> > > Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >> >> > >
> >> >> > > Greetings --
> >> >> > >
> >> >> > > I am running NetBackup 5.1 MP3 on a Sun V240 and using Sun
> >> >> > > StorEdge
> >> >> > > 3511 storage arrays with 6TB capacity for backups to disk (not
> >> >> > > using
> >> >>
> >> >> > > the disk-staging configuration).  The OS is Solaris 9.
> >> >> > > The backups are first written to the storage array and then staged
> >> >> > > to tape using cron-script managed duplications so that I can use
> >> >> > > backupid-files containing backupids having the same retentions to
> >> >> > > minimize tape mounts.  My tape library is a Qualstar 88264 LTO2
> >> >> > > library with 3 LTO2 drives.
> >> >> > >
> >> >> > > To optimize read and write speeds the storage arrays, I alternate
> >> >> > > directing all active backups to one storage array while staging
> >> >> > > diskimages to tape from the other storage array.  I'm able to
> >> >> > > alternate the active backup storage unit daily.
> >> >> > >
> >> >> > > In order to complete all of the backups during typical non-workday
> >> >> > > hours, I've set NUMBER_DATA_BUFFERS to 24 and SIZE_DATA_BUFFERS to
> >> >> > > 262144.
> >> >> > >
> >> >> > > The backups are finishing in their configured windows, but the CPU
> >> >> > > load is extremely high. The uptime commands shows a load of 25-40
> >> >> > > all during my backup windows and top shows the bpdm processes as
> >> >> > > using
> >> >> > > 1.0-1.4 percent of the CPU.
> >> >> > >
> >> >> > > I'm curious to see if there are others seeing high loads caused by
> >> >> > > bpdm whis is used for both disk and disk-staging storage units.
> >> >> > > I'd like to hear about how others are configuring NetBackup for
> >> >> > > backups to disk, mostly regarding the size of the disk storage
> >> >> > > units, NUMBER_DATA_BUFFERS, SIZE_DATA_BUFFERS, and number of
> >> >> concurrent jobs.
> >> >> > >
> >> >> > > Thanks for any information you can give me.
> >> >> > >
> >> >> > > --kathy
> >> >> > >
> >> >> > > _______________________________________________
> >> >> > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> >> > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> >> > >
> >> >> >
> >> >> > --kathy
> >> >> >
> >> >>
> >> >> --kathy
> >> >>
> >> >
> >> > --kathy
> >> >
> >> > _______________________________________________
> >> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> >
> >>
> >>
> >
> > --kathy
> >

--kathy