Veritas-bu

[Veritas-bu] NB51MP3 high cpu loads caused by bpdm?

2005-07-09 11:43:08
Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
From: kfhemness AT ucdavis DOT edu (Kathryn Hemness)
Date: Sat, 9 Jul 2005 08:43:08 -0700 (PDT)
Well, last night's backups as a test of the new settings was
a wash.  I'm still seeing fragment sizes of 2000MB on the
storage unit whose configurations I changed.  It looks like I
need to restart the NetBackup daemons when I modify the storage
unit settings. sigh.

I'll be modifying my other storage unit fragment size after
all duplications from it finish this afternoon.

You can be sure I'll stop and restart the daemons this time.
another sigh.


On Fri, 8 Jul 2005, Geyer, Gregory wrote:

> Date: Fri, 8 Jul 2005 11:37:42 -0700
> From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
> Kathy,
>
> Excellent news.  Please let me know how your weekend backups turn out
> with your new settings.  I'm going to review my setup next week and go
> back to experimenting with these settings.
>
> Thanks!
>
> Greg
>
> -----Original Message-----
> From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis DOT 
> edu] On
> Behalf Of Kathryn Hemness
> Sent: Friday, July 08, 2005 10:22 AM
> To: Geyer, Gregory
> Cc: veritas-bu AT mailman.eng.auburn DOT edu; Bill_Jorgensen AT csgsystems 
> DOT com
> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
> Hi Gregory --
>
> I made the following changes on my server and the number of 'waited for
> full buffer' messages in my bpdm log decreased by half.  Neither the cpu
> load nor the 'waited for empty buffer'
> showed any significant change:
>
> /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_DISK:     16
> /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK:  1048576
> /etc/system: set shmsys:shminfo_shmmax=536870912
>
>
> I left the tape data buffers the same:
> /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS 16
> /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS 262144
>
> And my storage unit fragment size was still 2000 MB.
>
> For my backups tonight, I plan on increasing the
> NUMBER_DATA_BUFFERS_DISK to 24 and decreasing the fragment size to 1000
> MB.
>
>
> On Wed, 6 Jul 2005, Geyer, Gregory wrote:
>
> > Date: Wed, 6 Jul 2005 15:52:41 -0700
> > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >
> > Kathy,
> >
> > Not sure, I've only experimented with them lately but I think it's a
> > case of trying a setting and watching and then perhaps tweaking them
> > until you've found a good spot.  I'm in your boat as well: still
> > looking for the best settings.
> >
> > G.
> >
> > -----Original Message-----
> > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis DOT 
> > edu] On
> > Behalf Of Kathryn Hemness
> > Sent: Wednesday, July 06, 2005 3:45 PM
> > To: Geyer, Gregory
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >
> > Thanks, Gregory -
> >
> > Is there a good rule of thumb for the number of disk data buffers
> > based on the size of the storage unit?  I've set the number to 16 for
> > my backups tonight.  Is the number of disk data buffers supposed to be
>
> > a power of 2 like the non-disk NUMBER_DATA_BUFFERS?
> >
> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >
> > > Date: Wed, 6 Jul 2005 15:31:17 -0700
> > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> > > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > >
> > > A trick a Veritas instructor gave me last year was this:
> > >
> > > grep out "waited for full buffer" and "waited for empty buffer" out
> > > of
> >
> > > your bpdm/bptm logs.
> > >
> > > If you see many "waited for full buffer" in your bpdm/bptm log then
> > > the storage unit disk/tape is waiting for the data buffer to fill.
> > > Tune down the size of the data buffer so it fills quicker.
> > >
> > > If you see many "waited for empty buffer" then the server is waiting
>
> > > on the storage unit.  Increase the number of buffers.
> > >
> > > G.
> > >
> > > -----Original Message-----
> > > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> > > DOT edu]
> > > On Behalf Of Kathryn Hemness
> > > Sent: Wednesday, July 06, 2005 3:18 PM
> > > To: Geyer, Gregory
> > > Cc: Kathryn Hemness; veritas-bu AT mailman.eng.auburn DOT edu
> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > >
> > > Thanks.  I'll give the DISK BUFFER settings a try.  I was kind of
> > > wondering if a smaller buffer size might be better.
> > >
> > >
> > > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> > >
> > > > Date: Wed, 6 Jul 2005 15:00:09 -0700
> > > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>,
> > > >      veritas-bu AT mailman.eng.auburn DOT edu
> > > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > >
> > > > Kathy,
> > > >
> > > > I believe SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS are for bptm
> > only.
> > > > I've only just discovered "SIZE_DATA_BUFFERS_DISK" and
> > > > "NUMBER_DATA_BUFFERS_DISK" for use by bpdm.
> > > >
> > > > However I also experienced shared memory issues with these as
> > > > we're a heavy disk-staging shop.  These might help your speeds,
> > > > though I don't
> > >
> > > > know about offloading your CPU.
> > > >
> > > > Reference Document ID 273532 for the above info.
> > > >
> > > > G.
> > > >
> > > > -----Original Message-----
> > > > From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> > > > [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of
> > > > Kathryn Hemness
> > > > Sent: Wednesday, July 06, 2005 2:38 PM
> > > > To: veritas-bu AT mailman.eng.auburn DOT edu
> > > > Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > >
> > > > Greetings --
> > > >
> > > > I am running NetBackup 5.1 MP3 on a Sun V240 and using Sun
> > > > StorEdge
> > > > 3511 storage arrays with 6TB capacity for backups to disk (not
> > > > using
> >
> > > > the disk-staging configuration).  The OS is Solaris 9.
> > > > The backups are first written to the storage array and then staged
>
> > > > to tape using cron-script managed duplications so that I can use
> > > > backupid-files containing backupids having the same retentions to
> > > > minimize tape mounts.  My tape library is a Qualstar 88264 LTO2
> > > > library with 3 LTO2 drives.
> > > >
> > > > To optimize read and write speeds the storage arrays, I alternate
> > > > directing all active backups to one storage array while staging
> > > > diskimages to tape from the other storage array.  I'm able to
> > > > alternate the active backup storage unit daily.
> > > >
> > > > In order to complete all of the backups during typical non-workday
>
> > > > hours, I've set NUMBER_DATA_BUFFERS to 24 and SIZE_DATA_BUFFERS to
>
> > > > 262144.
> > > >
> > > > The backups are finishing in their configured windows, but the CPU
>
> > > > load is extremely high. The uptime commands shows a load of 25-40
> > > > all during my backup windows and top shows the bpdm processes as
> > > > using
> > > > 1.0-1.4 percent of the CPU.
> > > >
> > > > I'm curious to see if there are others seeing high loads caused by
>
> > > > bpdm whis is used for both disk and disk-staging storage units.
> > > > I'd like to hear about how others are configuring NetBackup for
> > > > backups to disk, mostly regarding the size of the disk storage
> > > > units, NUMBER_DATA_BUFFERS, SIZE_DATA_BUFFERS, and number of
> > concurrent jobs.
> > > >
> > > > Thanks for any information you can give me.
> > > >
> > > > --kathy
> > > >
> > > > _______________________________________________
> > > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > > >
> > >
> > > --kathy
> > >
> >
> > --kathy
> >
>
> --kathy
>
> ========================================================================
> =======
> Kathryn Hemness                        kfhemness AT ucdavis DOT edu
> System Administrator                   phone: 530.752.6547
> Campus Data Center & Client Services   fax:   530.752.9154
>

--kathy

===============================================================================
Kathryn Hemness                        kfhemness AT ucdavis DOT edu
System Administrator                   phone: 530.752.6547
Campus Data Center & Client Services   fax:   530.752.9154