Veritas-bu

[Veritas-bu] NB51MP3 high cpu loads caused by bpdm?A

2005-07-11 11:39:18
Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?A
From: kfhemness AT ucdavis DOT edu (Kathryn Hemness)
Date: Mon, 11 Jul 2005 08:39:18 -0700 (PDT)
Thanks, Gregory -

Someone else on the list responded with the bpschedreq commands.

I haven't yet looked over all of my backup logs from Saturday and
Sunday, but I noticed that the load average on my server was less than
1 this morning (it's usually between 10 and 20).

I also noticed in my bpdm logs several messages about checkpoints.
I have 15-minute checkpoint restart enabled on several of my policies
which are backing up large filesystems; I'm planning on disabling the
checkpoint restart on the policies and see if it impacts the cpu load.

Also, have you heard anything about whether or not the SIZE_DATA_BUFFERS
and SIZE_DATA_BUFFERS_DISK should have the same values?  I'm wondering
if differences in the values contribute to load or other resource waste.



On Mon, 11 Jul 2005, Geyer, Gregory wrote:

> Date: Mon, 11 Jul 2005 07:40:26 -0700
> From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
> Kathy,
>
> I've noticed to add new storage units I have to bounce Netbackup.
> However, to make a change to a storage unit I just have to run these
> commands to make them stick.  See if it works for you:
>
> bpschedreq -read_stu_config
> bprdreq -rereadconfig
>
> Someone also has posted "bpsched -read_stunits" but I haven't run this
> in the past.
>
> G.
>
> -----Original Message-----
> From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of Kathryn
> Hemness
> Sent: Saturday, July 09, 2005 8:43 AM
> To: Geyer, Gregory
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
>
> Well, last night's backups as a test of the new settings was a wash.
> I'm still seeing fragment sizes of 2000MB on the storage unit whose
> configurations I changed.  It looks like I need to restart the NetBackup
> daemons when I modify the storage unit settings. sigh.
>
> I'll be modifying my other storage unit fragment size after all
> duplications from it finish this afternoon.
>
> You can be sure I'll stop and restart the daemons this time.
> another sigh.
>
>
> On Fri, 8 Jul 2005, Geyer, Gregory wrote:
>
> > Date: Fri, 8 Jul 2005 11:37:42 -0700
> > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >
> > Kathy,
> >
> > Excellent news.  Please let me know how your weekend backups turn out
> > with your new settings.  I'm going to review my setup next week and go
>
> > back to experimenting with these settings.
> >
> > Thanks!
> >
> > Greg
> >
> > -----Original Message-----
> > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis DOT 
> > edu] On
> > Behalf Of Kathryn Hemness
> > Sent: Friday, July 08, 2005 10:22 AM
> > To: Geyer, Gregory
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu; Bill_Jorgensen AT csgsystems 
> > DOT com
> > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> >
> > Hi Gregory --
> >
> > I made the following changes on my server and the number of 'waited
> > for full buffer' messages in my bpdm log decreased by half.  Neither
> > the cpu load nor the 'waited for empty buffer'
> > showed any significant change:
> >
> > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_DISK:     16
> > /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK:  1048576
> > /etc/system: set shmsys:shminfo_shmmax=536870912
> >
> >
> > I left the tape data buffers the same:
> > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS 16
> > /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS 262144
> >
> > And my storage unit fragment size was still 2000 MB.
> >
> > For my backups tonight, I plan on increasing the
> > NUMBER_DATA_BUFFERS_DISK to 24 and decreasing the fragment size to
> > 1000 MB.
> >
> >
> > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> >
> > > Date: Wed, 6 Jul 2005 15:52:41 -0700
> > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> > > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > >
> > > Kathy,
> > >
> > > Not sure, I've only experimented with them lately but I think it's a
>
> > > case of trying a setting and watching and then perhaps tweaking them
>
> > > until you've found a good spot.  I'm in your boat as well: still
> > > looking for the best settings.
> > >
> > > G.
> > >
> > > -----Original Message-----
> > > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> > > DOT edu]
> > > On Behalf Of Kathryn Hemness
> > > Sent: Wednesday, July 06, 2005 3:45 PM
> > > To: Geyer, Gregory
> > > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > >
> > > Thanks, Gregory -
> > >
> > > Is there a good rule of thumb for the number of disk data buffers
> > > based on the size of the storage unit?  I've set the number to 16
> > > for my backups tonight.  Is the number of disk data buffers supposed
>
> > > to be
> >
> > > a power of 2 like the non-disk NUMBER_DATA_BUFFERS?
> > >
> > > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> > >
> > > > Date: Wed, 6 Jul 2005 15:31:17 -0700
> > > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> > > > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > >
> > > > A trick a Veritas instructor gave me last year was this:
> > > >
> > > > grep out "waited for full buffer" and "waited for empty buffer"
> > > > out of
> > >
> > > > your bpdm/bptm logs.
> > > >
> > > > If you see many "waited for full buffer" in your bpdm/bptm log
> > > > then the storage unit disk/tape is waiting for the data buffer to
> fill.
> > > > Tune down the size of the data buffer so it fills quicker.
> > > >
> > > > If you see many "waited for empty buffer" then the server is
> > > > waiting
> >
> > > > on the storage unit.  Increase the number of buffers.
> > > >
> > > > G.
> > > >
> > > > -----Original Message-----
> > > > From: cckat AT reality.ucdavis DOT edu [mailto:cckat AT reality.ucdavis 
> > > > DOT edu]
> > > > On Behalf Of Kathryn Hemness
> > > > Sent: Wednesday, July 06, 2005 3:18 PM
> > > > To: Geyer, Gregory
> > > > Cc: Kathryn Hemness; veritas-bu AT mailman.eng.auburn DOT edu
> > > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > >
> > > > Thanks.  I'll give the DISK BUFFER settings a try.  I was kind of
> > > > wondering if a smaller buffer size might be better.
> > > >
> > > >
> > > > On Wed, 6 Jul 2005, Geyer, Gregory wrote:
> > > >
> > > > > Date: Wed, 6 Jul 2005 15:00:09 -0700
> > > > > From: "Geyer, Gregory" <Gregory.Geyer AT Avnet DOT com>
> > > > > To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>,
> > > > >      veritas-bu AT mailman.eng.auburn DOT edu
> > > > > Subject: RE: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > > >
> > > > > Kathy,
> > > > >
> > > > > I believe SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS are for bptm
> > > only.
> > > > > I've only just discovered "SIZE_DATA_BUFFERS_DISK" and
> > > > > "NUMBER_DATA_BUFFERS_DISK" for use by bpdm.
> > > > >
> > > > > However I also experienced shared memory issues with these as
> > > > > we're a heavy disk-staging shop.  These might help your speeds,
> > > > > though I don't
> > > >
> > > > > know about offloading your CPU.
> > > > >
> > > > > Reference Document ID 273532 for the above info.
> > > > >
> > > > > G.
> > > > >
> > > > > -----Original Message-----
> > > > > From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> > > > > [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of
> > > > > Kathryn Hemness
> > > > > Sent: Wednesday, July 06, 2005 2:38 PM
> > > > > To: veritas-bu AT mailman.eng.auburn DOT edu
> > > > > Subject: [Veritas-bu] NB51MP3 high cpu loads caused by bpdm?
> > > > >
> > > > > Greetings --
> > > > >
> > > > > I am running NetBackup 5.1 MP3 on a Sun V240 and using Sun
> > > > > StorEdge
> > > > > 3511 storage arrays with 6TB capacity for backups to disk (not
> > > > > using
> > >
> > > > > the disk-staging configuration).  The OS is Solaris 9.
> > > > > The backups are first written to the storage array and then
> > > > > staged
> >
> > > > > to tape using cron-script managed duplications so that I can use
>
> > > > > backupid-files containing backupids having the same retentions
> > > > > to minimize tape mounts.  My tape library is a Qualstar 88264
> > > > > LTO2 library with 3 LTO2 drives.
> > > > >
> > > > > To optimize read and write speeds the storage arrays, I
> > > > > alternate directing all active backups to one storage array
> > > > > while staging diskimages to tape from the other storage array.
> > > > > I'm able to alternate the active backup storage unit daily.
> > > > >
> > > > > In order to complete all of the backups during typical
> > > > > non-workday
> >
> > > > > hours, I've set NUMBER_DATA_BUFFERS to 24 and SIZE_DATA_BUFFERS
> > > > > to
> >
> > > > > 262144.
> > > > >
> > > > > The backups are finishing in their configured windows, but the
> > > > > CPU
> >
> > > > > load is extremely high. The uptime commands shows a load of
> > > > > 25-40 all during my backup windows and top shows the bpdm
> > > > > processes as using
> > > > > 1.0-1.4 percent of the CPU.
> > > > >
> > > > > I'm curious to see if there are others seeing high loads caused
> > > > > by
> >
> > > > > bpdm whis is used for both disk and disk-staging storage units.
> > > > > I'd like to hear about how others are configuring NetBackup for
> > > > > backups to disk, mostly regarding the size of the disk storage
> > > > > units, NUMBER_DATA_BUFFERS, SIZE_DATA_BUFFERS, and number of
> > > concurrent jobs.
> > > > >
> > > > > Thanks for any information you can give me.
> > > > >
> > > > > --kathy
> > > > >
> > > > > _______________________________________________
> > > > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > > > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > > > >
> > > >
> > > > --kathy
> > > >
> > >
> > > --kathy
> > >
> >
> > --kathy
> >
> > ======================================================================
> > ==
> > =======
> > Kathryn Hemness                        kfhemness AT ucdavis DOT edu
> > System Administrator                   phone: 530.752.6547
> > Campus Data Center & Client Services   fax:   530.752.9154
> >
>
> --kathy
>
> ========================================================================
> =======
> Kathryn Hemness                        kfhemness AT ucdavis DOT edu
> System Administrator                   phone: 530.752.6547
> Campus Data Center & Client Services   fax:   530.752.9154
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>

--kathy

===============================================================================
Kathryn Hemness                        kfhemness AT ucdavis DOT edu
System Administrator                   phone: 530.752.6547
Campus Data Center & Client Services   fax:   530.752.9154

<Prev in Thread] Current Thread [Next in Thread>