Veritas-bu

[Veritas-bu] Netbackup benchmark & performance tuning

2007-02-28 09:39:58
Subject: [Veritas-bu] Netbackup benchmark & performance tuning
From: Ueli.Schweizer at AGITE-Software.com (Ueli Schweizer)
Date: Wed, 28 Feb 2007 15:39:58 +0100
Hi Vijay

If your FS can be read with maximum 30 MB/s then, because of possible
shoe-shining, a backup throughput of 22 MB/s is more or less what I would
expect.

There are not many options to improve the performance of this single backup.
I would try to multiplex the backup with other backups so the tape drive
will not come into shoe shining. This will probably improve the performance
a little bit, but it is in any case healthier for your hardware (tape drives
and tapes hate show shining).

Buffer tuning may help a bit, but as mentioned by Bob: 128 x 256 KB is
usually ways enough for LTO2 so I don't expect this will change a lot.

Cheers

Ueli Schweizer

AGITE Software AG . Boesch 43 . CH-6331 Huenenberg . Switzerland
Direct: +41 79 204 9190 . Phone: +41 41 781 5678 . Fax: +41 41 781 5677
Ueli.Schweizer at AGITE-Software.com . http://www.AGITE-Software.com 

backupVISUAL ...
                  ... we make life on backup easy

-----Original Message-----
From: Vijay Kumar [mailto:vk_pune at yahoo.com] 
Sent: Wednesday, February 28, 2007 3:07 PM
To: Ueli.Schweizer at AGITE-Software.com; veritas-bu at mailman.eng.auburn.edu
Subject: RE: [Veritas-bu] Netbackup benchmark & performance tuning

Hi Ueli,

I checked the Disk I/O with following command
/usr/openv/netbackup/bin/bpbkar -nocont -dt 0
-nofileinfo -nokeepalives /usr/openv > /dev/null
This scenario I got around 170Mbps for 5Gb data of
/usr/openv file system

But after ur mail I tried with / file system
Output of this surprised me as this time I got only
30Mbps. So I rebooted and chekced again but same
througput. Again I executed same command that time I
got more than 150Mbps through put but I think it took
the data from memory.

So from this I can say I am getting Disk I/o around
30Mbps

And about logs 
Yes it is showing BPTM is waiting for data from the
source. For this what changes are required.

Thanks & regards,
Vijay




--- Ueli Schweizer <Ueli.Schweizer at AGITE-Software.com>
wrote:

> Hi Vijay
> 
> "Waited for full buffers" messages during a backup
> means that the tape is
> able to write more data than bpbkar is able to send.
> In this example the
> "11254 delays" indicates, that the backup would
> finish 11254 * 30 msec = 338
> seconds faster under <<optimum>> conditions. Of
> course it is very important
> to know how much data you've backed up during this
> attempt to find out if
> this number is extremely high or not. But anyway: it
> indicates that the
> filesystem can not be read as fast as the data can
> be written to tape.
> 
> A question to your performance test using bpbkar:
> What path are you using as
> <filesystem>? If you are using the device path (e.g.
> /dev/dsk/c1t0d0s5) you
> are reading the raw disk. Only if your using the
> mountpoint (e.g. /usr) you
> are reading the filesystem. Depending on the
> fragmentation of a disk the
> differences in performance are just huge. For me the
> number you measured
> (170 MB/s) seems to be very (actually VERY) high for
> filesystem backup, so
> probably you're backing up raw disk data or the
> measured performance was
> affected by read cache or somewhat else.
> 
> For me the 22 MB/s looks much more realistic than
> the 170 MB/s. For FS
> backup in general and non-RAID OS disk in special I
> think 22 MB/s is just
> average and I'm not sure it there is much tuning on
> this. I would focus on a
> tuning configuration parameters including MPX.
> 
> Cheers
> 
> Ueli Schweizer
> 
> AGITE Software AG . Boesch 43 . CH-6331 Huenenberg .
> Switzerland
> Direct: +41 79 204 9190 . Phone: +41 41 781 5678 .
> Fax: +41 41 781 5677
> Ueli.Schweizer at AGITE-Software.com .
> http://www.AGITE-Software.com 
> 
> backupVISUAL ...
>                   ... we make life on backup easy
> -----Original Message-----
> From: veritas-bu-bounces at mailman.eng.auburn.edu
> [mailto:veritas-bu-bounces at mailman.eng.auburn.edu]
> On Behalf Of Vijay Kumar
> Sent: Wednesday, February 28, 2007 12:55 PM
> To: bob944 at attglobal.net;
> veritas-bu at mailman.eng.auburn.edu
> Subject: Re: [Veritas-bu] Netbackup benchmark &
> performance tuning
> 
> Bob, Paul, murtuja
> 
> Some clarification. Backup of local server means
> Netbackup Master/Media server itself not any other
> system from the network. 
> 
> I am trying to backup OS Disk. NO raid
> configuration.
> 
> # df -kFufs
> Filesystem            kbytes    used   avail
> capacity 
> Mounted on
> /dev/dsk/c1t0d0s0    8263373  180817 7999923     3% 
>  
> /
> /dev/dsk/c1t0d0s5    19623529 1458826 17968468    
> 8% 
>   /usr
> /dev/dsk/c1t0d0s3    10327372 7534947 2689152    74%
>  
>  /var
> /dev/dsk/c1t0d0s4    15493995   30262 15308794    
> 1% 
>   /opt
> /dev/dsk/c1t1d0s0    41311843 5549567 35349158   
> 14% 
>   /usr/openv
> 
> 
> Following error message i am getting in bptm logs
> 
> 11:54:09.851 [927] <2> write_data: waited for full
> buffer 7898 times, delayed 11254 times
> 
> May be this is the bottleneck. So i tried to
> configure
> different buffer size but still messages.
> 
> I checked the Disk I/O by using 
> /usr/openv/netbackup/bin/bpbkar -nocont -dt 0
> -nofileinfo -nokeepalives filesystem > /dev/null
> Here I am getting Disk speed more that 170Mbps
> 
> Yes Bob its in magabytes/second
> 
> Thanks & regards,
> Vijay
> 
> 
> --- bob944 <bob944 at attglobal.net> wrote:
> 
> > First, congratulations on supplying a lot of
> useful
> > info in the first
> > posting.
> > 
> > > We are using Netbackup 5.1MP6 on Solaris 9.
> > > H/W Detasils : 
> > > Master/Media Server : v440, 4CPU, 16Gb RAM
> > > Tape Library : L500, 4 F-CAL Drive (LTO Gen 2)
> > > Network Speed : 1GBPS
> > > Tape library is directly connected to server.
> > > 
> > > Currently when trying to take backup of local
> > server i
> > > am getting 22Mbps throughput which i feel is
> very
> > > less. 
> > 
> > Not to quibble, but I'm sure you are saying 22
> > megabytes, not megabits,
> > right?  (I know someone who sized his tape HBA
> > requirements thinking the
> > specs were MB/s, when they were, of course,
> Mb/s--so
> > it's worth being
> > rigorous on the notation.)
> > 
> > > Current Buffer settings:
> > > NET_BUFFER_SZ : 262144
> > > NUMBER_DATA_BUFFERS :128
> > > SIZE_DATA_BUFFERS : 262144
> > > 
> > > /etc/system:
> > > *Added for Nebackup
> > > set msgsys:msginfo_msgmap=512
> > > [...]
> > > 
> > > So i would like to know  practically what is the
> > > maximum throughput i can get & buffer setting.
> > What
> > > all changes i need to do on server to get
> maximum
> > > throughput
> > 
> > LTO2 drives aren't all that fast--half-height
> LTO2s
> > are
> > manufacturer-rated at something like 24-28MB/s
> > native and the best
> > rating I know of is 35MB/s.  See point #2 below.
> > 
> > You want three things:  
> > 1.  NetBackup 6.0 Performance Tuning Guide
> > 2.  Follow the guide to determine where the
> > bottlenecks are--there's no
> > point in tuning NetBackup buffers if, say, the
> > client can't supply data
> > fast enough.
> > 3.  When you can demonstrate to yourself that
> > NetBackup is the slow link
> > in the chain, then it is worth a controlled,
> > rigorous buffer tuning
> > exercise.
> > 
> > 128 256KB buffers is way more than enough to drive
> > an LTO2.  
> > 
> > 
> 
> 
> 
>  
>
____________________________________________________________________________
> ________
> Don't pick lemons.
> See all the new 2007 cars at Yahoo! Autos.
> http://autos.yahoo.com/new_cars.html 
> _______________________________________________
> Veritas-bu maillist  - 
> Veritas-bu at mailman.eng.auburn.edu
>
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> 



 
____________________________________________________________________________
________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com