Veritas-bu

[Veritas-bu] LTO2 backup speed just up to 5 MB/sec under Netbackup 4.5

2004-03-15 08:27:54
Subject: [Veritas-bu] LTO2 backup speed just up to 5 MB/sec under Netbackup 4.5
From: mmlist AT mjmm DOT org (Michael Miller)
Date: Mon, 15 Mar 2004 13:27:54 -0000 (GMT)
Hi

on solaris the system call is failing due to the large size of the shared
memory NBU is trying to allocate.  The buffers are allocated from shared
memory.  You need to have atleast the following amount of shared memory
available:

SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS *
CONCURRENT_ACTIVE_JOBS_ON_THIS_MEDIA_SERVER

So, if the size is 64k and number is 16 and you have 2 drives with mpx of
8 you would calculate shared memory as:

64k * 16 * (2 * 8) = 16384k = 16MB

To tune this you modify /etc/system shm values, we run for example:

* Shared memory
set shmsys:shminfo_shmmax=1073741824
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=220
set shmsys:shminfo_shmseg=100


You may need to change semsys or msgsys parameters too.  you need to
reboot for the /etc/system changes to take effect.  As you can see, our
largest shared memory segment is 1GB on the box I took the above config
from.

Good luck.  I think with LTO drives,you may be able to push the
data_buffer_size to 256k (262144 bytes). And crank the buffers to, say,
32. That would require 8MB of shared memory PER ACTIVE backup.  Notice
that your drive could write that buffer to tape in less than a second, so
if you multiplex 4 clients/streams, your buffer requirements would just to
32MB.

Good luck... Just keep cranking the buffers up till your performance tails
off. If your server is purely a netbackup media server, you can safely aim
to use half (or maybe more even...) the machines ram for buffers...

The whole idea is to keep the drive streaming data, when the drive has to
stop and backtrack and restart writing due to lack of data, your
performance will suck.

Enable logging for the bptm process on the media server and look for
waited for empty / full buffers.  Waiting for full buffers means your tape
is writing faster than the media server can source data... if you are
waiting for empty buffers you are sourcing more data than you can write to
tape... try to avoid waiting for full buffers by incresasing the number of
buffers available.. and maybe try multiplexing multple clients together.

good luck!!

> Hi Douglas and Mark,
>
> Thanks for your input.
>
> I have created the files SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS with
> value of 64512 and 16 respectively, the throughtput has a significant
> improvement at about
> 19 MB /sec .
>
> I have tried the following combinations but it seems the above values pair
> is the best:
>
> SIZE_DATA_BUFFERS     32768   64512   64512   64512   131072  131072  262144  
> 262144
> NUMBER_DATA_BUFFERS   16      8       16      24      4       8       2       
> 4
> NET_BUFFER_SZ 32768   64512   64512   64512   131072  131072  262141  262141
> THROUGHPUT (KB/sec)   9662    9721    18758   failed  9887    failed  9580    
> failed
>
> I am still looking for an optimum value pairs.
>
> For those failed cases, I got the following errors:
>
> 11:27:13.070 [19805] <16> io_init: system call failed - Invalid argument
> (at
> tmcommon.c.1976)
> 11:27:13.082 [19805] <16> io_init: could not allocate enough shared memory
> for backup buffers
>
> How I can adjust the system shared memory in this case?
>
> Regards,
> Chadwick
>
>
>>From: Douglas Ritschel <dritschel62 AT yahoo DOT com>
>>To: mark <mark AT steelfamily DOT org>, Chadwick Leung <poleetea AT hotmail 
>>DOT com>,
>>veritas-bu AT mailman.eng.auburn DOT edu
>>Subject: RE: [Veritas-bu] LTO2 backup speed just up to 5 MB/sec under
>>Netbackup 4.5
>>Date: Sun, 14 Mar 2004 07:03:50 -0800 (PST)
>>
>>You can increase the blocksize that netBackup uses, by
>>creating a file:
>>/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS. The
>>file should contain one line with powers of 2 times
>>1024.
>>
>>For example, 256 times 1024:
>>
>>262144
>>
>>You can also increase the number of data buffers by
>>creating a file:
>>
>>/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS
>>
>>It should contain one line which is a multiple of 2,
>>the max is 256.
>>
>>Also can increase the fragment size of the storage
>>unit to 2000.
>>
>>
>>
>>
>>--- mark <mark AT steelfamily DOT org> wrote:
>> > On 4.5 the limitation of a backup from local disk to
>> > a tape is bpbkar which
>> > has a block size of 32kb internally. this is fixed
>> > and cannot be tuned,
>> > whereas tar or cpio you can set a much larger
>> > blocksize.
>> >
>> > in netbackup 5.0 this blocksize is 128k, but still
>> > not tuneable. best thing
>> > is to get multiple bpbkar processes running in
>> > parallel by breaking the
>> > filelist into streams of drives / mounts or files -
>> >
>> > ie rather than backing up
>> >
>> >
>> > /usr
>> >
>> > run
>> >
>> > /usr/*
>> >
>> > or
>> >
>> > /usr/[a-m]*                        ( i think you can do this on unix
>> > clients, but not windows )
>> > /usr/[n-z]*
>> >
>> > and you will get a different backup stream for each
>> > directory. This isn't a
>> > good example on /usr - you have to make sure the
>> > spec you pick doesnt
>> > generate too many streams which are very small, and
>> > also make sure the
>> > wildcard spec has no 'gaps' otherwise some point
>> > down the line you might
>> > find you haven't been backing everything up.
>> >
>> > regards
>> > mark
>> >
>> >
>> > -----Original Message-----
>> > From: veritas-bu-admin AT mailman.eng.auburn DOT edu
>> > [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]On
>> > Behalf Of Chadwick
>> > Leung
>> > Sent: 14 March 2004 06:23
>> > To: veritas-bu AT mailman.eng.auburn DOT edu
>> > Subject: [Veritas-bu] LTO2 backup speed just up to 5
>> > MB/sec under
>> > Netbackup 4.5
>> >
>> >
>> > Greeting!
>> >
>> > I have installed a Sun L8 autoloader which replaces
>> > a Sun L9 model . The L8
>> > has been successfully installed and can operate
>> > without problem. However,
>> > the average throughput is just up to 5 MB/sec under
>> > Netbackup 4.5.
>> >
>> > I tried to TAR a 1.2 GB file direct to the LTO2
>> > tape, it took about a
>> > minute. Under Netbackup, backing up the same file
>> > take about 4 mins. 30 sec.
>> >
>> > Any idea?
>> >
>> > Best Regards,
>> > Chadwick Leung
>> >
>> >
>>_________________________________________________________________
>> > Get 10Mb extra storage for MSN Hotmail. Subscribe
>> > Now!
>> > http://join.msn.com/?pgmarket=en-hk
>> >
>> > _______________________________________________
>> > Veritas-bu maillist  -
>> > Veritas-bu AT mailman.eng.auburn DOT edu
>> >
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>> >
>> >
>> >
>> > _______________________________________________
>> > Veritas-bu maillist  -
>> > Veritas-bu AT mailman.eng.auburn DOT edu
>> >
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>
>>
>>__________________________________
>>Do you Yahoo!?
>>Yahoo! Mail - More reliable, more storage, less spam
>>http://mail.yahoo.com
>
> _________________________________________________________________
> Linguaphone :  Learning English? Get Japanese lessons for FREE
> http://go.msnserver.com/HK/30476.asp
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>