Veritas-bu

[Veritas-bu] Lots of files: Backup Performance

2000-05-16 10:13:20
Subject: [Veritas-bu] Lots of files: Backup Performance
From: Dayne Medlyn dayne_medlyn AT agilent DOT com
Date: Tue, 16 May 2000 08:13:20 -0600
Originally, our target was to recover a 200GB NT volume, with about 1M
files, in less than 12 hours.  Extrapolating out, this meant backing up
the volume in about 8-10 hours.  Now, in order to do this there are
quite a few factors, namely:

* the speed of your disks
* the speed of your tape drives
* the speed of your bus (both internal and external -- like a network)
* and of course, the configuration of the backup

So, here is what we found on NT (haven't done the same rigorous
qualification for UX) using NetBackup (3.1.5 at the time):

The speed of your disks -- 
The old HP NetRaid controlers didn't seem to be able to source the data
any faster than about 12 GB/hr.  Newer NetRaid worked a bit better --
~17GB/hr if memory serves me.  EMC hosted disk worked rather well,
~36GB/hr.  Single volumes vs multiple volumes vs raid makes a big
difference.  This is a big factor.

The speed of your tape drives --
As it turns out with DLT7000, if you do not keep the tape streaming you
get worse performance than a DLT4000.  Generally we tried to keep 2
drives streaming.  But, as you point out, with compression the DLT7000
should be able to push upwards of 10mb/s = 36 Gb/h.  I never got
anywhere close to this with one drive.  I believe the drives are rated
at 5 mb/s with no compression.  Compression hurts you on the streaming
part too.

The speed of your bus --
In general, the internal bus of most systems should be able to handle
the data you are going to push across it.  But, you can overload a bus
too.  The network is a different issue.  NT has limited ability to fully
use the network.  I think the best we got was about 14 GB/hr over
100TX.  Gigabit helped a little, but you still don't really get gigabit
speed.

The configuration of the backup --
This is the trickiest piece.  The trick with lots of small files is lots
of streams.  With NT it turns out that two streams of the same volume is
significantly faster than a single stream.  We did not exactly find the
limit, but it is asymptotic.  I have run as many as 32 streams from a
single volume (how I got 36GB/hr).  So, the trick is to be able to
divide up your files into equal groups so you can put them into lots of
streams.  Not an easy task, but NBU 3.2 has a couple of things that help
this.

All that said, given that you have fast enough disks, enough tape drives
(you may need more than one), and enough data streams, you should be
able to source and sync the data from disk to tape very quickly with
NetBackup.  We selected NetBackup, over others, specifically for the
performance issue.

Hope this helps.

Dayne Medlyn



George Roper wrote:
> 
> How does NBU perform when asked to backup a lot of small files within the
> same filesystem?
> 
> I have need to backup 1,600,000 files averaging 2-3k each - total of 30GB of
> data. My backup to a single DLT7000 tape unit currently takes ~50 hours.
> 
> Theorectically if the system were able to drive the DLT at 5MB/s this should
> take 100 minutes. NBU obviously has lots of overhead when recording to the
> Files Database each of these 1,600,000 files. Also the backup process must
> slow when walking the directory to determine what file to backup next.
> 
> Before I start tuning this environment I would like to know what is a
> reasonable backup window to expect.
> 
> Anyone any experience or observations?
> 
> George
> ________________________________________________________________________
> Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
> 
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu



<Prev in Thread] Current Thread [Next in Thread>