On Wed, May 13, 2009 at 12:45:02PM -0400, George Sinclair wrote:
> If you have a very large number of files on a given file system can this 
> slow down the backup write speeds, not the actual completion times, but 
> the speed of the writing to tape itself?

Sure, because you can't write faster than you can read.  If all the
files are tiny (very likely when you have that many of them), then
you're spending more time in the filesystem trying to gather metadata,
transfer that to the backup indexes, figure out the next file to backup,
etc. than you are actually reading file data and writing it to tape.

> We have an ext3 file system on a Linux box that is extremely painful to 
> back up because it chugs along at less than 100 KB/sec. I finally gave 
> up and canceled the backup. I was running a level full. The strange 
> thing is that all the rest of the data on that box, located on the other 
> ext3 file systems, backs up at normal speed, even to the same tape and 
> drive (LTO-3).

Small files take a long time to back up if done through the filesystem.
You should see a similar (but probably not identical) slowdown just
doing a 'tar cf /dev/null /filesystem'.  The actual backup will be even
slower because the backup server has to do work recording all the
filenames and metadata as well.

> It's consuming less than half the disk space (300 out of 750 GB). 
> However, that particular file system is also consuming over 10 million 
> inodes. Nasty! As a result, running commands like 'du' or 'find' can 
> take a long time to complete. I'm not sure if the data is binary. It 
> could be, and this would obviously adversely affect compression, so I 
> would expect that we might see slower write speeds as a result. Also, if 
> you're only sending a single save stream to tape, it will be slower than 
> multiplexing multiple streams. All that makes sense, but I've never seen 
> anything this slow, even when running a single save set, except in 
> extreme cases where there was a switch problem, NIC issue, or extremely 
> slow client, but again, the other backups on that host do run at normal 
> speed.

> I think NetWorker walks through save sets before it backs them up so it 
> knows if a file's size has changed once it starts to back it up?

Not really.  That walk could take a *very* long time on big filesystems.
It does note the size when it starts that particular file, and if it
changes before that file is finished it will note it.

I think you can make it do the prewalk with "-E" (to estimate the size),
but I don't think that happens normally.

> Does anyone have any suggestions on how we might be able to speed this 
> up? The only thing I can think of is to identify the minimal number of 
> areas under there that need to be backed up and skip the rest, maybe 
> break those minimal backups into multiple save sets to multiplex the 
> data and force faster write speeds? I could turn compression off on the 
> drive, but I don't wanna leave it that way as it generally works to our 
> favor on most data. Would be a pain to dedicate a non-compression drive 
> to just one chunk of data.

SnapImage might be an option here.  If you bypass the filesystem and
just grab the blocks, the backup goes faster.

-- 
Darren

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER