Networker

Re: [Networker] How can data, or number of files, cause extremely slow write speeds?

2009-05-13 14:40:59
Subject: Re: [Networker] How can data, or number of files, cause extremely slow write speeds?
From: Len Philpot <Len.Philpot AT CLECO DOT COM>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 13 May 2009 13:36:56 -0500
> George Sinclair 
> 
> If you have a very large number of files on a given file system can this 

> slow down the backup write speeds, not the actual completion times, but 
> the speed of the writing to tape itself?

I've always assumed (but I'm all ears to learn better) that the displayed 
write speeds are calculated like most things of that type are: amount / 
time. So, if there are lots of files, per-file procedures internal to 
Networker will be repeated much more often than for a large file of 
equivalent size and performance will suffer (i.e., poor streaming). 
Subsequently, the computed write speeds will be slow(er) if they're not 
calculated indivdually for each file.

Or is that an incorrect deduction?

But, I've seen the exact same thing, although not to the scale you 
describe. Rarely do we have single savesets with more than a few hundred 
thousand small (similar) files. Some have more files, but they vary in 
size and composition. I assume all the file opening/closing is a major 
culprit...?

I guess I've just assumed that excessive_number_of_smaller_files = 
slower_backup_performance, since that's been my experience. Maybe I'm 
wrong?

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER