|
Re: [Networker] How can data, or number of files, cause extremely slow write speeds?
2009-05-13 15:35:09
Len Philpot wrote:
George Sinclair
If you have a very large number of files on a given file system can this
slow down the backup write speeds, not the actual completion times, but
the speed of the writing to tape itself?
What if browsing was turned off? To do this, I'd have to set up a
separate tape pool. That's a pain and kind of silly for just one path,
but is doable. If browsing is turned off then NW has no metadata to
manage except that 1. it would still need to determine the fsize
beforehand for comparison after the backup for the given file completes
and 2. it still has a lot of open and close operations to perform.
That's not going to change. So, the next question would be which is a
more significant performance hit: the fact the browsing is enabled or
the fact that there's a lot of files to open and close, never mind the
fact the browsing were disabled?
I've always assumed (but I'm all ears to learn better) that the displayed
write speeds are calculated like most things of that type are: amount /
time. So, if there are lots of files, per-file procedures internal to
Networker will be repeated much more often than for a large file of
equivalent size and performance will suffer (i.e., poor streaming).
Subsequently, the computed write speeds will be slow(er) if they're not
calculated indivdually for each file.
Or is that an incorrect deduction?
But, I've seen the exact same thing, although not to the scale you
describe. Rarely do we have single savesets with more than a few hundred
thousand small (similar) files. Some have more files, but they vary in
size and composition. I assume all the file opening/closing is a major
culprit...?
I guess I've just assumed that excessive_number_of_smaller_files =
slower_backup_performance, since that's been my experience. Maybe I'm
wrong?
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
--
George Sinclair
Voice: (301) 713-3284 x210
- The preceding message is personal and does not reflect any official or
unofficial position of the United States Department of Commerce -
- Any opinions expressed in this message are NOT those of the US Govt. -
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|
