|
Re: [Networker] How can data, or number of files, cause extremely slow write speeds?
2009-05-13 16:46:01
A Darren Dunham wrote:
On Wed, May 13, 2009 at 03:30:09PM -0400, George Sinclair wrote:
Len Philpot wrote:
George Sinclair
If you have a very large number of files on a given file system can this
slow down the backup write speeds, not the actual completion times, but
the speed of the writing to tape itself?
What if browsing was turned off? To do this, I'd have to set up a
separate tape pool. That's a pain and kind of silly for just one path,
but is doable. If browsing is turned off then NW has no metadata to
manage except that 1. it would still need to determine the fsize
beforehand for comparison after the backup for the given file completes
and 2. it still has a lot of open and close operations to perform.
That's not going to change. So, the next question would be which is a
more significant performance hit: the fact the browsing is enabled or
the fact that there's a lot of files to open and close, never mind the
fact the browsing were disabled?
I would expect a non-browseable backup to be similar in impact to
running 'tar cf /dev/null /filesystem'. Tar is going to find, open, and
read every file. When the files are tiny, that work dominates the job.
The fact that it's writing to a fast /dev/null instead of a limited
network pipe won't matter much. So try that and see how long it takes.
If it's similar to your current backup time, then you won't see much
benefit. If it's significantly faster, then it might be worthwhile.
Well, I just ran a tar to /dev/null of the whole enchilada, and it
finished in about 30-45 minutes. Wow! That was fast. Contrast that with
the browsable backup I was running this past weekend that ran for nearly
2 days and only backed up 110 GB before I finally aborted it. I think
I'll create a test pool with 'store index entries' turned off and re-run
a level full and watch to see what the write speeds average over, say, a
half hour or so and then stop it and run it again but this time with
'store index entries' turned back on.
Can you just switch the 'store index entries' on and off, or do you have
to also relabel or recycle the tape back into the same tape pool after
making that change?
Either way, I'd probably recycle the tape into another pool to get rid
of those index entries that were created from the second test and the
media db entries created from both tests.
Where it might matter is if you're getting middling tape speeds causing
some slowdown from the tape. But it sounds like you're getting very
slow speeds that won't have this effect.
--
George Sinclair
Voice: (301) 713-3284 x210
- The preceding message is personal and does not reflect any official or
unofficial position of the United States Department of Commerce -
- Any opinions expressed in this message are NOT those of the US Govt. -
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|
