I would agree with this but also add that "smal files" is a problem for
any filesystem (that I've used ) as well as the walking of the filesystem
there will be a size of file where the time taken to access the
information about the file (name, permissions etc) is the same as the time
taken to actually read the data contained in the file. This will vary
depending on CPU, disk i/o and filesystem design.
A few years ago we tested this on a fairly new wintel server and found
that this point was reached when the file was 200KBytes this for us meant
that the backup of this system when the average file size was <200KB was
not dependant on the size of the partition but the number of files. Newer
and more powerful CPU servers did not improve this situation. In this case
I would be mildly surprized if any increase in parrellism of the reads
would result in faster backups.
We didn't repeat these test on our Solaris servers but we could see from
our estate that the "small file" size was larger than 200KBytes.
Some statements that Networker is bad with small files/large numbers of
files are (in my opinion ) misdirected all filesystem based backups will
have this problem, to avoid it you need to ignore the filesystem and read
disk blocks directly.
On Thu, 19 Mar 2009 11:29:13 -0500, Tim Mooney <Tim.Mooney AT NDSU DOT EDU>
wrote:
>In regard to: [Networker] How to speed up this backup (4GB -> 8
hours) ?,...:
>
>> We have a client that always seems to stall until its backups finishes
(at
>> morning we see backup process at 99% and only one saveset for this
client
>> running) and we would like to know where the problem is: client, OS,
>> filesystem, etc. so any idea would be very appreciated.
>
>As several others have already pointed out, searching ("walking") a
>filesystem hierarchy with millions of files can take a very long time.
>The NetWorker client essentially spends most of its time wandering around
>looking for things to save, rather than actually saving data.
>
>Some filesystem types handle millions of files better than others, and
>there are some situations (hundreds of thousands of files in one
>directory) that can really exacerbate problems on certain file types.
>Directory layout and how those files are organized on the filesystem can
>make a big difference. Although we have plenty of RHEL here, we've never
>used GFS so I don't know enough about its complexities to offer any
>suggestions there. The shared nature of the filesystem and the additional
>complexities introduced because of that may play a big part in your issue.
>
>There may be things you can tune at the filesystem level on the client
>that would give you a very small speed improvement, but I don't believe it
>would be worth the time or effort.
>
>Your best bet for improving client backup times is to follow Rodney
>Rutherford's advice and investigate one of two common methods to
>get more parallelism from the client.
>
>- Don't use the "All" saveset for the client, instead dynamically generate
> the list of directories to back up, being certain that your script that
> generates the list splits the /home2 volume into multiple directory
> trees that can be backed up as separate concurrent savestreams.
>
>- Use two separate client resources, one that still uses "All" as the
> saveset BUT it uses a special directive that completely skips /home2
> (or skips just the parts of /home2 that you list in the second client
> resource), and a second client that does not use ALL but instead just
> lists a few directories under /home2. This has been referred to on the
> list in the past as the "split client" config.
>
>The problem with the "All" saveset is that although it finds all of the
>local filesystems it needs to back up dynamically (which is a really
>important feature when you don't control the client and administrators
>might add a new volume without telling you), it only sends one savestream
>per volume it finds.
>
>The two methods above artifically split a volume like /home2 into multiple
>savestreams, to increase the number of Networker processes that are
>searching the filesystem for things to save. It's a "divide and conquer"
>approach.
>
>Search the archives for more info on the two methods.
>
>Being able to still use just "All" while also somehow configuring
>NetWorker to automatically use multiple savestreams for a particular large
>volume would be a "killer feature" for EMC to add. I've been hoping to
>see it for years, and still no joy. There was hidden support for it in
>a few versions of NetWorker and how to do it became known to the list, but
>EMC/Legato strongly cautioned people against using it and it may have been
>pulled from the code base completely.
>
>Tim
>--
>Tim Mooney Tim.Mooney AT ndsu DOT
>edu
>Enterprise Computing & Infrastructure 701-231-1076
(Voice)
>Room 242-J6, IACC Building 701-231-8541 (Fax)
>North Dakota State University, Fargo, ND 58105-5164
>
>To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
>via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
>=========================================================================
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
