I would recommend you try this:
Set up several save streams to backup at the same time. IE instead of using
ALL as the backup, use \root, \usr, \data1, \data2, \data3, etc. as your
backups. If Tar can backup quickly, then your hardware is capable of
pulling lots of files in a short time, and the bottleneck is the legato
client building a database of filenames. By running parallel save streams,
you can have as many client processes running as you need to fully saturate
your hardware.
"Multitasking" is the answer here.....
Calvin Thomas
UNIX system admin
NACA Logistics
----- Original Message -----
From: "Stuart Lamble" <Stuart.Lamble AT ITS.MONASH.EDU DOT AU>
To: <NETWORKER AT LISTMAIL.TEMPLE DOT EDU>
Sent: Monday, September 09, 2002 12:05 AM
Subject: [Networker] Large filesystem backup revisited
> Since there seems to be a certain amount of interest on the list,
> here's a list of things we have (and have not :) tried to date. First,
> a quick rundown on the filesystem:
>
> * 522 GB hardware RAID partition.
> * Approximately 50 GB in 2,995,971 files (average of 16 kB/file)
> * Fibre channel 1 Gbps interface.
>
> General consensus was that we are being hit by the "many small files"
> problem, which certainly fits in with our situation.
>
> Things we've tried to date:
>
> * Increased the DNLC and inode cache (see
> http://www.princeton.edu/~unix/Solaris/troubleshoot/kerntune.html --
> very useful web page)
>
> Slight improvement from this, but Networker is still timing out.
>
> * Running uasm over the whole filesystem, piped to /dev/null to
> eliminate the network as a bottleneck -- I aborted this after
> eight hours (considering that we're "only" using 60 GB of space at
> peak, and less than 50 GB after the processes using the disk clean
> up after themselves, eight hours _should_ be adequate.)
>
> * Removing the temporary working area from the backup -- again, maybe
> a slight improvement, but it doesn't resolve the underlying issue.
>
> Executive summary: we've managed to get slight improvements, but Networker
> still likes to time out whilst backing up this filesystem. Sigh. Too many
> small files.
>
> At the moment, we're preparing to evaluate Sun's QFS as a possible
> alternative to UFS for this particular situation. Does anybody have any
> knowledge of how Networker interacts with QFS? Sun's representative is
> saying that our situation seems ideal for QFS, but then, I'd be inclined
> to take his words with a grain of salt :-) The biggest problem from my
> point of view is that Legato's compatibility matrix only lists UFS and
> VxFS; VxFS is an option, but it would be nice (if only from a bargaining
> point of view ;-) to have QFS as an option as well.
>
> As an aside, I'll be going on leave at the end of this week for a six
> week holiday, so the testing will be somewhat abbreviated (we have a
> thirty day evaluation license) ... I'll be picking up the ball once I get
> back, in late October.
>
> Ta muchly,
>
> Stuart.
>
> --
> Note: To sign off this list, send a "signoff" command via email
> to listserv AT listmail.temple DOT edu or visit the list's Web site at
> http://listmail.temple.edu/archives/networker.html where you can
> also view and post messages to the list.
> =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
>
--
Note: To sign off this list, send a "signoff" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
|
