Re: Speed up 400GB backup?

On Tue, 2004-07-20 at 16:05, Frank Smith wrote:

--On Tuesday, July 20, 2004 14:35:53 -0700 Kris Vassallo <kris AT linuxcertified DOT com> wrote:
>> NOTES:
> 
>>   driver: WARNING: /tmp: not 102400 KB free.

I overlooked this last night.  I've never seen this message myself,
but perhaps it is relevant.  Any thoughts, anyone?

> 
> I am using tar to do this. The bda1 system is a CVS server which gets
> hammered on all day long and does have tons of smaller files as well as
> a decent amount of larger ones.

As Stefan mentioned, there are probably subdirectories you could exclude
from the backup to speed things up.  You mentioned part of it was used
for CVS, perhaps you can exclude some of the build trees and just backup
the source trees.

Scrap the CVS part of it, the CVS repository is on a completely different client and that client isn't taking all that long to backup. However, point well taken, I'm sure that there is plenty of junk that does not need to be backed up. So fixing this would reduce the backup time as a whole but would still leave me with outrageous estimate times on the other server.


> The disks in the venus box are all SATA 150 drives, SCSI is way out of
> the price range for this amount of space. If venus is the machine that
> is taking forever to do the estimates, is it possible that 1. estimates
> start on all machines, 2. the estimates finish on the smaller remote
> file systems first; these systems begin to dump. 3. now along with the
> backup server trying to do an estimate on its own disks, its also
> dealing with a dump coming in from remote systems and all of this
> together is slowing it down? Do I have any valid ideas here?

Possible, although the estimates write to /dev/null, so the remote
dumps shouldn't be slowing them down unless it's your controller
limiting you and not the disks themselves.  You could try commenting
out all the other filesystems in your disklist and see if the estimate
still takes as long.

Will do!

   Is the system otherwise idle when you are running Amanda?  If
the disks are fairly active (whether from user activity or perhaps
automated nightly builds) it will slow down your backups considerably.

There are nightly builds, however I have set the backup to run during a time at which disk access is minimal.

   It could also be kernel related.  Our first attempt at Linux
fileservers had problems under heavy load, the sytem would slow to
a crawl (and sometimes appear to hang) under concurrent loads (a
CVS build and an rsync of the filesystem in our case).  Moving from
a 2.4 kernel to 2.6 solved the problem completely.

Well, if I can't get this going with anything else, then I am going to have to try 2.6. That will be my last resort (along with migrating to a new file system) as there is going to be an insane amount of work involved. I can't just plop in the 2.6 kernel without breaking the module utils and all sorts of other things. This probably means building a similar piece of hardware up and then building the kernel on that to make sure it boots and then replacing the system disk when its ready. That along with maintaining minimal down time and getting screamed at by impatient developers... I can already predict a week without sleep and the headache coming on.. oie!


Frank

> -Kris
> 
>