Re: [BackupPC-users] Backing up many, many files over a medium speed lin

with 370,000 files rsync should use 370,000*100B=35MB+/- 10% on each side. How fast is your CPU? are you sure that you can process the checksums fast enough?

Are you compressing the rsync process and if so what compression level? rsync compression at level 3 is only slightly worse that level 9 but uses a tiny fraction of the CPU time that level 9 does.

Also, are you firing off both backups at the same time. depending on your hardware, you could be adding very very large I/O delays from disk seeks. I/O performance is a big factor in system load. slow IO can give you a 2 digit system load and have plenty or ram and CPU left over.

I would definitely try to break down the shares a bit. You might also consider not doing a checksum on older files that are in storage but doing checksums on newer files that may be altered. Most systems will always update the mtime of a file when it is altered and unless you have something special going on then the mtime is a reliable and easy check. even if the file was not actually changed but was opened and re-saved as is the mtime will change. (again, there are some special circumstances)

I think the most likely cause here is the checksumming eating up RAM as more and more files/larger files are checksummed in parallel and spilling over to SWAP. Then SWAP is adding I/O transactions to the system which in the best case is adding work to the CPU and disk controller and worse case is that the SWAP is on the same physical drive as TOP_DIR. It them becomes a traffic jam where every extra file takes more ram theirfore more swap which slows down the existing checksum processes because they have to wait to write which keeps more in the pipe which causes more memory to be used=more swap=more io=x=x=x==x=x=x=and so on.

try just trusting mtime on the files and see what happens. 370,000 isnt that much for backuppc or rsync. I dont see issues until around 1million files. I then break that backup set down to seperate machines.

On Tue, Nov 24, 2009 at 8:42 PM, GB <pseudo AT gmail DOT com> wrote:

Thanks Chris. I will give it a shot and see if I can make it behave in any way... was hoping for a bit of a magic bullet, I suppose :)

On Tue, Nov 24, 2009 at 10:41 PM, Chris Bennett <chris AT ceegeebee DOT com> wrote:

Hi,

> Thanks for the reply. The data is, in fact, "all time" in the sense that it
> goes back years, but it's sorted by filename, rather than date; it's
> essentially equivalent to how BackupPC stores data in cpool/, i.e. the first
> 3 characters of the filename will generate 3 levels of subdirectories. The
> best I was able to do, to date, was to make 10 shares, 1-9, and back up 10
> separate backup trees. But that was before, when I had about 100k files... I
> tried this recently, and seem to have made it go under. So I guess I'd need
> to make TWO levels of shares, so 1/0-1/9, 2/0-2/9, etc. Then, maybe, once I
> go through the full loop, it'll be easier to perform future incrementals
> since the delta will be small.

Yeah, I've been able to archive large pools of files that have aged,
so that backuppc doesn't have to consider such a large filelist. I'm
not too sure on the mechanics of backuppc and overhead - e.g. what
amount of work does backuppc perform to perform a full and
incremental.. how much memory is consumed per considered file. I
expect someone else can more succintly answer these kind of questions
to help you build a more scalable configuration.

> My BackupPC box doesn't swap too much, it doesn't behave like it's under
> massive load at all; but then again, I think my IO subsystem (Dell Perc6 +
> 4x WD Greens in RAID5) hopefully outperforms the speed of the link+any
> overhead :) I haven't tried stracing rsync on the remote server. Any
> suggestions on how to use it? I've never tried it before.

Get the pid of your rsync process on the source of data.

Then perform something like
# -s3000 specified 3000 characters printed per system call
strace -p <pid> -s3000

This will give you insight into the open/stat/read/close cycle that
rsync will be doing when copying data. I would expect it to be
cycling faster than you can read, although in the case where I've seen
high swap activity, you'll see batches of the cycle followed by
pauses.

Similarly, running:
vmstat 1

in another console and looking at the bi/bo columsn that represent
blocks in/out helps you to know whether swap is being heavily used.

Good luck and let me know if you find a good solution to your problem.

Regards,

Chris Bennett
cgb

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Backing up many, many files over a medium speed link - kills CPU and fails halfway. Any help?