I've only looked at the code briefly, but I believe this *should* be
possible.  I don't know if I'll be implementing it, at least not
right away, but it shouldn't actually be that hard, so I wanted to
throw it out so someone else could run with it if ey wants.

It's an idea I had about rsync resumption:

Keep an array of all the things you haven't backed up yet, starting
with the inital arguments; let's say we're transferring "/a" and
"/b" from the remote machine.

Start by putting "a/" and "b/" in the array.  Then get the directory
listing for a/, and replace "a/" in the array with "a/d", "a/e", ...
for all files and directories in there.  When each file is
transferred, it gets removed.  Directories are replaced with their
contents.

If the transfer breaks, you can resume with that list of
things-what-still-need-transferring/recursing-through without having
to walk the parts of the tree you've already walked.

This should solve the SIGPIPE problem.  In fact, it could even deal
with incrementals from things like laptops: if you have settings for
NumRetries and RetryDelay, you could, say, retry every 60 seconds
for a week if you wanted.

On top of that, you could use the same retry system to
*significantly* limit the memory usage: stop rsyncing every N files
(where N is a config value).  If you only do, say, 1000 files at a
time, the memory usage will be very low indeed.

As I said, none of this should be *especially* hard to implement;
it's just changes to lib/BackupPC/Xfer/RsyncFileIO.pm and
lib/BackupPC/Xfer/Rsync.pm , and doing this would make BackupPC be
as robust as a lot of commercial backup packages, on top of all its
current benefits.

(Note the "-d" option to rsync; you'd want to pass that to the
remote rsync to get the directory listings, I think.)

-Robin


-- 
They say:  "The first AIs will be built by the military as weapons."
And I'm  thinking:  "Does it even occur to you to try for something
other  than  the default  outcome?"  See http://shrunklink.com/cdiz
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/