Les Mikesell wrote:
> Jeff Siddall wrote:
>> Rob Owens wrote:
>>> Kevin DeGraaf wrote:
>>>>    2. Use the remote server to hold a copy of the on-site BackupPC 
>>>> server's file pool.  The pool would be rsync'ed on-site initially and 
>>>> then rsync'ed remotely from then on.
>>>>
>>> Against all advice on this list, I did this yesterday.  I rsync'd my
>>> pool to a larger drive in the same machine.  My pool size is 347G and
>>> it's been running for the past 15 hours or so.  I've transfered 334G so
>>> far.  My server has 3GB of RAM.  It hasn't crashed, like many people
>>> thought it might, and it has only used 2MB of swap.
>>>
>>> -Rob
>> I never asked for anyone's opinion before trying this myself, but I have
>> been [apparently] successfully rsyncing a copy of my BackupPC pool
>> remotely over a DSL line for months now.  I have yet to encounter
>> issues, other than a few rsync warnings when the pool changes during the
>> rsync -- which does take a few hours at DSL's 800 kbps uplink.  The pool
>> is about 120 GB, and changes by about 1 GB each day.
>>
>> I did my initial "remote" disk setup a bit differently.  Instead of
>> rsyncing the pool, I populated my "remote" drive by mounting it locally
>> on the BackupPC server and doing a cp -a.  No issues with that of
>> course.  Then I took the drive out of the BackupPC server and stuck it
>> in the remote machine.  From that point on rsync with the -H option has
>> been keeping the pools in sync.
>>
>> Can anyone explain why this is a bad idea?
> 
> Rsync tends to have a hard time with a backuppc archive because of the 
> sheer number of directory/file entries and also the number of hardlinks 
> which need a brute-force lookup to find the matching entries.  Are you 
> using rsync 3.x?  It may be that the recent changes have significantly 
> improved its ability to handle this scenario.

Nope, I'm using rsync-2.6.9-5.fc8 on relatively slow hardware.  The PC
is a dual core Athlon 2.1 GHz with 4 GB RAM, and the pool filesystem is
on an external SATA drive running ext3.  As this job runs in the middle
of the night I have no idea how much RAM it is gobbling.  Maybe I should
look into that, although the file list size of 150 MB or so wouldn't
indicate I should be running short of RAM.

All that being said my pool is relatively small at 5.4 million files and
645 GB (as reported by rsync -- counting all copies of linked files I
assume).  I suspect (or perhaps hope) that rsync 3 may solve the known
problems before my pool size becomes an issue.

Here's a dump from the last log:

Backup started Sun Oct  5 23:03:01 EDT 2008
Number of files: 5350429
Number of files transferred: 1691
Total file size: 692701780215 bytes
Total transferred file size: 813298248 bytes
Literal data: 806422416 bytes
Matched data: 6876306 bytes
File list size: 157784599
File list generation time: 3547.271 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1513518
Total bytes received: 965761770

sent 1513518 bytes  received 965761770 bytes  53177.67 bytes/sec
total size is 692701780215  speedup is 716.14
Backup finished Mon Oct  6 04:13:33 EDT 2008

It took about 5 hours to sync the 120 GB or so, one hour of which was
building the file list.  Maybe more files (possibly more data) would
break things but so far it seems to be running fine.

Jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/