On Thu, Nov 19, 2009 at 12:42:04PM +0100, Christian Völker wrote:
> Hi,
> 
> [I know, you don't want to have this topic again, treat it as a collection]
> 
> has somebody tried to rsync a file system image?
> 
> As my BackupPC pool is too large for any backup programm (~600GB) to
> follow all the hardlinks, I'm looking (As everyone here) for a better
> solution.
> 
> Before I waste days on testing I want to ask how you perform you backups
> of the pool.
> 
> So my question is how rsync handles large files? Let's say I have my
> file system image of 600GB. I transfer it to the remote site by USB
> disk. Now I create a new image from the pool and perform an rsync on
> these files. Assuming 1GB has changed spread over the file- how many
> data will rsync transfer? Will it start transferring from the first
> mismatched bit? Or just transfer the changed data?

Our offsite backup is implemented as an rsync of an image of an LVM
snapshot of the backup filesystem. A script issues xfs_freeze on the
filesystem to bring the on-disk form in a consistent state, a snapshot
is taken, the snapshot is mounted using a script[1] as a directory of
images (part0001.img, part0002.img, ...), and this directory is rsynced
with an offsite location (the offsite server has 2 data directories, and
rsyncing is done alternatingly to each - that way we have a valid offsite
backup even if something happends during the transfer itself, which is
done with rsync --inplace).

Splitting it in several (500x 1GiB) files allows rsync to start faster
(since it won't go looking for data matching the first GiB in the
remote other GiBs).

The scripts logs how much data-difference LVM measures, how much changed
bytes rsync sees, and how many bytes are sent on the network. LVM measures
things differently (other block sizes, blocks that are changed during each of
the two previous runs are counted twice, ...), but it seems to produce numbers
comparable to the ones rsync gives.

Overall, this procedure works very well. One synchronisation run takes
between 4 and 7 hours, and transfers on average a few GiB worth of changed
backup data. (the offsite locations are connected using a 100Mbps university
network, but rsync is rate-limited to 10Mbps)

  [1] https://svn.ulyssis.org/repos/sipa/backuppc-fuse/devfiles.pl

kind regards,

-- 
Pieter

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/