BackupPC-users

Re: [BackupPC-users] Don't use rsync -H!! (was Re: copying the pool to a new filesystem?!)

2010-12-07 03:39:44
Subject: Re: [BackupPC-users] Don't use rsync -H!! (was Re: copying the pool to a new filesystem?!)
From: Robin Lee Powell <rlpowell AT digitalkingdom DOT org>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Tue, 7 Dec 2010 00:38:00 -0800
On Tue, Dec 07, 2010 at 07:58:05AM +0000, Tyler J. Wagner wrote:
> On Mon, 2010-12-06 at 10:51 -0800, Robin Lee Powell wrote:
> > IME it's *much* faster that way; you do BackupPC_TarPCCopy, and
> > then rsync the cpoll *without -H*.  It shouldn't take any longer
> > than the actual data transfer time itself.
> 
> So, if you wanted to copy the entire /var/lib/backuppc directory
> to another filesystem, what commands would you use, for example?

sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarPCCopy 
/var/lib/backuppc/pc | ssh $host 'cat >/tmp/pc.tar'
rsync -aPv /var/lib/backuppc/cpool $host:/var/lib/backuppc/
ssh $host 'tar -C /var/lib/backuppc/ -xvf /tmp/pc.tar'

I have a script for this, in fact, for transferring single
/var/lib/backuppc/pc/[foo]/ directories between hosts.  Going to
post it when it's a bit more tested.

> Why use BackupPC_TarPCCopy over just tar?

The backuppc pool is a bunch of files.

The pc directories are a bunch of *hard links* into the polo files.

There is no way to find out all the hard links to the same inode
except to check *each and every* file on the file system.

So if you use "rsync -H" or "tar -l", they have to generate a giant
table of the inode values of each and every file in both the
backuppc pool *and* the /var/lib/backuppc/pc/ dirs.  On a decently
sized backup system this takes a long time, and a *LOT* of RAM.

BackupPC_tarPCCopy, OTOH, knows how backuppc's checksums are
generated.  So it just walks all the /var/lib/backuppc/pc/ files,
generates a checksum, checks that that checksum is in the pool as
expected, and emits a tar hardlink line.  You end up with a tar
that's just attrib files, hard link instructions, and logs, and you
end up with it *MUCH* faster than checking all possible inodes for a
match.

-Robin

-- 
http://singinst.org/ :  Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei".   My personal page: http://www.digitalkingdom.org/rlp/

------------------------------------------------------------------------------
What happens now with your Lotus Notes apps - do you make another costly 
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus 
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>