Hi list,

on the topic of importing files from a local source to speed up the first
remote backup (eg. by sending a HDD with the current contents),

I wrote on 2009-04-09 01:19:17 +0200 [Re: [BackupPC-users] How do the files 
look like when transferred?]:
> [...]
> there should be a really
> simple way to do this right now without much coding. It's untested, but I will
> be needing it myself sometime soon, so I might as well write it down.
> 
> 1. Let's suppose you've got a host named 'remote', of which you want to back
>    up '/some/path/to/share' via 'rsync' (it really shouldn't matter what
>    share name and transfer method you're going to use).

... but in the context of "speeding up the initial transfer" only rsync/rsyncd
make sense.

>    So you'd have
>    remote.pl containing something like:
> 
>    $Conf {XferMethod} = 'rsync';
>    $Conf {RsyncShareName} = [ '/some/path/to/share' ];
>    $Conf {RsyncClientCmd} = '$sshPath -q -x $host /usr/bin/sudo $rsyncPath+ 
> $argList+';
>    $Conf {BackupFilesExclude} = {
>      '/some/path/to/share' => [ '/.Trash', '/.thumbnails' ],
>    };
> 
>    Yes, "$rsyncPath+" is being pedantic ;-).
> 
> 2. You've got a local copy on '/mnt/cdrom/import' on the BackupPC server,
>    which you want to import (that means remote:/some/path/to/share/dir/file
>    appears at /mnt/cdrom/import/dir/file). Let's try adding this to remote.pl:
> 
>    $Conf {XferMethod} = 'tar';
>    $Conf {TarShareName} = [ '/some/path/to/share' ];
>    $Conf {TarClientCmd} = '/usr/bin/sudo $tarPath -c -v -f - -C 
> /mnt/cdrom/import --totals';
>    $Conf {TarFullArgs} = '$fileList';
>    $Conf {PingCmd} = '&{sub {0}}';
>    $Conf {BackupFilesExclude} = {
>      '/some/path/to/share' => [ '/.Trash', '/.thumbnails' ],
>    };
> 
>    Note that TarClientCmd makes use of neither $host nor $shareName - nobody
>    ever said it had to. Note also that BackupFilesExclude is identical.

(Meaning you can omit it, unless you actually want to change it for the import
for some reason.)

>    So you
>    basically just add the tar configuration and the PingCmd and temporarily
>    change the XferMethod to tar. Then run the (full!) backup. Then change
>    XferMethod back, remove the PingCmd, and all should be set for remote
>    backups.

I should add that the first *remote* backup should also be a *full* backup!

Unfortunately, this does *not* work without further adjustments - at least with
BackupPC 2.1.2/File::RsyncP 0.64, but it doesn't seem to have changed with
BackupPC 3.2.0beta0/File::RsyncP 0.68, as far as I can see from Changelog and
source.
The details: rsync backups and tar backups both store attrib files in a common
format - with one exception (well, two that I know of, but only one is
important here). In the "mode" field, rsync has a (redundant) bit named
S_IFREG to explicitly indicate a regular file (i.e. not directory, block or
character device, or fifo). The tar transport (i.e. BackupPC_tarExtract) does
not set this bit for "regular files". When File::RsyncP later attempts to
transfer a file and detects that the local file (from the tar import) is
apparently not a regular file (no S_IFREG bit set) but the remote one is,
it transfers the whole remote file, making the import pointless.

The consequence of this is that whenever you change the XferMethod from tar to
rsync, all regular files (meaning all data!) will be re-transferred.

I'm not sure whether that qualifies as a bug or a nuisance, but I'll try to
work out a patch just the same.

My way around this for my local import was to write a few lines of Perl which
currently don't even qualify as "quick hack", which change all attrib files of
the tar backup to contain the S_IFREG bit for all files which I think should
have it set. It worked for me :), but my backup didn't include anything fancy
(like devices or fifos). The alternative would have been to use rsync for the
local import. The problem with that is that it's not as easy to "fool" rsync
into using a different source directory to what BackupPC would like it to
(you'd need to modify $fileList, which is possible with a script, but I'd have
had to wait several hours for a new local backup to complete ...).

So, if anyone is interested, I'd clean up my attrib file patching (or, better
yet, he could test a patch to BackupPC_tarExtract).

Regards,
Holger

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations 
Conference from O'Reilly Media. Velocity features a full day of 
expert-led, hands-on workshops and two days of sessions from industry 
leaders in dedicated Performance & Operations tracks. Use code vel09scf 
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/