BackupPC-users

Re: [BackupPC-users] Copying in a file instead of backing up?

2009-01-13 22:23:47
Subject: Re: [BackupPC-users] Copying in a file instead of backing up?
From: Adam Goryachev <mailinglists AT websitemanagers.com DOT au>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Wed, 14 Jan 2009 14:20:40 +1100
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeffrey J. Kosowsky wrote:
> Christian Völker wrote at about 22:13:24 +0100 on Tuesday, January 13,
2009:
>  > But now there is a 4GB file on the remote box- and the broadband
>  > connection is dropped every 24h hours (f!ck Telekom), so any running
>  > backup process will fail.
>  >
>  > Whatever, because I have this file locally as well (same file, md5sum
>  > and times are the same!) can I "insert" tthis file somehow into my
>  > backupPC under /var/lib so it recognizes it as "already there"and does
>  > not try to backup all the time?
>  >
>  > Thanks for further ideas.
>  >
> The following "kludge" should work.
>
> 1. Create a new temporary backuppc host that points to your local
>    machine. Set up the corresponding 'share' so it only backs up the
>    4GB file you have locally. This will give you a valid (c)pool
>    backup of that file.
>
> 2. Then run backuppc on your remote machine and it will create a
>    hard-link to the now already existing pool file.
>
> 3. Delete the temporary host config and the corresponding tree under
>    TopDir.
>
> Alternatively, you could cobble together a routine for directly
> injecting a file into the pool or cpool (by calculating the md5sum
> name and compressing as appropriate). Be sure to either add an extra
> temporary link to it or keep BackupPC_nightly from running until you
> do a real backup of the remote machine or otherwise the pool entry
> will be dropped.
>  

Even if the file exists in the (c)pool, it will still be transferred
by rsync/backuppc during the backup, unless it existed in the previous
backup of the same share/host.

One thing I found when backing up (or copying) very large files with
rsync is that copying the first portion of the file, then stopping the
transfer, preserving the portion transferred (--partial), and then
re-starting the transfer can have a huge impact on the amount of data
rsync will transfer.

This can easily seen like this:
1 Create a file of 1000M with the letter 'a' on a remote machine
2 run rsync --partial --progress to sync the remote file to the local file
3 After noting the throughput/etc (wait for a few MB), hit CTRL-C, and
then restart the transfer.
4 Notice the transfer is much quicker now

I'd suggest rename the remote 4G file, copy the first 50M (dd
if=filerename of=fileorig bs=1k count=10k), exclude the renamed copy
of the full file, run a full backup, then rename the full file back,
re-run another full backup.

If it doesn't help, then you may just need to incrementally increase
the size of the file so that each chunk can be added to the end of the
file in a single run....

Also check you have enabled ssh compression (-C) to improve bandwidth
usage...

Hope that helps,

Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJbVoIGyoxogrTyiURAiacAKCC4WI0JWHb5rV/qRHrerW1m6o10QCfRKYi
GvvxfwVIl5M7VVXomSJBjBo=
=KDyx
-----END PGP SIGNATURE-----


------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/