[BackupPC-users] Correct rsync parameters for doing incremental transfers of large image-files
2012-05-11 17:04:58
Hello Backuppc-users,
I stuck while trying to identify the suitable rsync parameters to handle large
image file backups with backuppc.
Following scenario: I use partimage to do LVM-snapshot based full images of my
virtual (windows-) machines (KVM) blockdevices. I want to save theses images
from the virtualization server to my backup machine running backuppc. The
images are between 40 and 60 Gigs uncompressed each. The time-window for the
backup needs to stay outside the working hours and is not large enough to
transfer the images over the line every night. I red about rsync's capability
to only transfer the changed parts in the file by a clever checksum-algorithm
to minimize the network traffic. That's what I want.
I tested it by creating a initial backup of one image, created a new one with
only a few megs of changed data and triggered a new backup process. But I
noticed that the whole file was re-transfered. I waited till the end to get
sure about that and decided that it was not the ultimate idea to check this
with a compressed 18 GB image file but this was my real woking data image and I
expected it to work like expected. Searching for reasons for the complete
re-transmission I ended in a discussion-thread where they talked about rsync
backups of compressed large files. The explanation made sense to me: The
compression algorithm can cause a complete different archive file even if just
some megs of data at the beginning of the file hast been altered, because of
recursion and back-references.
So I decided to store my image uncompressed which is about 46 Gigs now. I found
out that I need to add the "-C" parameter to rsync, since data compression is
not enabled per default. Anyway: the whole file was re-created in the second
backup run instead of just transfering the changed parts, again.
My backuppc-option "RsyncClientCmd" is set to "$sshPath -C -q -x -l root $host
$rsyncPath $argList+" which is backup-pcs default disregarding the "-C".
Honestly, I don't understand the exact reason for this. There are some
possibilities that may be guilty:
-> partimage does not create a linear backup image file, even if it is
uncompressed
-> there is just another parameter for rsync I missed which enables
differential file-changes-transfers
-> rsync exames the file but decides to not use differential updates for this
one because of it's size or just because it's created-timestamp is not the same
as the prior one
Please give me a hint if you've successfully made differential backups of large
image files.
Thank you very much,
Andreas Piening
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
<Prev in Thread] |
Current Thread |
[Next in Thread>
|
- [BackupPC-users] Correct rsync parameters for doing incremental transfers of large image-files,
Andreas Piening <=
|
|
|