BackupPC-users

[BackupPC-users] Uneffective rsync transfers?

2014-06-26 08:42:10
Subject: [BackupPC-users] Uneffective rsync transfers?
From: František Starý <frantisek AT stary DOT net>
To: backuppc-users AT lists.sourceforge DOT net
Date: Thu, 26 Jun 2014 14:40:33 +0200
Hello.

I've started experimenting with rsync xfer method and I've found that the rsync transfers always the changes against the last backup.
There is a directory "testdir" with 3 files.

First full backup:
~~~~~~~~~~~~~~~~~~
full backup started for directory /testdir/
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . /testdir/
Xfer PIDs are now 27260
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 27260,27468
  create d 755       0/0        4096 .
  create   644       0/0     1048576 testfile1
  create   644       0/0     1048576 testfile2
  create   644       0/0     1048576 testfile3
Done: 3 files, 3145728 bytes

It's what I've expected. First backup, everything is transferred.
Try to do next backup without any changes.

Second full backup (when 0 files changed):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
full backup started for directory /testdir/ (baseline backup #0)
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . /testdir/
Xfer PIDs are now 27502
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 27502,27551
  create d 755       0/0        4096 .
Done: 0 files, 0 bytes
Got fatal error during xfer (No files dumped for share /testdir/)
Backup aborted (No files dumped for share /testdir/)
Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)

Shouldn't be this considered as "non error" status? It means no files were changed. I think it is an correctly finished full backup with no files changed, but backuppc evaluates it as "Got fatal error during xfer".
Next try - I've added one new file, other files remain unchanged.

Second full backup (when 1 file added):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
full backup started for directory /testdir/ (baseline backup #0)
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . /testdir/
Xfer PIDs are now 28931
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 28931,28980
  create d 755       0/0        4096 .
  create   644       0/0     1048576 testfile4
Done: 1 files, 1048576 bytes.

It seems to be okay. Only the new file was transferred. But when I look in the web gui at this full backup, I see only this file. The other files (testfile1,2,3) are not here. Shouldn't be the full backup really full (merged somehow with the previous full backup)? After te previous full backup will be deleted, this backup will be almos useless..
I've continued testing. No changes in the testdir, run another full backup.

Third full backup (when 0 files changed after the previous backup):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
full backup started for directory /testdir/ (baseline backup #1)
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . /testdir/
Xfer PIDs are now 29029
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 29029,29078
  create d 755       0/0        4096 .
  create   644       0/0     1048576 testfile1
  create   644       0/0     1048576 testfile2
  create   644       0/0     1048576 testfile3
Done: 3 files, 3145728 bytes

Again. Only the files that are not present in the previous backup are present.
Next try. Next full backup, no files changed..

Fourth full backup (when 0 files changed after the previous backup):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
full backup started for directory /testdir/ (baseline backup #2)
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . /testdir/
Xfer PIDs are now 30426
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 30426,30474
  create d 755       0/0        4096 .
  create   644       0/0     1048576 testfile4
Done: 1 files, 1048576 bytes

And again. Only the files that are not present in the previous backup are present.

Is this a normal behavior? Is there a way to configure backuppc to transfer only changed files and merge them with the previous backup? I want to backup 200GB of email files. There will be only small amount of changes. If it will work as described, it transfers everything, then only the changes, then everything except the last changes and so on..

My testing config file:
~~~~~~~~~~~~~~~~~~~~~~~
$Conf{XferMethod} = 'rsync';
$Conf{ClientNameAlias} = 'localhost';
$Conf{RsyncShareName} = ['/testdir/'];

$Conf{BackupsDisable} = 1;
$Conf{FullKeepCnt} = 10;
$Conf{IncrKeepCnt} = 5;

Thank you.
With regards,
Frantisek Stary.

------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/