Hello.
27.6.2014 2:34, Holger Parplies wrote:
> Hi,
>
> Les Mikesell wrote on 2014-06-26 10:47:19 -0500 [Re: [BackupPC-users]
> Uneffective rsync transfers?]:
>> On Thu, Jun 26, 2014 at 7:40 AM, Frantisek Stary <frantisek AT stary DOT
>> net> wrote:
>>> I've started experimenting with rsync xfer method and I've found that the
>>> rsync transfers always the changes against the last backup.
>> You don't need to experiment to find that - see $Conf{IncrLevels} in
>> the documentation.
> and it's not "the last backup", it's "the reference backup". Of course, if
> you're only doing full backups, that's the same thing.
I use only full backups to show the issue as clear as possible.
>
>>> Second full backup (when 0 files changed):
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> full backup started for directory /testdir/ (baseline backup #0)
>>> [...]
>>> Got fatal error during xfer (No files dumped for share /testdir/)
>>> Backup aborted (No files dumped for share /testdir/)
>>> Not saving this as a partial backup since it has fewer files than the prior
>>> one (got 0 and 0 files versus 0)
>>>
>>> Shouldn't be this considered as "non error" status? It means no files were
>>> changed.
>> That's optional, See $Conf{BackupZeroFilesIsFatal}
> No, I believe the objection is correct. BackupZeroFilesIsFatal is for a share
> with 0 files, not 0 *changed* files. If there's something there, but it
> hasn't changed since the last backup, that is not an error. Apparently, rsync
> (or rather the rsync XferMethod) is complaining about not having any files in
> the backup - possibly due to failing hard linking.
The log file contains:
2014-06-22 11:58:22 full backup started for directory /testdir/
(baseline backup #0)
2014-06-22 11:58:24 Got fatal error during xfer (No files dumped for
share /testdir/)
2014-06-22 11:58:29 Backup aborted (No files dumped for share /testdir/)
It complains about "No file dumped for share" only when no files changed
or a file is deleted (I've tested it more times). I test it via
localhost (testhost is the same machine as the backuppc server), no
network errors or similar issues.
>>> Second full backup (when 1 file added):
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> [...]
>>> Done: 1 files, 1048576 bytes.
>>>
>>> It seems to be okay. Only the new file was transferred. But when I look in
>>> the web gui at this full backup, I see only this file. The other files
>>> (testfile1,2,3) are not here. Shouldn't be the full backup really full
>>> (merged somehow with the previous full backup)?
>> Full
> rsync
>
>> backups should link unchanged files from the previous full into
>> the current tree.
> Other XferMethods will "coincidentally" create a link to the same pool file
> because the content matches (hard link count permitting). They are, however,
> unaware that the file is unchanged. Only rsync creates "same" log entries -
> other XferMethods only have "pool". You should check your XferLogs for this.
> It would seem they might not contain "same" entries, although they really
> should.
Here is the xfer log:
Contents of file /var/lib/backuppc/pc/testhost/XferLOG.1.z, modified
2014-06-22 12:01:33
full backup started for directory /testdir/ (baseline backup #0)
Running: /usr/bin/ssh -q -x -l root localhost /usr/bin/rsync --server --sender
--numeric-ids --perms --owner --group -D --links --hard-links --times
--block-size=2048 --recursive . /testdir/
Xfer PIDs are now 28931
Got remote protocol 30
Negotiated protocol version 28
Xfer PIDs are now 28931,28980
create d 755 0/0 4096 .
create 644 0/0 1048576 testfile4
Done: 1 files, 1048576 bytes
It shows only one file was transferred. I think it is correct, because
only one file was modyfied. But how (when) should be the rest of the
files merged to this one?
When I create another test host, create full backup, delete one file and
do another backup, there is "No file dumped for share" error too:
2014-06-27 07:09:08 full backup started for directory /testdir2/
2014-06-27 07:09:09 full backup 0 complete, 4 files, 4194304 bytes, 0 xferErrs
(0 bad files, 0 bad shares, 0 other)
{deleted one file}
2014-06-27 07:09:28 full backup started for directory /testdir2/ (baseline
backup #0)
2014-06-27 07:09:29 Got fatal error during xfer (No files dumped for share
/testdir2/)
2014-06-27 07:09:34 Backup aborted (No files dumped for share /testdir2/)
Why? It seems to me that rsync xfer method does the comparsion against
the last reference backup and it logically cannot delete from there, so
it does only "appendable" rsync into the new backup directory. And there
is no new or changed files. So rsync full backup works in this aspect
similar as the incremental backup with tar xfer method (not covering the
deleted files)?
Shouldnt be better to create a "hardlink copy" of the reference backup
into a new backup directory at the start of the rsync full backup and
then use rsync method to update this new backup directory? Or are the
hardlinks created afterwards? If so, how backuppc knows which files were
deleted (in the last example xfermethod rsync did NOT noticed the
deleted file) and which files should be merged?
I just want to understand how it *should* work to be able to debug my
issues..
>> Is your xfer log full of 'can't link' errors?
> Since the quote was from the xfer log, apparently it isn't. And we're talking
> about a different piece of code than usual - it would seem to be the
> XferMethod that is failing to create the hard links, not BackupPC_link (though
> it's hard to imagine *how* it would fail in such a way).
There are no "can't link" errors at all. I use this backuppc server for
another 10 hosts with xfermethod tar without any issues.
>
>>> Fourth full backup (when 0 files changed after the previous backup):
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> [...]
>>> And again. Only the files that are not present in the previous backup are
>>> present.
>>>
>>> Is this a normal behavior?
>> No, you have some problem with the filesystem. Does it support hard links?
> In particular, does Perl link() report success although the file system is not
> creating hard links?
>
> Honestly, if there were a bug in BackupPC that would re-transfer unchanged
> data every second backup, someone would have noticed ;-). There are people
> here doing large backups over slow links who depend on the bandwidth savings
> from rsync transfer.
>
> The analysis that hard links may be failing fits what you are seeing very
> well, though from the code I can't see how BackupPC
> a) would not notice,
> b) still complain about a zero file backup.
> Linking would need to *appear* to be working for BackupPC to even start.
>
> Of course, it's hard to analyse the code when you don't know which code to
> look at. Which version of BackupPC are you using on which distribution (or
> rather: where did you get it from)? What type of file system is your pool on,
> and how do you access it?
>
> A different possibility would be that the code for building a view of the last
> backup is not working correctly for you, though I can't even begin to imagine
> why it wouldn't.
I'm sure hardlinks for my testhost are created successfully:
root@morgan:~# ls -l /var/lib/backuppc/pc/testhost/0/f%2ftestdir%2f/
total 28
-rw-r----- 4 backuppc backuppc 48 Jun 22 11:56 attrib
-rw-r----- 36 backuppc backuppc 4596 Jun 22 11:56 ftestfile1
-rw-r----- 36 backuppc backuppc 4596 Jun 22 11:56 ftestfile2
-rw-r----- 36 backuppc backuppc 4596 Jun 22 11:56 ftestfile3
Info about my server:
root@morgan:~# uname -a
Linux morgan.selfip.net 3.8.0-19-generic #30-Ubuntu SMP Wed May 1
16:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root@morgan:~# cat /etc/issue
Ubuntu 13.10 \n \l
root@morgan:~# dpkg -la|grep backuppc
ii backuppc 3.2.1-4ubuntu1
amd64 high-performance, enterprise-grade system for backing up PCs
root@morgan:~# mount|grep /var/lib/backuppc
/dev/loop7 on /var/lib/backuppc type ext4 (rw)
root@morgan:~# losetup /dev/loop7
/dev/loop7: [0005]:1874 (/dev/vgdata/lvbackuppc)
encryption=CryptoAPI/aes-cbc
I believe that rsync xfermethod work for other people :-). I was unable
to find any forum with similar thread over the internet. And I'm out of
ideas how to debug it.
>>> Is there a way to configure backuppc to transfer
>>> only changed files and merge them with the previous backup? I want to backup
>>> 200GB of email files.
>> Yes, that is the way it is supposed to work. If the files are mbox
>> format (many messages in one file) it can be rather slow to merge the
>> changes but it does work. Maildir format works more efficiently.
> The important thing to note here is that with large mbox files, you'll have an
> individual copy after every change and not much savings from BackupPC pooling.
> Maildir on the other hand fits in with BackupPC pooling perfectly.
I use maildirs for which the xfermethod rsync should be realy usefull
for me.
Thank you for your time and for any ideas.
Regards,
Frantisek.
>
> Regards,
> Holger
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users AT lists.sourceforge DOT net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
