Hi all,
First thank you Boris for your reply and sorry for not getting back to you
earlier.
I have reinstalled the server with Debian 7 and the problem still
persists. The rsync client on the servers being backed up is the stock
Centos 5 one, 3.0.6. Using the latest 3.1.1 from rsync.samba.org makes no
difference.
I am still able to backup smaller hosts with a high success rate, but a
couple of them (220GB/1M files and 600GB/10M files) fail with a similar
message than before:
rsync_bpc[1145]: segfault at 7fea2fe79da8 ip 0000000000447653 sp
00007fffd79115c0 error 4 in rsync_bpc[400000+73000]
I had a go at debugging this with gdb, but had difficulties getting it to
follow the rsync_bpc thread. Setting gdb to follow child on fork resulted
backuppc_dump to hang trying to ping the host. I might have another try
later on.
Looking at the logs from the weekend there is actually something
interesting. The full backup on a problematic host seems to fail between
1:50 and 2:30 after staring the backup, which lead me to check where the
backup is failing. Being a full instead of incremental as before, I traced
it to certain directory that has 3.6M files in it with a size of 230MB
(yes, that's right), so maybe this is an rsync/large dir issue after all
and not really a problem in rsync_bpc.
I will have to do some tests and see how it goes, in the mean time any
ideas will be highly regarded as well.
Cheers,
Lasse
On Thu, 19 Jun 2014, Dr. Boris Neubert wrote:
> Dear Lasse,
>
> Am 19.06.2014 06:08, schrieb Lasse R:
>> Having used BackupPC 3 for years I've recently been trialling BackupPC
>> 4.0.0alpha3 and seem to keep hitting an issue with rsync_bpc segfaulting
>> while backing up. BackupPC is running on a fresh up to date Centos 6.5
>> x86_64 machine with base install. BackupPC is installed using instructions
>> found in the readme. This is a fresh install and not an upgrade, starting
>> off with an empty pool and the last run before triggering the problem on
>> the machine managed to back up 5GB of data before the segfault.
> I run BackupPC 4.0.0alpha3 on Centos 6.5 (desktop mainboard, 4 GB RAM, 4
> TB hard disk) with XFS.
>> The segfault message is always similar. In dmesg I get:
>> rsync_bpc[23851]: segfault at 7ff491592aa8 ip 0000000000448203 sp
>> 00007fff12fa0790 error 4 in rsync_bpc[400000+74000]
>>
>>> From another machine:
>
>> rsync_bpc[3361]: segfault at 7fbbe909eaa8 ip 0000000000448203 sp
>> 00007fff01c71e00 error 4 in rsync_bpc[400000+74000]
> I checked the logs of the previous weeks and the only occurence I saw is
>
> Apr 27 00:44:53 rhun kernel: INFO: task rsync_bpc:12201 blocked for more
> than 120 seconds.
> Apr 27 00:44:53 rhun kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
> Apr 27 00:44:53 rhun kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 27 00:44:53 rhun kernel: rsync_bpc D 0000000000000000 0
> 12201 12044 0x00000080
> Apr 27 00:44:53 rhun kernel: ffff88010a4ddd28 0000000000000082
> ffff88010a4ddcf8 ffffffff8122dab1
> Apr 27 00:44:53 rhun kernel: ffff88010a4ddda8 ffffffffa03034cc
> ffff88010a4ddda8 0000000000000058
> Apr 27 00:44:53 rhun kernel: ffff880139d7e5f8 ffff88010a4ddfd8
> 000000000000fbc8 ffff880139d7e5f8
> Apr 27 00:44:53 rhun kernel: Call Trace:
> Apr 27 00:44:53 rhun kernel: [<ffffffff8122dab1>] ? avc_has_perm+0x71/0x90
> Apr 27 00:44:53 rhun kernel: [<ffffffffa03034cc>] ?
> xfs_iunlock+0x6c/0xd0 [xfs]
> Apr 27 00:44:53 rhun kernel: [<ffffffff8109b5be>] ?
> prepare_to_wait+0x4e/0x80
> etc.
>
>>
>> Possible hardware and disk issues have been eliminated by having to swap
>> servers and filesystems (not due to BackuPC). Both servers have had fresh
>> Centos 6.5 installed either with xfs or ext4 as the pool filesystem. I
>> might have to try another distro to see if this is Centos specific.
>
> So you already tried it the hard way to rule out hardware problems
> (rotten bits in RAM or the like).
>>
>> Anyway, I'm wondering if anyone else has ran into this and any ideas how
>> to further diagnose the problem?
> I wonder if the "error in the rsync protocol data stream" is the result
> or the cause of the segfault.
>
> Did you try to backup machines with other versions of rsync at the
> client side?
>
> Sorry that I cannot help you better.
>
> Best regards,
> Boris
>
>
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
