Bacula-users

Re: [Bacula-users] bacula-fd crashes on FreeBSD 9.2

2013-10-17 19:32:37
Subject: Re: [Bacula-users] bacula-fd crashes on FreeBSD 9.2
From: David Newman <dnewman AT networktest DOT com>
To: bacula-users AT lists.sourceforge DOT net
Date: Thu, 17 Oct 2013 16:29:22 -0700
On 10/17/13 5:33 AM, Martin Simmons wrote:
>>>>>> On Wed, 16 Oct 2013 12:13:26 -0700, David Newman said:
>>
>> On 10/14/13 2:44 AM, Martin Simmons wrote:
>>>>>>>> On Sun, 13 Oct 2013 18:25:07 -0700, David Newman said:
>>>>
>>>> On 10/9/13 4:41 PM, David Newman wrote:
>>>>> FreeBSD 9.2-RELEASE, bacula-client-5.2.12_3 installed from ports
>>>>>
>>>>> Ever since upgrading this host to FreeBSD 9.2, bacula-fd crashes as soon
>>>>> as bacula-dir starts a backup job. The entry in /var/log/messages is:
>>>>>
>>>>> Oct  9 16:25:50 o bacula-fd: Bacula interrupted by signal 0: UNKNOWN 
>>>>> SIGNAL
>>>>>
>>>>> Backups worked fine on this host running FreeBSD 9.1 and other hosts
>>>>> upgraded to FreeBSD 9.2 run backups OK.
>>>>>
>>>>> I've done the uninstall/reinstall thing with the bacula-client port, but
>>>>> that made no difference.
>>>>>
>>>>> Thanks in advance for troubleshooting clues.
>>>>>
>>>>> dn
>>>>
>>>> Is there a Wireshark decode for Bacula?
>>>>
>>>> I'm still stuck on this problem, and need more info on what's causing
>>>> that UNKNOWN SIGNAL error. Wireshark 1.8.6 just shows strings of bytes
>>>> for the Bacula stuff.
>>>>
>>>> Thanks.
>>>>
>>>> dn
>>>
>>> A wireshark decode won't help much here because problems like this must be 
>>> in
>>> the fd itself.
>>>
>>> Try attaching gdb to the bacula-fd process and see if it catches the
>>> mysterious signal (see
>>> http://www.bacula.org/5.2.x-manuals/en/problems/problems/What_Do_When_Bacula.html#SECTION00640000000000000000).
>>
>> No luck with this. Per that URL, I've put the btraceback.gdb file in the
>> same directory as the bacula-fd executable on the client (in this case,
>> /usr/local/sbin) and made the .gdb file executable.
>>
>> At run time it produces this error:
>>
>> /usr/local/sbin/btraceback.gdb:1: Error in sourced command file:
>> No symbol table is loaded.  Use the "file" command.
>>
>> That's problem 1. Problem 2 is that the syntax given for capturing
>> STDERR and STDOUT -- 2>\&1 -- doesn't work on either csh (root's default
>> on FreeBSD) or bash.
>>
>> Any ideas on remedying either issue?
> 
> It looks like you missed the part after the # in the URL -- you don't need the
> btraceback.gdb file.
> 
> The section I meant is called "Manually Running Bacula Under The Debugger" on
> that page (you'll have to adapt it for the bacula-fd).

Sorry for missing that.

The backup runs fine under the debugger, including the backup job
beforehand, but not with the FreeBSD startup script in /usr/local/etc/rc.d.

I've pasted below the debugger output and the startup script.

Thanks in advance for further troubleshooting clues.

dn


==========

Successful run, via /usr/local/sbin/bacula-fd run via gdb:

(gdb) thread apply all bt
Thread 5 (Thread 28c08b00 (LWP 100213/bacula-fd)):
#0  0x282302b3 in pthread_kill () from /lib/libthr.so.3
#1  0x2822f9b2 in pthread_kill () from /lib/libthr.so.3
#2  0x282328f9 in pthread_cond_signal () from /lib/libthr.so.3
#3  0x281f5d20 in bthread_cond_timedwait_p () from
/usr/local/lib/libbac.so.5
#4  0x281ef9b0 in watchdog_thread () from /usr/local/lib/libbac.so.5
#5  0x281f7167 in lmgr_thread_launcher () from /usr/local/lib/libbac.so.5
#6  0x28227f3a in pthread_getprio () from /lib/libthr.so.3
#7  0x00000000 in ?? ()

Thread 3 (Thread 28805e00 (LWP 100211/bacula-fd)):
#0  0x28624323 in nanosleep () from /lib/libc.so.7
#1  0x2822ad8b in nanosleep () from /lib/libthr.so.3
#2  0x281c1a90 in bmicrosleep () from /usr/local/lib/libbac.so.5
#3  0x281f7349 in check_deadlock () from /usr/local/lib/libbac.so.5
#4  0x28227f3a in pthread_getprio () from /lib/libthr.so.3
#5  0x00000000 in ?? ()

Thread 2 (Thread 28804300 (LWP 100133/bacula-fd)):
#0  0x28646103 in select () from /lib/libc.so.7
#1  0x2822a960 in select () from /lib/libthr.so.3
#2  0x281c45a8 in bnet_thread_server () from /usr/local/lib/libbac.so.5
#3  0x0804f5c6 in main ()
#0  0x282302b3 in pthread_kill () from /lib/libthr.so.3

==========

FreeBSD startup script:

#!/bin/sh
#
# $FreeBSD: sysutils/bacula-server/files/bacula-fd.in 323275 2013-07-19
09:44:58Z rm $
#
# PROVIDE: bacula_fd
# REQUIRE: DAEMON
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# bacula_fd_enable  (bool):  Set to NO by default.
#               Set it to YES to enable bacula_fd.
# bacula_fd_flags (params):  Set params used to start bacula_fd.
#

. /etc/rc.subr

name="bacula_fd"
rcvar=${name}_enable
command=/usr/local/sbin/bacula-fd

load_rc_config $name

: ${bacula_fd_enable="NO"}
: ${bacula_fd_flags=" -u root -g wheel -v -c
/usr/local/etc/bacula/bacula-fd.conf"}
: ${bacula_fd_pidfile="/var/run/bacula-fd.9102.pid"}

pidfile="${bacula_fd_pidfile}"

run_rc_command "$1"

==========




> 
> 
>>
>> Thanks.
>>
>> dn
>>
>>
>>
>>>
>>> If that doesn't catch it, then try the gdb command
>>>
>>> break signal_handler
>>>
>>> (signal_handler prints the "Bacula interrupted by signal" message).
>>>
>>> __Martin
> 
> 
> __Martin
> 
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users