[Bacula-users] Storage Daemon crash backtrace

My SD has been crashing every night and I finally got a backtrace. I don't know what all this means, but I could sure use some help to figure out why it keeps crashing and how to stop it so that I can get some back-ups done.

Thanks,

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Starting program: /usr/sbin/bacula-sd -s -f -c /etc/bacula/bacula-sd.conf

[Thread debugging using libthread_db enabled]

[New Thread 0x7ffff509c710 (LWP 27308)]

[New Thread 0x7ffff489b710 (LWP 27309)]

[Thread 0x7ffff509c710 (LWP 27308) exited]

[New Thread 0x7ffff509c710 (LWP 27412)]

[New Thread 0x7ffff409a710 (LWP 27413)]

[New Thread 0x7ffff368d710 (LWP 27414)]

[New Thread 0x7ffff2e8c710 (LWP 27415)]

[Thread 0x7ffff2e8c710 (LWP 27415) exited]

[Thread 0x7ffff368d710 (LWP 27414) exited]

[New Thread 0x7ffff368d710 (LWP 27416)]

[New Thread 0x7ffff2e8c710 (LWP 27417)]

[New Thread 0x7ffff268b710 (LWP 27418)]

[New Thread 0x7ffff1e8a710 (LWP 27423)]

[New Thread 0x7ffff1689710 (LWP 27424)]

[New Thread 0x7ffff0e88710 (LWP 27425)]

[New Thread 0x7fffebfff710 (LWP 27426)]

[New Thread 0x7fffeb7fe710 (LWP 27427)]

[New Thread 0x7fffeaffd710 (LWP 27428)]

[New Thread 0x7fffea7fc710 (LWP 27429)]

[New Thread 0x7fffe9ffb710 (LWP 27430)]

[New Thread 0x7fffe97fa710 (LWP 27431)]

[Thread 0x7ffff1689710 (LWP 27424) exited]

[Thread 0x7ffff0e88710 (LWP 27425) exited]

[Thread 0x7fffeaffd710 (LWP 27428) exited]

[Thread 0x7ffff409a710 (LWP 27413) exited]

[Thread 0x7fffea7fc710 (LWP 27429) exited]

[Thread 0x7fffe97fa710 (LWP 27431) exited]

[Thread 0x7fffe9ffb710 (LWP 27430) exited]

[New Thread 0x7fffe97fa710 (LWP 27432)]

[New Thread 0x7fffe9ffb710 (LWP 27433)]

[Thread 0x7fffe9ffb710 (LWP 27433) exited]

[New Thread 0x7fffe9ffb710 (LWP 27434)]

[New Thread 0x7fffea7fc710 (LWP 27435)]

[Thread 0x7fffeb7fe710 (LWP 27427) exited]

[New Thread 0x7fffeb7fe710 (LWP 27436)]

[New Thread 0x7ffff409a710 (LWP 27437)]

[Thread 0x7fffea7fc710 (LWP 27435) exited]

[Thread 0x7fffe9ffb710 (LWP 27434) exited]

[Thread 0x7ffff409a710 (LWP 27437) exited]

[New Thread 0x7ffff409a710 (LWP 27438)]

[New Thread 0x7fffe9ffb710 (LWP 27439)]

[Thread 0x7fffeb7fe710 (LWP 27436) exited]

[Thread 0x7fffe9ffb710 (LWP 27439) exited]

[New Thread 0x7fffe9ffb710 (LWP 27440)]

[New Thread 0x7fffeb7fe710 (LWP 27441)]

[Thread 0x7fffeb7fe710 (LWP 27441) exited]

[Thread 0x7fffe97fa710 (LWP 27432) exited]

[New Thread 0x7fffe97fa710 (LWP 27442)]

[New Thread 0x7fffeb7fe710 (LWP 27443)]

[Thread 0x7ffff409a710 (LWP 27438) exited]

[Thread 0x7fffeb7fe710 (LWP 27443) exited]

[Thread 0x7fffe9ffb710 (LWP 27440) exited]

[New Thread 0x7fffe9ffb710 (LWP 27444)]

[New Thread 0x7fffeb7fe710 (LWP 27445)]

[Thread 0x7fffeb7fe710 (LWP 27445) exited]

[New Thread 0x7fffeb7fe710 (LWP 27446)]

[New Thread 0x7ffff409a710 (LWP 27447)]

[Thread 0x7fffe97fa710 (LWP 27442) exited]

[Thread 0x7ffff409a710 (LWP 27447) exited]

[New Thread 0x7ffff409a710 (LWP 27448)]

[Thread 0x7ffff409a710 (LWP 27448) exited]

[New Thread 0x7ffff409a710 (LWP 27449)]

[New Thread 0x7fffe97fa710 (LWP 27450)]

[Thread 0x7fffe97fa710 (LWP 27450) exited]

Program received signal SIGUSR2, User defined signal 2.

[Switching to Thread 0x7ffff2e8c710 (LWP 27417)]

0x00007ffff67d30bd in read () from /lib/libpthread.so.0

(gdb) thread apply all bt

Thread 37 (Thread 0x7ffff409a710 (LWP 27449)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0xe24ae8, ptr=0x7ffff40999fc "", nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xe24ae8) at bsock.c:451

#3 0x0000000000424120 in do_fd_commands (jcr=0x8d0ac8) at fd_cmds.c:149

#4 0x0000000000424b7a in run_job (jcr=0x8d0ac8) at fd_cmds.c:124

#5 0x000000000042541b in run_cmd (jcr=0x8d0ac8) at job.c:225

#6 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#7 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#8 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#9 0x00007ffff538401d in clone () from /lib/libc.so.6

#10 0x0000000000000000 in ?? ()

Thread 34 (Thread 0x7fffeb7fe710 (LWP 27446)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0xbe3788, ptr=0x7fffeb7fd9fc "", nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xbe3788) at bsock.c:451

#3 0x0000000000424120 in do_fd_commands (jcr=0x6a7b78) at fd_cmds.c:149

#4 0x0000000000424b7a in run_job (jcr=0x6a7b78) at fd_cmds.c:124

#5 0x000000000042541b in run_cmd (jcr=0x6a7b78) at job.c:225

#6 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#7 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#8 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#9 0x00007ffff538401d in clone () from /lib/libc.so.6

#10 0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7fffe9ffb710 (LWP 27444)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0xe053d8, ptr=0x7fffe9ffa77c "\377\377\377\377 \247\265", nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xe053d8) at bsock.c:451

#3 0x00007ffff7171fda in bget_msg (sock=0xe053d8) at bget_msg.c:60

#4 0x00000000004104b9 in do_append_data (jcr=0xcbc598) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0xcbc598) at fd_cmds.c:203

#6 0x00000000004241bb in do_fd_commands (jcr=0xcbc598) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0xcbc598) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0xcbc598) at job.c:225

#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fffebfff710 (LWP 27426)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0xcbb408, ptr=0x7fffebffe77c "\377\377\377\377\020\265", <incomplete sequence \313>, nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xcbb408) at bsock.c:451

#3 0x00007ffff7171fda in bget_msg (sock=0xcbb408) at bget_msg.c:60

#4 0x00000000004104b9 in do_append_data (jcr=0x68a8f8) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0x68a8f8) at fd_cmds.c:203

#6 0x00000000004241bb in do_fd_commands (jcr=0x68a8f8) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0x68a8f8) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0x68a8f8) at job.c:225

#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7ffff1e8a710 (LWP 27423)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x74d498,

ptr=0x7fffec0dca7c "\344mZ\231I=\241\237]QY\037\003\247\247\355\063@\204\210\031\314\373ϓ \340\rj\277e}\fۈ\313\362\204\311ג\a:\241\341\343&\303;\017؊\261.7\376\274\310\346v$\270\224E\226d\245\201<\271\274}$%\f\v=EC\270\071\335\021\212l\201\262_s\340\067\062/N\016a\322\025\066B\t\276\340r\320B\006[\360\206<\352\204\374JAr\306u\235\241\nd\262\244\370i\275\335w\247\261\346\022\204\025\017+\203\230{I\364\205\376\065\251;\260\325\227ՖO\247\025e\r\257\006\346\210\021\023\177\323\r\364\254\241\207N\023\205`SG{\023-\372\235&\332t\334\017?\360\330\023\255m\346\036E\373\023-\262"..., nbytes=37797) at tls.c:642

#2 tls_bsock_readn (bsock=0x74d498,

ptr=0x7fffec0dca7c "\344mZ\231I=\241\237]QY\037\003\247\247\355\063@\204\210\031\314\373ϓ \340\rj\277e}\fۈ\313\362\204\311ג\a:\241\341\343&\303;\017؊\261.7\376\274\310\346v$\270\224E\226d\245\201<\271\274}$%\f\v=EC\270\071\335\021\212l\201\262_s\340\067\062/N\016a\322\025\066B\t\276\340r\320B\006[\360\206<\352\204\374JAr\306u\235\241\nd\262\244\370i\275\335w\247\261\346\0---Type <return> to continue, or q <return> to quit---

22\204\025\017+\203\230{I\364\205\376\065\251;\260\325\227ՖO\247\025e\r\257\006\346\210\021\023\177\323\r\364\254\241\207N\023\205`SG{\023-\372\235&\332t\334\017?\360\330\023\255m\346\036E\373\023-\262"..., nbytes=37797) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x74d498) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0x74d498) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0x8aced8) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x8aced8) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x8aced8) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0x8aced8) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x8aced8) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7ffff268b710 (LWP 27418)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x689b48, ptr=0xe412cc "h\323{Ψ\246\330", <incomplete sequence \345>, nbytes=16388) at tls.c:642

#2 tls_bsock_readn (bsock=0x689b48, ptr=0xe412cc "h\323{Ψ\246\330", <incomplete sequence \345>, nbytes=16388) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x689b48) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0x689b48) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0x8c0118) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x8c0118) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x8c0118) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0x8c0118) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x8c0118) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7ffff2e8c710 (LWP 27417)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff5e93fd1 in ?? () from /usr/lib/libcrypto.so.0.9.8

#2 0x00007ffff5e92279 in BIO_read () from /usr/lib/libcrypto.so.0.9.8

#3 0x00007ffff6189ffd in ssl3_read_n () from /usr/lib/libssl.so.0.9.8

#4 0x00007ffff618a5bf in ssl3_read_bytes () from /usr/lib/libssl.so.0.9.8

#5 0x00007ffff6186fbc in ssl3_shutdown () from /usr/lib/libssl.so.0.9.8

#6 0x00007ffff71922ac in tls_bsock_shutdown (bsock=0xbfa938) at tls.c:578

#7 0x00007ffff717531f in BSOCK::close (this=0xbfa938) at bsock.c:889

#8 0x0000000000424c45 in stored_free_jcr (jcr=0x8ae318) at job.c:367

#9 0x00007ffff717f78f in b_free_jcr (file=<value optimized out>, line=<value optimized out>, jcr=0x8ae318) at jcr.c:567

#10 0x00000000004215b8 in handle_connection_request (arg=<value optimized out>) at dircmd.c:252

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7ffff368d710 (LWP 27416)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x688f18, ptr=0x7fffec0cc9cc " \373\377\377\271\020\231D", nbytes=49156) at tls.c:642

#2 tls_bsock_readn (bsock=0x688f18, ptr=0x7fffec0cc9cc " \373\377\377\271\020\231D", nbytes=49156) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x688f18) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0x688f18) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0x7f8d88) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x7f8d88) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x7f8d88) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0x7f8d88) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x7f8d88) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ffff509c710 (LWP 27412)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0x688a68, ptr=0x7ffff509b77c "\377\377\377\377\f", <incomplete sequence \374>, nbytes=4) at bnet.c:80

---Type <return> to continue, or q <return> to quit---

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0x688a68) at bsock.c:451

#3 0x00007ffff7171fda in bget_msg (sock=0x688a68) at bget_msg.c:60

#4 0x00000000004104b9 in do_append_data (jcr=0x690dd8) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0x690dd8) at fd_cmds.c:203

#6 0x00000000004241bb in do_fd_commands (jcr=0x690dd8) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0x690dd8) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0x690dd8) at job.c:225

#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ffff489b710 (LWP 27309)):

#0 0x00007ffff67d04d9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

#1 0x00007ffff719a18c in watchdog_thread (arg=<value optimized out>) at watchdog.c:308

#2 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#3 0x00007ffff538401d in clone () from /lib/libc.so.6

#4 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fe6720 (LWP 27304)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71739e1 in bnet_thread_server (addrs=<value optimized out>, max_clients=<value optimized out>, client_wq=<value optimized out>, handle_client_request=<value optimized out>)

at bnet_server.c:161

#2 0x0000000000408ac2 in main (argc=<value optimized out>, argv=<value optimized out>) at stored.c:312

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate 
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
lucky parental unit.  See the prize list and enter to win: 
http://p.sf.net/sfu/thinkgeek-promo

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users