Re: [Bacula-users] Storage Daemon crash backtrace

On Wed, Jun 23, 2010 at 8:42 AM, Robert LeBlanc <robert AT leblancnet DOT us> wrote:

My SD has been crashing every night and I finally got a backtrace. I don't know what all this means, but I could sure use some help to figure out why it keeps crashing and how to stop it so that I can get some back-ups done.

Thanks,

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Starting program: /usr/sbin/bacula-sd -s -f -c /etc/bacula/bacula-sd.conf
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff509c710 (LWP 27308)]
[New Thread 0x7ffff489b710 (LWP 27309)]
[Thread 0x7ffff509c710 (LWP 27308) exited]
[New Thread 0x7ffff509c710 (LWP 27412)]
[New Thread 0x7ffff409a710 (LWP 27413)]
[New Thread 0x7ffff368d710 (LWP 27414)]

[New Thread 0x7ffff2e8c710 (LWP 27415)]
[Thread 0x7ffff2e8c710 (LWP 27415) exited]
[Thread 0x7ffff368d710 (LWP 27414) exited]
[New Thread 0x7ffff368d710 (LWP 27416)]
[New Thread 0x7ffff2e8c710 (LWP 27417)]

[New Thread 0x7ffff268b710 (LWP 27418)]
[New Thread 0x7ffff1e8a710 (LWP 27423)]
[New Thread 0x7ffff1689710 (LWP 27424)]
[New Thread 0x7ffff0e88710 (LWP 27425)]
[New Thread 0x7fffebfff710 (LWP 27426)]

[New Thread 0x7fffeb7fe710 (LWP 27427)]
[New Thread 0x7fffeaffd710 (LWP 27428)]
[New Thread 0x7fffea7fc710 (LWP 27429)]
[New Thread 0x7fffe9ffb710 (LWP 27430)]
[New Thread 0x7fffe97fa710 (LWP 27431)]

[Thread 0x7ffff1689710 (LWP 27424) exited]
[Thread 0x7ffff0e88710 (LWP 27425) exited]
[Thread 0x7fffeaffd710 (LWP 27428) exited]
[Thread 0x7ffff409a710 (LWP 27413) exited]
[Thread 0x7fffea7fc710 (LWP 27429) exited]

[Thread 0x7fffe97fa710 (LWP 27431) exited]
[Thread 0x7fffe9ffb710 (LWP 27430) exited]
[New Thread 0x7fffe97fa710 (LWP 27432)]
[New Thread 0x7fffe9ffb710 (LWP 27433)]
[Thread 0x7fffe9ffb710 (LWP 27433) exited]

[New Thread 0x7fffe9ffb710 (LWP 27434)]
[New Thread 0x7fffea7fc710 (LWP 27435)]
[Thread 0x7fffeb7fe710 (LWP 27427) exited]
[New Thread 0x7fffeb7fe710 (LWP 27436)]
[New Thread 0x7ffff409a710 (LWP 27437)]

[Thread 0x7fffea7fc710 (LWP 27435) exited]
[Thread 0x7fffe9ffb710 (LWP 27434) exited]
[Thread 0x7ffff409a710 (LWP 27437) exited]
[New Thread 0x7ffff409a710 (LWP 27438)]
[New Thread 0x7fffe9ffb710 (LWP 27439)]

[Thread 0x7fffeb7fe710 (LWP 27436) exited]
[Thread 0x7fffe9ffb710 (LWP 27439) exited]
[New Thread 0x7fffe9ffb710 (LWP 27440)]
[New Thread 0x7fffeb7fe710 (LWP 27441)]
[Thread 0x7fffeb7fe710 (LWP 27441) exited]

[Thread 0x7fffe97fa710 (LWP 27432) exited]
[New Thread 0x7fffe97fa710 (LWP 27442)]
[New Thread 0x7fffeb7fe710 (LWP 27443)]
[Thread 0x7ffff409a710 (LWP 27438) exited]
[Thread 0x7fffeb7fe710 (LWP 27443) exited]

[Thread 0x7fffe9ffb710 (LWP 27440) exited]
[New Thread 0x7fffe9ffb710 (LWP 27444)]
[New Thread 0x7fffeb7fe710 (LWP 27445)]
[Thread 0x7fffeb7fe710 (LWP 27445) exited]
[New Thread 0x7fffeb7fe710 (LWP 27446)]

[New Thread 0x7ffff409a710 (LWP 27447)]
[Thread 0x7fffe97fa710 (LWP 27442) exited]
[Thread 0x7ffff409a710 (LWP 27447) exited]
[New Thread 0x7ffff409a710 (LWP 27448)]
[Thread 0x7ffff409a710 (LWP 27448) exited]

[New Thread 0x7ffff409a710 (LWP 27449)]
[New Thread 0x7fffe97fa710 (LWP 27450)]
[Thread 0x7fffe97fa710 (LWP 27450) exited]

Program received signal SIGUSR2, User defined signal 2.

[Switching to Thread 0x7ffff2e8c710 (LWP 27417)]
0x00007ffff67d30bd in read () from /lib/libpthread.so.0
(gdb) thread apply all bt

Thread 37 (Thread 0x7ffff409a710 (LWP 27449)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0
#1 0x00007ffff7172d56 in read_nbytes (bsock=0xe24ae8, ptr=0x7ffff40999fc "", nbytes=4) at bnet.c:80
#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xe24ae8) at bsock.c:451

#3 0x0000000000424120 in do_fd_commands (jcr=0x8d0ac8) at fd_cmds.c:149
#4 0x0000000000424b7a in run_job (jcr=0x8d0ac8) at fd_cmds.c:124
#5 0x000000000042541b in run_cmd (jcr=0x8d0ac8) at job.c:225

#6 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#7 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346
#8 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#9 0x00007ffff538401d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 34 (Thread 0x7fffeb7fe710 (LWP 27446)):
#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0xbe3788, ptr=0x7fffeb7fd9fc "", nbytes=4) at bnet.c:80
#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xbe3788) at bsock.c:451
#3 0x0000000000424120 in do_fd_commands (jcr=0x6a7b78) at fd_cmds.c:149

#4 0x0000000000424b7a in run_job (jcr=0x6a7b78) at fd_cmds.c:124
#5 0x000000000042541b in run_cmd (jcr=0x6a7b78) at job.c:225
#6 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#7 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346
#8 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#9 0x00007ffff538401d in clone () from /lib/libc.so.6

#10 0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7fffe9ffb710 (LWP 27444)):
#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0
#1 0x00007ffff7172d56 in read_nbytes (bsock=0xe053d8, ptr=0x7fffe9ffa77c "\377\377\377\377 \247\265", nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xe053d8) at bsock.c:451
#3 0x00007ffff7171fda in bget_msg (sock=0xe053d8) at bget_msg.c:60
#4 0x00000000004104b9 in do_append_data (jcr=0xcbc598) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0xcbc598) at fd_cmds.c:203
#6 0x00000000004241bb in do_fd_commands (jcr=0xcbc598) at fd_cmds.c:162
#7 0x0000000000424b7a in run_job (jcr=0xcbc598) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0xcbc598) at job.c:225
#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#12 0x00007ffff538401d in clone () from /lib/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fffebfff710 (LWP 27426)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0
#1 0x00007ffff7172d56 in read_nbytes (bsock=0xcbb408, ptr=0x7fffebffe77c "\377\377\377\377\020\265", <incomplete sequence \313>, nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0xcbb408) at bsock.c:451
#3 0x00007ffff7171fda in bget_msg (sock=0xcbb408) at bget_msg.c:60
#4 0x00000000004104b9 in do_append_data (jcr=0x68a8f8) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0x68a8f8) at fd_cmds.c:203
#6 0x00000000004241bb in do_fd_commands (jcr=0x68a8f8) at fd_cmds.c:162
#7 0x0000000000424b7a in run_job (jcr=0x68a8f8) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0x68a8f8) at job.c:225
#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#12 0x00007ffff538401d in clone () from /lib/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7ffff1e8a710 (LWP 27423)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6
#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x74d498,
   ptr=0x7fffec0dca7c "\344mZ\231I=\241\237]QY\037\003\247\247\355\063@\204\210\031\314\373ϓ \340\rj\277e}\fۈ\313\362\204\311ג\a:\241\341\343&\303;\017؊\261.7\376\274\310\346v$\270\224E\226d\245\201<\271\274}$%\f\v=EC\270\071\335\021\212l\201\262_s\340\067\062/N\016a\322\025\066B\t\276\340r\320B\006[\360\206<\352\204\374JAr\306u\235\241\nd\262\244\370i\275\335w\247\261\346\022\204\025\017+\203\230{I\364\205\376\065\251;\260\325\227ՖO\247\025e\r\257\006\346\210\021\023\177\323\r\364\254\241\207N\023\205`SG{\023-\372\235&\332t\334\017?\360\330\023\255m\346\036E\373\023-\262"..., nbytes=37797) at tls.c:642

#2 tls_bsock_readn (bsock=0x74d498,
   ptr=0x7fffec0dca7c "\344mZ\231I=\241\237]QY\037\003\247\247\355\063@\204\210\031\314\373ϓ \340\rj\277e}\fۈ\313\362\204\311ג\a:\241\341\343&\303;\017؊\261.7\376\274\310\346v$\270\224E\226d\245\201<\271\274}$%\f\v=EC\270\071\335\021\212l\201\262_s\340\067\062/N\016a\322\025\066B\t\276\340r\320B\006[\360\206<\352\204\374JAr\306u\235\241\nd\262\244\370i\275\335w\247\261\346\0---Type <return> to continue, or q <return> to quit---

22\204\025\017+\203\230{I\364\205\376\065\251;\260\325\227ՖO\247\025e\r\257\006\346\210\021\023\177\323\r\364\254\241\207N\023\205`SG{\023-\372\235&\332t\334\017?\360\330\023\255m\346\036E\373\023-\262"..., nbytes=37797) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x74d498) at bsock.c:509
#4 0x00007ffff7171fda in bget_msg (sock=0x74d498) at bget_msg.c:60
#5 0x0000000000410538 in do_append_data (jcr=0x8aced8) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x8aced8) at fd_cmds.c:203
#7 0x00000000004241bb in do_fd_commands (jcr=0x8aced8) at fd_cmds.c:162
#8 0x0000000000424b7a in run_job (jcr=0x8aced8) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x8aced8) at job.c:225
#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#13 0x00007ffff538401d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7ffff268b710 (LWP 27418)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6
#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x689b48, ptr=0xe412cc "h\323{Ψ\246\330", <incomplete sequence \345>, nbytes=16388) at tls.c:642

#2 tls_bsock_readn (bsock=0x689b48, ptr=0xe412cc "h\323{Ψ\246\330", <incomplete sequence \345>, nbytes=16388) at tls.c:693
#3 0x00007ffff7175e17 in BSOCK::recv (this=0x689b48) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0x689b48) at bget_msg.c:60
#5 0x0000000000410538 in do_append_data (jcr=0x8c0118) at append.c:183
#6 0x00000000004249fb in append_data_cmd (jcr=0x8c0118) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x8c0118) at fd_cmds.c:162
#8 0x0000000000424b7a in run_job (jcr=0x8c0118) at fd_cmds.c:124
#9 0x000000000042541b in run_cmd (jcr=0x8c0118) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346
#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7ffff2e8c710 (LWP 27417)):
#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff5e93fd1 in ?? () from /usr/lib/libcrypto.so.0.9.8
#2 0x00007ffff5e92279 in BIO_read () from /usr/lib/libcrypto.so.0.9.8
#3 0x00007ffff6189ffd in ssl3_read_n () from /usr/lib/libssl.so.0.9.8

#4 0x00007ffff618a5bf in ssl3_read_bytes () from /usr/lib/libssl.so.0.9.8
#5 0x00007ffff6186fbc in ssl3_shutdown () from /usr/lib/libssl.so.0.9.8
#6 0x00007ffff71922ac in tls_bsock_shutdown (bsock=0xbfa938) at tls.c:578

#7 0x00007ffff717531f in BSOCK::close (this=0xbfa938) at bsock.c:889
#8 0x0000000000424c45 in stored_free_jcr (jcr=0x8ae318) at job.c:367
#9 0x00007ffff717f78f in b_free_jcr (file=<value optimized out>, line=<value optimized out>, jcr=0x8ae318) at jcr.c:567

#10 0x00000000004215b8 in handle_connection_request (arg=<value optimized out>) at dircmd.c:252
#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346
#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7ffff368d710 (LWP 27416)):
#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x688f18, ptr=0x7fffec0cc9cc " \373\377\377\271\020\231D", nbytes=49156) at tls.c:642
#2 tls_bsock_readn (bsock=0x688f18, ptr=0x7fffec0cc9cc " \373\377\377\271\020\231D", nbytes=49156) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x688f18) at bsock.c:509
#4 0x00007ffff7171fda in bget_msg (sock=0x688f18) at bget_msg.c:60
#5 0x0000000000410538 in do_append_data (jcr=0x7f8d88) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x7f8d88) at fd_cmds.c:203
#7 0x00000000004241bb in do_fd_commands (jcr=0x7f8d88) at fd_cmds.c:162
#8 0x0000000000424b7a in run_job (jcr=0x7f8d88) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x7f8d88) at job.c:225
#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233
#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#13 0x00007ffff538401d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ffff509c710 (LWP 27412)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0
#1 0x00007ffff7172d56 in read_nbytes (bsock=0x688a68, ptr=0x7ffff509b77c "\377\377\377\377\f", <incomplete sequence \374>, nbytes=4) at bnet.c:80

---Type <return> to continue, or q <return> to quit---
#2 0x00007ffff7175bd7 in BSOCK::recv (this=0x688a68) at bsock.c:451
#3 0x00007ffff7171fda in bget_msg (sock=0x688a68) at bget_msg.c:60

#4 0x00000000004104b9 in do_append_data (jcr=0x690dd8) at append.c:151
#5 0x00000000004249fb in append_data_cmd (jcr=0x690dd8) at fd_cmds.c:203
#6 0x00000000004241bb in do_fd_commands (jcr=0x690dd8) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0x690dd8) at fd_cmds.c:124
#8 0x000000000042541b in run_cmd (jcr=0x690dd8) at job.c:225
#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346
#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ffff489b710 (LWP 27309)):
#0 0x00007ffff67d04d9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007ffff719a18c in watchdog_thread (arg=<value optimized out>) at watchdog.c:308

#2 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0
#3 0x00007ffff538401d in clone () from /lib/libc.so.6
#4 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fe6720 (LWP 27304)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6
#1 0x00007ffff71739e1 in bnet_thread_server (addrs=<value optimized out>, max_clients=<value optimized out>, client_wq=<value optimized out>, handle_client_request=<value optimized out>)

   at bnet_server.c:161
#2 0x0000000000408ac2 in main (argc=<value optimized out>, argv=<value optimized out>) at stored.c:312

I started bacula-sd again and got another backtrace pretty quickly, then it finished the rest of the jobs after restarting it a third time.

Starting program: /usr/sbin/bacula-sd -s -f -c /etc/bacula/bacula-sd.conf

[Thread debugging using libthread_db enabled]

[New Thread 0x7ffff509c710 (LWP 27456)]

[New Thread 0x7ffff489b710 (LWP 27457)]

[New Thread 0x7ffff409a710 (LWP 27467)]

[New Thread 0x7ffff3899710 (LWP 27468)]

[New Thread 0x7ffff3098710 (LWP 27469)]

[New Thread 0x7ffff2897710 (LWP 27486)]

[New Thread 0x7ffff2096710 (LWP 27487)]

[New Thread 0x7ffff1895710 (LWP 27488)]

[New Thread 0x7ffff1094710 (LWP 27489)]

[Thread 0x7ffff509c710 (LWP 27456) exited]

[New Thread 0x7ffff509c710 (LWP 27561)]

[Thread 0x7ffff3899710 (LWP 27468) exited]

[Thread 0x7ffff2897710 (LWP 27486) exited]

[Thread 0x7ffff509c710 (LWP 27561) exited]

[Thread 0x7ffff409a710 (LWP 27467) exited]

[Thread 0x7ffff1895710 (LWP 27488) exited]

[Thread 0x7ffff1094710 (LWP 27489) exited]

[Thread 0x7ffff2096710 (LWP 27487) exited]

[New Thread 0x7ffff2096710 (LWP 27563)]

[Thread 0x7ffff2096710 (LWP 27563) exited]

[New Thread 0x7ffff2096710 (LWP 27565)]

[New Thread 0x7ffff1094710 (LWP 27566)]

[New Thread 0x7ffff1895710 (LWP 27567)]

[New Thread 0x7ffff409a710 (LWP 27568)]

[New Thread 0x7ffff509c710 (LWP 27569)]

[New Thread 0x7ffff3899710 (LWP 27570)]

[New Thread 0x7ffff2897710 (LWP 27571)]

[Thread 0x7ffff3098710 (LWP 27469) exited]

[Thread 0x7ffff409a710 (LWP 27568) exited]

[Thread 0x7ffff2897710 (LWP 27571) exited]

[Thread 0x7ffff3899710 (LWP 27570) exited]

[New Thread 0x7ffff3899710 (LWP 27572)]

[New Thread 0x7ffff2897710 (LWP 27573)]

[New Thread 0x7ffff409a710 (LWP 27574)]

[New Thread 0x7ffff3098710 (LWP 27575)]

[New Thread 0x7fffebfff710 (LWP 27576)]

[New Thread 0x7fffeb7fe710 (LWP 27577)]

[New Thread 0x7fffeaffd710 (LWP 27578)]

[New Thread 0x7fffea7fc710 (LWP 27580)]

[Thread 0x7fffeaffd710 (LWP 27578) exited]

[Thread 0x7fffea7fc710 (LWP 27580) exited]

[Thread 0x7fffebfff710 (LWP 27576) exited]

[Thread 0x7fffeb7fe710 (LWP 27577) exited]

[New Thread 0x7fffeb7fe710 (LWP 27581)]

[New Thread 0x7fffebfff710 (LWP 27582)]

[Thread 0x7ffff1895710 (LWP 27567) exited]

[Thread 0x7fffebfff710 (LWP 27582) exited]

[New Thread 0x7fffebfff710 (LWP 27583)]

[Thread 0x7ffff509c710 (LWP 27569) exited]

[New Thread 0x7ffff509c710 (LWP 27584)]

[Thread 0x7ffff509c710 (LWP 27584) exited]

[New Thread 0x7ffff509c710 (LWP 27585)]

[New Thread 0x7ffff1895710 (LWP 27586)]

[Thread 0x7fffeb7fe710 (LWP 27581) exited]

[Thread 0x7ffff1895710 (LWP 27586) exited]

[New Thread 0x7ffff1895710 (LWP 27587)]

[Thread 0x7fffebfff710 (LWP 27583) exited]

[New Thread 0x7fffebfff710 (LWP 27588)]

[Thread 0x7fffebfff710 (LWP 27588) exited]

[New Thread 0x7fffebfff710 (LWP 27589)]

[New Thread 0x7fffeb7fe710 (LWP 27590)]

[Thread 0x7fffeb7fe710 (LWP 27590) exited]

[Thread 0x7ffff1895710 (LWP 27587) exited]

[New Thread 0x7ffff1895710 (LWP 27591)]

[New Thread 0x7fffeb7fe710 (LWP 27592)]

[New Thread 0x7fffea7fc710 (LWP 27593)]

[Thread 0x7ffff1895710 (LWP 27591) exited]

[Thread 0x7ffff3098710 (LWP 27575) exited]

[Thread 0x7fffea7fc710 (LWP 27593) exited]

[New Thread 0x7fffea7fc710 (LWP 27594)]

[New Thread 0x7ffff3098710 (LWP 27595)]

[New Thread 0x7ffff1895710 (LWP 27596)]

[Thread 0x7fffebfff710 (LWP 27589) exited]

[Thread 0x7ffff1094710 (LWP 27566) exited]

[Thread 0x7ffff1895710 (LWP 27596) exited]

[Thread 0x7ffff509c710 (LWP 27585) exited]

[New Thread 0x7ffff509c710 (LWP 27597)]

[New Thread 0x7ffff1895710 (LWP 27598)]

[Thread 0x7ffff1895710 (LWP 27598) exited]

[New Thread 0x7ffff1895710 (LWP 27599)]

[New Thread 0x7ffff1094710 (LWP 27600)]

[Thread 0x7ffff1094710 (LWP 27600) exited]

[Thread 0x7ffff509c710 (LWP 27597) exited]

[New Thread 0x7ffff509c710 (LWP 27601)]

[New Thread 0x7ffff1094710 (LWP 27602)]

[Thread 0x7ffff3899710 (LWP 27572) exited]

[Thread 0x7ffff1094710 (LWP 27602) exited]

[New Thread 0x7ffff1094710 (LWP 27603)]

[New Thread 0x7ffff3899710 (LWP 27604)]

[Thread 0x7fffeb7fe710 (LWP 27592) exited]

[Thread 0x7ffff3899710 (LWP 27604) exited]

[New Thread 0x7ffff3899710 (LWP 27605)]

[New Thread 0x7fffeb7fe710 (LWP 27606)]

[Thread 0x7ffff509c710 (LWP 27601) exited]

[Thread 0x7fffeb7fe710 (LWP 27606) exited]

[New Thread 0x7fffeb7fe710 (LWP 27608)]

[New Thread 0x7ffff509c710 (LWP 27609)]

[Thread 0x7ffff1094710 (LWP 27603) exited]

[Thread 0x7ffff509c710 (LWP 27609) exited]

[New Thread 0x7ffff509c710 (LWP 27610)]

[New Thread 0x7ffff1094710 (LWP 27611)]

[Thread 0x7ffff3899710 (LWP 27605) exited]

[Thread 0x7ffff1094710 (LWP 27611) exited]

[Thread 0x7fffeb7fe710 (LWP 27608) exited]

[New Thread 0x7fffeb7fe710 (LWP 27612)]

[New Thread 0x7ffff1094710 (LWP 27613)]

[Thread 0x7ffff1094710 (LWP 27613) exited]

[New Thread 0x7ffff1094710 (LWP 27614)]

[New Thread 0x7ffff3899710 (LWP 27615)]

[New Thread 0x7fffebfff710 (LWP 27616)]

[Thread 0x7ffff3098710 (LWP 27595) exited]

[Thread 0x7ffff1094710 (LWP 27614) exited]

[Thread 0x7fffebfff710 (LWP 27616) exited]

[New Thread 0x7fffebfff710 (LWP 27617)]

[New Thread 0x7ffff1094710 (LWP 27618)]

[Thread 0x7fffeb7fe710 (LWP 27612) exited]

[Thread 0x7ffff1094710 (LWP 27618) exited]

[New Thread 0x7ffff1094710 (LWP 27619)]

[Thread 0x7ffff1094710 (LWP 27619) exited]

[New Thread 0x7ffff1094710 (LWP 27620)]

[New Thread 0x7fffeb7fe710 (LWP 27621)]

[New Thread 0x7ffff3098710 (LWP 27622)]

[Thread 0x7ffff3899710 (LWP 27615) exited]

[Thread 0x7ffff3098710 (LWP 27622) exited]

[Thread 0x7fffebfff710 (LWP 27617) exited]

[New Thread 0x7fffebfff710 (LWP 27623)]

[New Thread 0x7ffff3098710 (LWP 27624)]

[New Thread 0x7ffff3899710 (LWP 27625)]

[Thread 0x7fffeb7fe710 (LWP 27621) exited]

[Thread 0x7ffff3899710 (LWP 27625) exited]

[New Thread 0x7ffff3899710 (LWP 27626)]

[New Thread 0x7fffeb7fe710 (LWP 27627)]

[Thread 0x7ffff3098710 (LWP 27624) exited]

[Thread 0x7fffeb7fe710 (LWP 27627) exited]

[New Thread 0x7fffeb7fe710 (LWP 27628)]

[New Thread 0x7ffff3098710 (LWP 27629)]

[Thread 0x7ffff3899710 (LWP 27626) exited]

[Thread 0x7ffff3098710 (LWP 27629) exited]

[Thread 0x7fffeb7fe710 (LWP 27628) exited]

Program received signal SIGUSR2, User defined signal 2.

[Switching to Thread 0x7ffff1895710 (LWP 27599)]

0x00007ffff67d30bd in read () from /lib/libpthread.so.0

(gdb) thread apply all bt

Thread 69 (Thread 0x7fffebfff710 (LWP 27623)):

#0 0x00007ffff67d04d9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

#1 0x000000000042547c in run_cmd (jcr=0xe2d5e8) at job.c:212

#2 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#3 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#4 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#5 0x00007ffff538401d in clone () from /lib/libc.so.6

#6 0x0000000000000000 in ?? ()

Thread 66 (Thread 0x7ffff1094710 (LWP 27620)):

#0 0x00007ffff67d04d9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

#1 0x000000000042547c in run_cmd (jcr=0xb88ab8) at job.c:212

#2 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#3 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#4 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#5 0x00007ffff538401d in clone () from /lib/libc.so.6

#6 0x0000000000000000 in ?? ()

Thread 56 (Thread 0x7ffff509c710 (LWP 27610)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0x942fa8, ptr=0x7ffff509b77c "\377\377\377\377\200\326\301", nbytes=4) at bnet.c:80

#2 0x00007ffff7175bd7 in BSOCK::recv (this=0x942fa8) at bsock.c:451

#3 0x00007ffff7171fda in bget_msg (sock=0x942fa8) at bget_msg.c:60

#4 0x00000000004104b9 in do_append_data (jcr=0x942668) at append.c:151

#5 0x00000000004249fb in append_data_cmd (jcr=0x942668) at fd_cmds.c:203

#6 0x00000000004241bb in do_fd_commands (jcr=0x942668) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0x942668) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0x942668) at job.c:225

#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 46 (Thread 0x7ffff1895710 (LWP 27599)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff5e93fd1 in ?? () from /usr/lib/libcrypto.so.0.9.8

#2 0x00007ffff5e92279 in BIO_read () from /usr/lib/libcrypto.so.0.9.8

#3 0x00007ffff6189ffd in ssl3_read_n () from /usr/lib/libssl.so.0.9.8

#4 0x00007ffff618a5bf in ssl3_read_bytes () from /usr/lib/libssl.so.0.9.8

#5 0x00007ffff6186fbc in ssl3_shutdown () from /usr/lib/libssl.so.0.9.8

#6 0x00007ffff71922ac in tls_bsock_shutdown (bsock=0x8a5f38) at tls.c:578

#7 0x00007ffff717531f in BSOCK::close (this=0x8a5f38) at bsock.c:889

#8 0x0000000000424c45 in stored_free_jcr (jcr=0x679688) at job.c:367

#9 0x00007ffff717f78f in b_free_jcr (file=<value optimized out>, line=<value optimized out>, jcr=0x679688) at jcr.c:567

#10 0x00000000004215b8 in handle_connection_request (arg=<value optimized out>) at dircmd.c:252

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 41 (Thread 0x7fffea7fc710 (LWP 27594)):

#0 0x00007ffff67d30bd in read () from /lib/libpthread.so.0

#1 0x00007ffff7172d56 in read_nbytes (bsock=0x940858, ptr=0xc60b10 "\001", nbytes=65536) at bnet.c:80

#2 0x00007ffff7175e17 in BSOCK::recv (this=0x940858) at bsock.c:509

#3 0x00007ffff7171fda in bget_msg (sock=0x940858) at bget_msg.c:60

#4 0x0000000000410538 in do_append_data (jcr=0x697268) at append.c:183

#5 0x00000000004249fb in append_data_cmd (jcr=0x697268) at fd_cmds.c:203

#6 0x00000000004241bb in do_fd_commands (jcr=0x697268) at fd_cmds.c:162

#7 0x0000000000424b7a in run_job (jcr=0x697268) at fd_cmds.c:124

#8 0x000000000042541b in run_cmd (jcr=0x697268) at job.c:225

#9 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#10 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#11 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#12 0x00007ffff538401d in clone () from /lib/libc.so.6

#13 0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7ffff409a710 (LWP 27574)):

---Type <return> to continue, or q <return> to quit---

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0xcda628, ptr=0xf6cb0c "s:%d:-1:%u:%u:%u:-1:%u:%u:%u:%lu:%u:%u:%u:%u:%s", nbytes=32772) at tls.c:642

#2 tls_bsock_readn (bsock=0xcda628, ptr=0xf6cb0c "s:%d:-1:%u:%u:%u:-1:%u:%u:%u:%lu:%u:%u:%u:%u:%s", nbytes=32772) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0xcda628) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0xcda628) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0xb65858) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0xb65858) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0xb65858) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0xb65858) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0xb65858) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7ffff2897710 (LWP 27573)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0xcebaa8, ptr=0x7ffff289677c "\377\177", nbytes=4) at tls.c:642

#2 tls_bsock_readn (bsock=0xcebaa8, ptr=0x7ffff289677c "\377\177", nbytes=4) at tls.c:693

#3 0x00007ffff7175bd7 in BSOCK::recv (this=0xcebaa8) at bsock.c:451

#4 0x00007ffff7171fda in bget_msg (sock=0xcebaa8) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0x7fffec0008d8) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x7fffec0008d8) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x7fffec0008d8) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0x7fffec0008d8) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x7fffec0008d8) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7ffff2096710 (LWP 27565)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71921df in openssl_bsock_readwrite (bsock=0x697968, ptr=0xb16b4c "\302\016\\\004", nbytes=16388) at tls.c:642

#2 tls_bsock_readn (bsock=0x697968, ptr=0xb16b4c "\302\016\\\004", nbytes=16388) at tls.c:693

#3 0x00007ffff7175e17 in BSOCK::recv (this=0x697968) at bsock.c:509

#4 0x00007ffff7171fda in bget_msg (sock=0x697968) at bget_msg.c:60

#5 0x0000000000410538 in do_append_data (jcr=0x74ce98) at append.c:183

#6 0x00000000004249fb in append_data_cmd (jcr=0x74ce98) at fd_cmds.c:203

#7 0x00000000004241bb in do_fd_commands (jcr=0x74ce98) at fd_cmds.c:162

#8 0x0000000000424b7a in run_job (jcr=0x74ce98) at fd_cmds.c:124

#9 0x000000000042541b in run_cmd (jcr=0x74ce98) at job.c:225

#10 0x000000000042169f in handle_connection_request (arg=<value optimized out>) at dircmd.c:233

#11 0x00007ffff719a619 in workq_server (arg=<value optimized out>) at workq.c:346

#12 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#13 0x00007ffff538401d in clone () from /lib/libc.so.6

#14 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ffff489b710 (LWP 27457)):

#0 0x00007ffff67d04d9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

#1 0x00007ffff719a18c in watchdog_thread (arg=<value optimized out>) at watchdog.c:308

#2 0x00007ffff67cb8ba in start_thread () from /lib/libpthread.so.0

#3 0x00007ffff538401d in clone () from /lib/libc.so.6

#4 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fe6720 (LWP 27454)):

#0 0x00007ffff537d8b3 in select () from /lib/libc.so.6

#1 0x00007ffff71739e1 in bnet_thread_server (addrs=<value optimized out>, max_clients=<value optimized out>, client_wq=<value optimized out>, handle_client_request=<value optimized out>)

at bnet_server.c:161

#2 0x0000000000408ac2 in main (argc=<value optimized out>, argv=<value optimized out>) at stored.c:312

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate 
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
lucky parental unit.  See the prize list and enter to win: 
http://p.sf.net/sfu/thinkgeek-promo

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users