Bacula-users

[Bacula-users] bacula storage daemon hang issue

2009-04-24 14:16:26
Subject: [Bacula-users] bacula storage daemon hang issue
From: Bob Hetzel <beh AT case DOT edu>
To: bacula-users AT lists.sourceforge DOT net
Date: Fri, 24 Apr 2009 14:08:52 -0400
Greetings all,

I'm running Bacula 3.0.0 on x86_64 Centos 5.2. The upgrade from 2.4.4 to 3.0 went fairly well and there haven't been any significant problems until now--about a week later. I restarted the SD and Dir and things ran ok today so far but I'd expect this will happen again and perhaps not just to me.

Yesterday I ran into this problem in the middle of my backup run. Here's the relevant section of the bacula log file:
Note the gap of 4 hours and 12 minutes--normally backups complete by 13:15
====

23-Apr 12:09 gyrus-dir JobId 74969: Using Device "IBMLTO2-1"
23-Apr 16:21 gyrus-dir JobId 74969: Fatal error: Network error with FD during Backup: ERR=Connection timed out 23-Apr 16:32 gyrus-dir JobId 74969: Fatal error: No Job status returned from FD. 23-Apr 16:32 gyrus-dir JobId 74969: Error: Bacula gyrus-dir 3.0.0 (06Apr09): 23-Apr-2009 16:32:01
  Build OS:               x86_64-unknown-linux-gnu redhat
  JobId:                  74969
  Job:                    dxa2-d630.2009-04-23_11.53.01_35
  Backup Level:           Incremental, since=2009-04-22 12:09:02
Client: "dxa2-d630" 3.0.0 (06Apr09) Linux,Cross-compile,Win32
  FileSet:                "c-drive-dirs" 2007-10-24 16:45:34
  Pool:                   "Default" (From Job resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "Dell-PV136T" (From Job resource)
  Scheduled time:         23-Apr-2009 11:53:01
  Start time:             23-Apr-2009 12:09:20
  End time:               23-Apr-2009 16:32:01
  Elapsed time:           4 hours 22 mins 41 secs
  Priority:               15
  FD Files Written:       0
  SD Files Written:       0
  FD Bytes Written:       0 (0 B)
  SD Bytes Written:       0 (0 B)
  Rate:                   0.0 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):
  Volume Session Id:      1730
  Volume Session Time:    1239378672
  Last Volume Bytes:      278,061,493,248 (278.0 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  Error
  SD termination status:  Error
  Termination:            *** Backup Error ***

23-Apr 16:32 gyrus-dir JobId 74973: shell command: run BeforeJob "/etc/bacula/pingtest.sh emb39" 23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob: PING emb39.MDST.cwru.edu (129.22.181.161) 56(84) bytes of data. 23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob: 64 bytes from emb39.MDST.CWRU.Edu (129.22.181.161): icmp_seq=1 ttl=125 time=0.234 ms
23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob:
23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob: --- emb39.MDST.cwru.edu ping statistics --- 23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob: 1 packets transmitted, 1 received, 0% packet loss, time 0ms 23-Apr 16:32 gyrus-dir JobId 74973: BeforeJob: rtt min/avg/max/mdev = 0.234/0.234/0.234/0.000 ms 23-Apr 16:32 gyrus-dir JobId 74973: Start Backup JobId 74973, Job=emb39.2009-04-23_11.53.01_39 23-Apr 16:42 gyrus-dir JobId 74973: Fatal error: authenticate.c:120 Director unable to authenticate with Storage daemon at "gyrus:9103". Possible causes:
Passwords or names not the same or
Maximum Concurrent Jobs exceeded on the SD or
SD networking messed up (restart daemon).
Please see http://www.bacula.org/en/rel-manual/Bacula_Freque_Asked_Questi.html#SECTION003760000000000000000 for help.

====
I've also attached the SD and Dir traceback files in case that will be of any help. The traceback errors I presume are related to the configure options I've used which were as follows.

export LDFLAGS="-lncurses"

CFLAGS="-g -Wall" \
 ./configure \
  --enable-batch-insert \
  --disable-ipv6 \
  --disable-libtool \
  --enable-smartalloc \
  --enable-bwx-console \
  --enable-tray-monitor \
  --enable-conio \
  --sbindir=/usr/sbin \
  --with-subsys-dir=/var/lock/subsys \
  --with-mysql \
  --with-dir-user=bacula \
  --with-dir-group=bacula \
  --with-sd-user=root \
  --with-pid-dir=/var/bacula \
  --with-sd-group=bacula \
  --enable-bat \
  --with-qwt=/usr/lib \
  --enable-gnome \
  --enable-static-fd


>>From root AT localhost.case DOT edu  Fri Apr 24 10:44:04 2009
Return-Path: <root AT localhost.case DOT edu>
X-Original-To: root@localhost
Delivered-To: root AT localhost.case DOT edu
Received: from gyrus.case.edu (gyrus.case.edu [127.0.0.1])
        by gyrus.case.edu (Postfix) with SMTP id ED416DFD64
        for <root@localhost>; Fri, 24 Apr 2009 10:44:03 -0400 (EDT)
From: root AT localhost.case DOT edu
Subject: Bacula GDB traceback of bacula-sd on gyrus.case.edu
Sender: root AT gyrus.case DOT edu
To: root AT localhost.case DOT edu
Date: Fri, 24 Apr 2009 10:44:03 -0400 (EDT)
Message-Id: <20090424144403.ED416DFD64 AT gyrus.case DOT edu>
Status: RO
Content-Length: 46327
Lines: 767


warning: .dynamic section for "/usr/lib64/libncurses.so.5" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libacl.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libz.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libpthread.so.0" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libssl.so.6" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcrypto.so.6" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libm.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libc.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libdl.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libattr.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libgssapi_krb5.so.2" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5.so.3" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcom_err.so.2" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libk5crypto.so.3" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5support.so.0" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libkeyutils.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libresolv.so.2" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libselinux.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libsepol.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations
[Thread debugging using libthread_db enabled]
[New Thread 0x2b518e7ab5c0 (LWP 22571)]
[New Thread 0x5b589940 (LWP 22119)]
[New Thread 0x5ab88940 (LWP 22101)]
[New Thread 0x5a187940 (LWP 22064)]
[New Thread 0x59786940 (LWP 22055)]
[New Thread 0x58d85940 (LWP 22026)]
[New Thread 0x58384940 (LWP 22014)]
[New Thread 0x57983940 (LWP 21986)]
[New Thread 0x56f82940 (LWP 21977)]
[New Thread 0x56581940 (LWP 21949)]
[New Thread 0x55b80940 (LWP 21943)]
[New Thread 0x5517f940 (LWP 21905)]
[New Thread 0x5477e940 (LWP 21896)]
[New Thread 0x53d7d940 (LWP 21853)]
[New Thread 0x5337c940 (LWP 21844)]
[New Thread 0x5297b940 (LWP 21804)]
[New Thread 0x51f7a940 (LWP 21789)]
[New Thread 0x51579940 (LWP 21760)]
[New Thread 0x50b78940 (LWP 21753)]
[New Thread 0x4f776940 (LWP 21722)]
[New Thread 0x4ed75940 (LWP 21716)]
[New Thread 0x4e374940 (LWP 21691)]
[New Thread 0x4d973940 (LWP 21685)]
[New Thread 0x4cf72940 (LWP 21656)]
[New Thread 0x4c571940 (LWP 21647)]
[New Thread 0x4bb70940 (LWP 21601)]
[New Thread 0x4b16f940 (LWP 21590)]
[New Thread 0x4a76e940 (LWP 21573)]
[New Thread 0x47569940 (LWP 21545)]
[New Thread 0x46167940 (LWP 21516)]
[New Thread 0x49d6d940 (LWP 20998)]
[New Thread 0x50177940 (LWP 20976)]
[New Thread 0x45766940 (LWP 20949)]
[New Thread 0x41546940 (LWP 20859)]
[New Thread 0x4896b940 (LWP 20829)]
[New Thread 0x44364940 (LWP 20814)]
[New Thread 0x44d65940 (LWP 20800)]
[New Thread 0x46b68940 (LWP 20789)]
[New Thread 0x41f47940 (LWP 20725)]
[New Thread 0x4936c940 (LWP 20717)]
[New Thread 0x43963940 (LWP 20622)]
[New Thread 0x47f6a940 (LWP 20514)]
[New Thread 0x42f62940 (LWP 22588)]
0x00002b518d3f94c2 in select () from /lib64/libc.so.6
$1 = "gyrus-sd", '\0' <repeats 21 times>
$2 = 0x9573538 "bacula-sd"
$3 = 0x9573578 "/usr/sbin/"
$4 = 0x0
$5 = 0x483d88 "3.0.0 (06 April 2009)"
$6 = 0x483d9e "x86_64-unknown-linux-gnu"
$7 = 0x483db7 "redhat"
$8 = 0x483dbe ""
$9 = "gyrus.case.edu", '\0' <repeats 35 times>
#0  0x00002b518d3f94c2 in select () from /lib64/libc.so.6
#1  0x000000000044b04c in bnet_thread_server (addrs=0x9575388, max_clients=41, 
client_wq=0x695c00, 
    handle_client_request=0x422fe2 <handle_connection_request(void*)>) at 
bnet_server.c:161
#2  0x000000000040748b in main (argc=0, argv=0x7fff1e8e04b8) at stored.c:306

Thread 43 (Thread 0x42f62940 (LWP 22588)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x696920) at lockmgr.c:63
#4  0x00000000004573da in lock_jcr_chain () at jcr.c:855
#5  0x000000000045741a in jcr_walk_start () at jcr.c:892
#6  0x0000000000458379 in jcr_timeout_check (self=0x958cee8) at jcr.c:972
#7  0x000000000046aa75 in watchdog_thread (arg=0x0) at watchdog.c:277
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x47f6a940 (LWP 20514)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x959f1e0) at lockmgr.c:63
#4  0x000000000040d6cf in DEVICE::dlock (this=0x959f1a8) at dev.h:452
#5  0x000000000042c622 in DEVICE::r_dlock (this=0x959f1a8) at lock.c:208
#6  0x000000000042c7a0 in DEVICE::dblock (this=0x959f1a8, why=7) at lock.c:142
#7  0x000000000043d25d in DCR::dblock (this=0x967fb48, why=7) at dev.h:534
#8  0x000000000043bc98 in despool_data (dcr=0x967fb48, commit=true) at 
spool.c:248
#9  0x000000000043cd54 in commit_data_spool (dcr=0x967fb48) at spool.c:137
#10 0x00000000004108fb in do_append_data (jcr=0x967ded8) at append.c:307
#11 0x0000000000427270 in append_data_cmd (jcr=0x967ded8) at fd_cmds.c:203
#12 0x00000000004268aa in do_fd_commands (jcr=0x967ded8) at fd_cmds.c:166
#13 0x00000000004273c2 in run_job (jcr=0x967ded8) at fd_cmds.c:128
#14 0x0000000000427ebb in run_cmd (jcr=0x967ded8) at job.c:212
#15 0x000000000042361c in handle_connection_request (arg=0x2aaaac0078c8) at 
dircmd.c:231
#16 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#17 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#18 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x43963940 (LWP 20622)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x958d418) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x958d418, why=7) at lock.c:142
#3  0x000000000043d25d in DCR::dblock (this=0x9598da8, why=7) at dev.h:534
#4  0x000000000043bc98 in despool_data (dcr=0x9598da8, commit=true) at 
spool.c:248
#5  0x000000000043cd54 in commit_data_spool (dcr=0x9598da8) at spool.c:137
#6  0x00000000004108fb in do_append_data (jcr=0x9598288) at append.c:307
#7  0x0000000000427270 in append_data_cmd (jcr=0x9598288) at fd_cmds.c:203
#8  0x00000000004268aa in do_fd_commands (jcr=0x9598288) at fd_cmds.c:166
#9  0x00000000004273c2 in run_job (jcr=0x9598288) at fd_cmds.c:128
#10 0x0000000000427ebb in run_cmd (jcr=0x9598288) at job.c:212
#11 0x000000000042361c in handle_connection_request (arg=0x2aaaac007f18) at 
dircmd.c:231
#12 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#13 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#14 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x4936c940 (LWP 20717)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x959f1e0) at lockmgr.c:63
#4  0x000000000040d6cf in DEVICE::dlock (this=0x959f1a8) at dev.h:452
#5  0x000000000042c622 in DEVICE::r_dlock (this=0x959f1a8) at lock.c:208
#6  0x000000000042c7a0 in DEVICE::dblock (this=0x959f1a8, why=7) at lock.c:142
#7  0x000000000043d25d in DCR::dblock (this=0x9771d58, why=7) at dev.h:534
#8  0x000000000043bc98 in despool_data (dcr=0x9771d58, commit=true) at 
spool.c:248
#9  0x000000000043cd54 in commit_data_spool (dcr=0x9771d58) at spool.c:137
#10 0x00000000004108fb in do_append_data (jcr=0x958f038) at append.c:307
#11 0x0000000000427270 in append_data_cmd (jcr=0x958f038) at fd_cmds.c:203
#12 0x00000000004268aa in do_fd_commands (jcr=0x958f038) at fd_cmds.c:166
#13 0x00000000004273c2 in run_job (jcr=0x958f038) at fd_cmds.c:128
#14 0x0000000000427ebb in run_cmd (jcr=0x958f038) at job.c:212
#15 0x000000000042361c in handle_connection_request (arg=0x2aaaac0068b8) at 
dircmd.c:231
#16 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#17 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#18 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x41f47940 (LWP 20725)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000046460b in rwl_writelock (rwl=0x697420) at rwlock.c:241
#2  0x000000000046a897 in wd_lock () at watchdog.c:324
#3  0x000000000046ac9a in unregister_watchdog (wd=0x9607078) at watchdog.c:209
#4  0x000000000046c09e in stop_btimer (wid=0x959f128) at btimers.c:264
#5  0x000000000046c1d7 in stop_child_timer (wid=0x959f128) at btimers.c:90
#6  0x000000000044e389 in close_bpipe (bpipe=0x959eff8) at bpipe.c:261
#7  0x000000000044eb76 in run_program_full_output (prog=0x967b5a0 
"/etc/bacula/mtx-changer /dev/sg3 unload 3 /dev/nst0 0", wait=300, 
results=@0x41f46550)
    at bpipe.c:448
#8  0x000000000040c87d in unload_autochanger (dcr=0x958ebb8, loaded=3) at 
autochanger.c:363
#9  0x000000000042f1b8 in DCR::release_volume (this=0x958ebb8) at mount.c:716
#10 0x000000000042f3e7 in DCR::do_unload (this=0x958ebb8) at mount.c:508
#11 0x0000000000430d9b in DCR::mount_next_write_volume (this=0x958ebb8) at 
mount.c:116
#12 0x000000000041fff3 in fixup_device_block_write_error (dcr=0x958ebb8) at 
device.c:127
#13 0x0000000000416b86 in write_block_to_device (dcr=0x958ebb8) at block.c:386
#14 0x000000000043bf2d in despool_data (dcr=0x958ebb8, commit=true) at 
spool.c:295
#15 0x000000000043cd54 in commit_data_spool (dcr=0x958ebb8) at spool.c:137
#16 0x00000000004108fb in do_append_data (jcr=0x959b1c8) at append.c:307
#17 0x0000000000427270 in append_data_cmd (jcr=0x959b1c8) at fd_cmds.c:203
#18 0x00000000004268aa in do_fd_commands (jcr=0x959b1c8) at fd_cmds.c:166
#19 0x00000000004273c2 in run_job (jcr=0x959b1c8) at fd_cmds.c:128
#20 0x0000000000427ebb in run_cmd (jcr=0x959b1c8) at job.c:212
#21 0x000000000042361c in handle_connection_request (arg=0x2aaaac003b08) at 
dircmd.c:231
#22 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#23 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#24 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x46b68940 (LWP 20789)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x959f1e0) at lockmgr.c:63
#4  0x000000000040d6cf in DEVICE::dlock (this=0x959f1a8) at dev.h:452
#5  0x000000000042c622 in DEVICE::r_dlock (this=0x959f1a8) at lock.c:208
#6  0x0000000000416a12 in write_block_to_device (dcr=0x960b3f8) at block.c:351
#7  0x000000000043bf2d in despool_data (dcr=0x960b3f8, commit=true) at 
spool.c:295
#8  0x000000000043cd54 in commit_data_spool (dcr=0x960b3f8) at spool.c:137
#9  0x00000000004108fb in do_append_data (jcr=0x95a6968) at append.c:307
#10 0x0000000000427270 in append_data_cmd (jcr=0x95a6968) at fd_cmds.c:203
#11 0x00000000004268aa in do_fd_commands (jcr=0x95a6968) at fd_cmds.c:166
#12 0x00000000004273c2 in run_job (jcr=0x95a6968) at fd_cmds.c:128
#13 0x0000000000427ebb in run_cmd (jcr=0x95a6968) at job.c:212
#14 0x000000000042361c in handle_connection_request (arg=0x2aaaac002788) at 
dircmd.c:231
#15 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#16 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#17 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x44d65940 (LWP 20800)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x959f1a8) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x959f1a8, why=7) at lock.c:142
#3  0x000000000043d25d in DCR::dblock (this=0x95a31a8, why=7) at dev.h:534
#4  0x000000000043bc98 in despool_data (dcr=0x95a31a8, commit=true) at 
spool.c:248
#5  0x000000000043cd54 in commit_data_spool (dcr=0x95a31a8) at spool.c:137
#6  0x00000000004108fb in do_append_data (jcr=0x9772658) at append.c:307
#7  0x0000000000427270 in append_data_cmd (jcr=0x9772658) at fd_cmds.c:203
#8  0x00000000004268aa in do_fd_commands (jcr=0x9772658) at fd_cmds.c:166
#9  0x00000000004273c2 in run_job (jcr=0x9772658) at fd_cmds.c:128
#10 0x0000000000427ebb in run_cmd (jcr=0x9772658) at job.c:212
#11 0x000000000042361c in handle_connection_request (arg=0x2aaaac008768) at 
dircmd.c:231
#12 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#13 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#14 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x44364940 (LWP 20814)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x958d418) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x958d418, why=7) at lock.c:142
#3  0x000000000043d25d in DCR::dblock (this=0x95a06e8, why=7) at dev.h:534
#4  0x000000000043bc98 in despool_data (dcr=0x95a06e8, commit=true) at 
spool.c:248
#5  0x000000000043cd54 in commit_data_spool (dcr=0x95a06e8) at spool.c:137
#6  0x00000000004108fb in do_append_data (jcr=0x97721d8) at append.c:307
#7  0x0000000000427270 in append_data_cmd (jcr=0x97721d8) at fd_cmds.c:203
#8  0x00000000004268aa in do_fd_commands (jcr=0x97721d8) at fd_cmds.c:166
#9  0x00000000004273c2 in run_job (jcr=0x97721d8) at fd_cmds.c:128
#10 0x0000000000427ebb in run_cmd (jcr=0x97721d8) at job.c:212
#11 0x000000000042361c in handle_connection_request (arg=0x2aaaac0080b8) at 
dircmd.c:231
#12 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#13 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#14 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x4896b940 (LWP 20829)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x958d418) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x958d418, why=7) at lock.c:142
#3  0x000000000043d25d in DCR::dblock (this=0x9609ae8, why=7) at dev.h:534
#4  0x000000000043bc98 in despool_data (dcr=0x9609ae8, commit=true) at 
spool.c:248
#5  0x000000000043cd54 in commit_data_spool (dcr=0x9609ae8) at spool.c:137
#6  0x00000000004108fb in do_append_data (jcr=0x967da78) at append.c:307
#7  0x0000000000427270 in append_data_cmd (jcr=0x967da78) at fd_cmds.c:203
#8  0x00000000004268aa in do_fd_commands (jcr=0x967da78) at fd_cmds.c:166
#9  0x00000000004273c2 in run_job (jcr=0x967da78) at fd_cmds.c:128
#10 0x0000000000427ebb in run_cmd (jcr=0x967da78) at job.c:212
#11 0x000000000042361c in handle_connection_request (arg=0x2aaaac003618) at 
dircmd.c:231
#12 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#13 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#14 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x41546940 (LWP 20859)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x959f1a8) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x959f1a8, why=3) at lock.c:142
#3  0x000000000040e2d0 in acquire_device_for_append (dcr=0x959df18) at 
acquire.c:344
#4  0x000000000040fb2f in do_append_data (jcr=0x967d178) at append.c:85
#5  0x0000000000427270 in append_data_cmd (jcr=0x967d178) at fd_cmds.c:203
#6  0x00000000004268aa in do_fd_commands (jcr=0x967d178) at fd_cmds.c:166
#7  0x00000000004273c2 in run_job (jcr=0x967d178) at fd_cmds.c:128
#8  0x0000000000427ebb in run_cmd (jcr=0x967d178) at job.c:212
#9  0x000000000042361c in handle_connection_request (arg=0x2aaaac002598) at 
dircmd.c:231
#10 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#11 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#12 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x45766940 (LWP 20949)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x958d418) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x958d418, why=3) at lock.c:142
#3  0x000000000040e2d0 in acquire_device_for_append (dcr=0x959d598) at 
acquire.c:344
#4  0x000000000040fb2f in do_append_data (jcr=0x958e2d8) at append.c:85
#5  0x0000000000427270 in append_data_cmd (jcr=0x958e2d8) at fd_cmds.c:203
#6  0x00000000004268aa in do_fd_commands (jcr=0x958e2d8) at fd_cmds.c:166
#7  0x00000000004273c2 in run_job (jcr=0x958e2d8) at fd_cmds.c:128
#8  0x0000000000427ebb in run_cmd (jcr=0x958e2d8) at job.c:212
#9  0x000000000042361c in handle_connection_request (arg=0x2aaaac0081d8) at 
dircmd.c:231
#10 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#11 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#12 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x50177940 (LWP 20976)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000042c6e6 in DEVICE::r_dlock (this=0x958d418) at lock.c:217
#2  0x000000000042c7a0 in DEVICE::dblock (this=0x958d418, why=3) at lock.c:142
#3  0x000000000040e2d0 in acquire_device_for_append (dcr=0x9770f78) at 
acquire.c:344
#4  0x000000000040fb2f in do_append_data (jcr=0x95a0b68) at append.c:85
#5  0x0000000000427270 in append_data_cmd (jcr=0x95a0b68) at fd_cmds.c:203
#6  0x00000000004268aa in do_fd_commands (jcr=0x95a0b68) at fd_cmds.c:166
#7  0x00000000004273c2 in run_job (jcr=0x95a0b68) at fd_cmds.c:128
#8  0x0000000000427ebb in run_cmd (jcr=0x95a0b68) at job.c:212
#9  0x000000000042361c in handle_connection_request (arg=0x2aaaac007048) at 
dircmd.c:231
#10 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#11 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#12 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x49d6d940 (LWP 20998)):
#0  0x00002b518c3ee899 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x000000000046460b in rwl_writelock (rwl=0x6964c0) at rwlock.c:241
#2  0x00000000004418e9 in _lock_volumes () at vol_mgr.c:113
#3  0x0000000000437f14 in DCR::unreserve_device (this=0x9608868) at 
reserve.c:148
#4  0x000000000040d758 in detach_dcr_from_dev (dcr=0x9608868) at acquire.c:652
#5  0x000000000040d7a1 in free_dcr (dcr=0x9608868) at acquire.c:668
#6  0x00000000004276fd in stored_free_jcr (jcr=0x960a668) at job.c:393
#7  0x0000000000458195 in b_free_jcr (file=0x47591c "dircmd.c", line=328, 
jcr=0x960a668) at jcr.c:571
#8  0x0000000000422cb7 in cancel_cmd (cjcr=0x9608ce8) at dircmd.c:328
#9  0x000000000042361c in handle_connection_request (arg=0x2aaaac002b28) at 
dircmd.c:231
#10 0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#11 0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#12 0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x46167940 (LWP 21516)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x696920) at lockmgr.c:63
#4  0x00000000004573da in lock_jcr_chain () at jcr.c:855
#5  0x0000000000458c3d in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:386
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac0012f8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x47569940 (LWP 21545)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac009648) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x4a76e940 (LWP 21573)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac009768) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x4b16f940 (LWP 21590)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac003d18) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x4bb70940 (LWP 21601)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac006b98) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x4c571940 (LWP 21647)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac005f18) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x4cf72940 (LWP 21656)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac0017a8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x4d973940 (LWP 21685)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac001ea8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x4e374940 (LWP 21691)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac0046d8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x4ed75940 (LWP 21716)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00a188) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x4f776940 (LWP 21722)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00a888) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x50b78940 (LWP 21753)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00b058) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x51579940 (LWP 21760)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00b828) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x51f7a940 (LWP 21789)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00bff8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x5297b940 (LWP 21804)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00c7c8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x5337c940 (LWP 21844)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00cf98) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x53d7d940 (LWP 21853)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00d768) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x5477e940 (LWP 21896)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00df38) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x5517f940 (LWP 21905)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00e708) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x55b80940 (LWP 21943)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00eed8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x56581940 (LWP 21949)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00f6a8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x56f82940 (LWP 21977)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac00fe78) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x57983940 (LWP 21986)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac010648) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x58384940 (LWP 22014)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac010e18) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x58d85940 (LWP 22026)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac0115e8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x59786940 (LWP 22055)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac011db8) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x5a187940 (LWP 22064)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac012588) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x5ab88940 (LWP 22101)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac012d58) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x5b589940 (LWP 22119)):
#0  0x00002b518c3f0e74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b518c3ec874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x00002b518c3ec2e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000046efda in lmgr_p (m=0x6969a0) at lockmgr.c:63
#4  0x00000000004573fa in lock_jobs () at jcr.c:95
#5  0x0000000000458c38 in new_jcr (size=1064, daemon_free_jcr=0x427478 
<stored_free_jcr(JCR*)>) at jcr.c:385
#6  0x00000000004232a9 in handle_connection_request (arg=0x2aaaac013528) at 
dircmd.c:188
#7  0x000000000046b56c in workq_server (arg=0x695c00) at workq.c:346
#8  0x00002b518c3ea367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002b518d3fff7d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b518e7ab5c0 (LWP 22571)):
#0  0x00002b518d3f94c2 in select () from /lib64/libc.so.6
#1  0x000000000044b04c in bnet_thread_server (addrs=0x9575388, max_clients=41, 
client_wq=0x695c00, 
    handle_client_request=0x422fe2 <handle_connection_request(void*)>) at 
bnet_server.c:161
#2  0x000000000040748b in main (argc=0, argv=0x7fff1e8e04b8) at stored.c:306
#0  0x00002b518d3f94c2 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x000000000044b04c in bnet_thread_server (addrs=0x9575388, max_clients=41, 
client_wq=0x695c00, 
    handle_client_request=0x422fe2 <handle_connection_request(void*)>) at 
bnet_server.c:161
161           if ((stat = select(maxfd + 1, &sockset, NULL, NULL, NULL)) < 0) {
Current language:  auto; currently c++
maxfd = 5
sockset = {fds_bits = {32, 0 <repeats 15 times>}}
newsockfd = 111
stat = 0
clilen = 16
cli_addr = {sa_family = 2, sa_data = 
"Äÿ\177\000\000\001\000\000\000\000\000\000\000"}
tlog = 0
turnon = 1
p = (IPADDR *) 0x0
fd_ptr = (s_sockfd *) 0x0
buf = 
"127.0.0.1\000.224\000\000\000\000\000\000\000\000\000\000p\004\216\036ÿ\177", 
'\0' <repeats 26 times>, "5=\035\214Q+\000\000\005", '\0' <repeats 15 times>, 
"\b^>\214Q+\000\000H> 
\214Q+\000\000x]>\214Q+\000\000Å\033\216\036ÿ\177\000\000°\002\216\036ÿ\177\000\000²\230\035\214Q+\000"
sockfds = {<SMARTALLOC> = {<No data fields>}, head = 0x7fff1e8df620, tail = 
0x7fff1e8df620, loffset = 0, num_items = 1}
allbuf = 
"\000\000\000\000\000\000\000\000]\005\000\000\000\000\000\000Øø\215\036ÿ\177\000\000ÀÂX\t\000\000\000\0000ø\215\036ÿ\177\000\000à÷\215\036ÿ\177\000\000Û\214\222D\000\000\000\000ø÷\215\036ÿ\177\000\000\000\000\000\000\000\000\000\000B\002\035\214Q+\000\000\000\000\000\000\000\000\000\0008·X\t",
 '\0' <repeats 24 times>, "ØÚüâ\b.ø\221Q+\000\000`¿X\t", '\0' <repeats 20 
times>, 
"Àég\215Q+\000\0008\000\000\000\000\000\000\0008·X\t\000\000\000\000ÀÂX\t\000\000\000\000èø\215\036ÿ\177\000\000`¿X\t\000\000\000\000\030¢×\221Q+\000\000\003\000\000\000\000\000\000\000\215R\035\214Q+\000\000"...
#2  0x000000000040748b in main (argc=0, argv=0x7fff1e8e04b8) at stored.c:306
306                           &dird_workq, handle_connection_request);
ch = -1
no_signals = false
test_config = false
thid = 1112938816
uid = 0x7fff1e8e1bb0 "root"
gid = 0x7fff1e8e1bb8 "bacula"
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.

>>From root AT localhost.case DOT edu  Fri Apr 24 10:43:47 2009
Return-Path: <root AT localhost.case DOT edu>
X-Original-To: root@localhost
Delivered-To: root AT localhost.case DOT edu
Received: from gyrus.case.edu (gyrus.case.edu [127.0.0.1])
        by gyrus.case.edu (Postfix) with SMTP id 0A64EDFD61
        for <root@localhost>; Fri, 24 Apr 2009 10:43:47 -0400 (EDT)
From: root AT localhost.case DOT edu
Subject: Bacula GDB traceback of bacula-dir on gyrus.case.edu
Sender: root AT gyrus.case DOT edu
To: root AT localhost.case DOT edu
Date: Fri, 24 Apr 2009 10:43:47 -0400 (EDT)
Message-Id: <20090424144347.0A64EDFD61 AT gyrus.case DOT edu>
Status: RO
Content-Length: 20574
Lines: 383


warning: .dynamic section for "/usr/lib64/libncurses.so.5" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libpthread.so.0" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libssl.so.6" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcrypto.so.6" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libm.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libc.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libdl.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libgssapi_krb5.so.2" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5.so.3" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libcom_err.so.2" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libk5crypto.so.3" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libz.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib64/libkrb5support.so.0" is not at the 
expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libkeyutils.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libresolv.so.2" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libselinux.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libsepol.so.1" is not at the expected 
address

warning: difference appears to be caused by prelink, adjusting expectations
[Thread debugging using libthread_db enabled]
[New Thread 0x2aea60f8d930 (LWP 22594)]
[New Thread 0x4a07f940 (LWP 25095)]
[New Thread 0x4e686940 (LWP 20977)]
[New Thread 0x4fa88940 (LWP 20830)]
[New Thread 0x4be82940 (LWP 20815)]
[New Thread 0x4c883940 (LWP 20801)]
[New Thread 0x4b481940 (LWP 20790)]
[New Thread 0x4aa80940 (LWP 20726)]
[New Thread 0x50489940 (LWP 20718)]
[New Thread 0x4967e940 (LWP 20623)]
[New Thread 0x4f087940 (LWP 20538)]
[New Thread 0x48c7d940 (LWP 20475)]
[New Thread 0x4827c940 (LWP 20474)]
[New Thread 0x4787b940 (LWP 20473)]
[New Thread 0x46e7a940 (LWP 20472)]
[New Thread 0x46479940 (LWP 20456)]
[New Thread 0x45077940 (LWP 20455)]
[New Thread 0x44676940 (LWP 20454)]
[New Thread 0x43274940 (LWP 20453)]
[New Thread 0x411a9940 (LWP 20452)]
[New Thread 0x45a78940 (LWP 20451)]
[New Thread 0x50e8a940 (LWP 20450)]
[New Thread 0x43c75940 (LWP 20449)]
[New Thread 0x42873940 (LWP 22598)]
[New Thread 0x41e72940 (LWP 22597)]
0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
$1 = "gyrus-dir", '\0' <repeats 20 times>
$2 = 0xf3ce538 "bacula-dir"
$3 = 0xf3ce578 "/usr/sbin/"
$4 = 0xf495548 "MySQL"
$5 = 0x53bf28 "3.0.0 (06 April 2009)"
$6 = 0x53bf3e "x86_64-unknown-linux-gnu"
$7 = 0x53bf57 "redhat"
$8 = 0x53bf5e ""
$9 = "gyrus.case.edu", '\0' <repeats 35 times>
#0  0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000004762ca in bmicrosleep (sec=60, usec=0) at bsys.c:71
#2  0x0000000000437844 in wait_for_next_job (one_shot_job_to_run=0x0) at 
scheduler.c:178
#3  0x000000000040fbaa in main (argc=0, argv=0x7fff4c0ff588) at dird.c:322

Thread 25 (Thread 0x41e72940 (LWP 22597)):
#0  0x00002aea5fbdb4c2 in select () from /lib64/libc.so.6
#1  0x0000000000477e98 in bnet_thread_server (addrs=0xf3d0398, max_clients=20, 
client_wq=0x888900, 
    handle_client_request=0x45551c <handle_UA_client_request>) at 
bnet_server.c:161
#2  0x0000000000455515 in connect_thread (arg=0xf3d0398) at ua_server.c:82
#3  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#4  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x42873940 (LWP 22598)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00000000004a0306 in watchdog_thread (arg=0x0) at watchdog.c:308
#2  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#3  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x43c75940 (LWP 20449)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab0067a88) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab0067a88, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab0067a88) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab0067a88) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x50e8a940 (LWP 20450)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab00c16c8) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab00c16c8, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab00c16c8) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab00c16c8) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x45a78940 (LWP 20451)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf3f3e08, ptr=0x45a77c68 "\v", 
nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf3f3e08) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf3f3e08) at getmsg.c:137
#4  0x000000000041351b in wait_for_job_termination (jcr=0x2aaab001e708, 
timeout=0) at backup.c:366
#5  0x0000000000413f8e in do_backup (jcr=0x2aaab001e708) at backup.c:321
#6  0x0000000000429884 in job_thread (arg=0x2aaab001e708) at job.c:310
#7  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#8  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x411a9940 (LWP 20452)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab00cb538) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab00cb538, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab00cb538) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab00cb538) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x43274940 (LWP 20453)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab006f458) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab006f458, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab006f458) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab006f458) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x44676940 (LWP 20454)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0x2aaab40e3aa8, ptr=0x44675e18 
"E", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0x2aaab40e3aa8) at bsock.c:418
#3  0x0000000000428244 in cancel_storage_daemon_job (jcr=0x2aaab003da18) at 
job.c:479
#4  0x00000000004135fb in wait_for_job_termination (jcr=0x2aaab003da18, 
timeout=180) at backup.c:396
#5  0x000000000041402d in do_backup (jcr=0x2aaab003da18) at backup.c:334
#6  0x0000000000429884 in job_thread (arg=0x2aaab003da18) at job.c:310
#7  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#8  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#9  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x45077940 (LWP 20455)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab00f4388) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab00f4388, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab00f4388) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab00f4388) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x46479940 (LWP 20456)):
#0  0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000004762ca in bmicrosleep (sec=2, usec=0) at bsys.c:71
#2  0x000000000042bbfa in jobq_server (arg=0x888660) at jobq.c:595
#3  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#4  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x46e7a940 (LWP 20472)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab0043058) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab0043058, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab0043058) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab0043058) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x4787b940 (LWP 20473)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab008aa28) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab008aa28, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab008aa28) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab008aa28) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x4827c940 (LWP 20474)):
#0  0x00002aea5ebd0b00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000000000430cf3 in wait_for_storage_daemon_termination 
(jcr=0x2aaab003a098) at msgchan.c:412
#2  0x0000000000413604 in wait_for_job_termination (jcr=0x2aaab003a098, 
timeout=0) at backup.c:400
#3  0x0000000000413f8e in do_backup (jcr=0x2aaab003a098) at backup.c:321
#4  0x0000000000429884 in job_thread (arg=0x2aaab003a098) at job.c:310
#5  0x000000000042b3bd in jobq_server (arg=0x888660) at jobq.c:450
#6  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x48c7d940 (LWP 20475)):
#0  0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000004762ca in bmicrosleep (sec=2, usec=0) at bsys.c:71
#2  0x000000000042bbfa in jobq_server (arg=0x888660) at jobq.c:595
#3  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#4  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x4f087940 (LWP 20538)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf409188, ptr=0x4f086d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf409188) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf409188) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab00c16c8) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x4967e940 (LWP 20623)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0x2aaab40e38f8, ptr=0x4967dd68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0x2aaab40e38f8) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0x2aaab40e38f8) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab00f4388) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x50489940 (LWP 20718)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf40bb18, ptr=0x50488d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf40bb18) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf40bb18) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab0067a88) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x4aa80940 (LWP 20726)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf3f56f8, ptr=0x4aa7fd68 "·+I", 
nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf3f56f8) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf3f56f8) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab008aa28) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x4b481940 (LWP 20790)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf4023f8, ptr=0x4b480d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf4023f8) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf4023f8) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab0043058) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x4c883940 (LWP 20801)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf4165f8, ptr=0x4c882d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf4165f8) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf4165f8) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab003a098) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x4be82940 (LWP 20815)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf829a18, ptr=0x4be81d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf829a18) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf829a18) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab00cb538) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x4fa88940 (LWP 20830)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0x2aaab40e4bc8, ptr=0x4fa87d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0x2aaab40e4bc8) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0x2aaab40e4bc8) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab006f458) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x4e686940 (LWP 20977)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf3fff78, ptr=0x4e685d68 
"\220,B", nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf3fff78) at bsock.c:418
#3  0x0000000000422e7d in bget_dirmsg (bs=0xf3fff78) at getmsg.c:137
#4  0x000000000043102b in msg_thread (arg=0x2aaab001e708) at msgchan.c:376
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x4a07f940 (LWP 25095)):
#0  0x00002aea5ebd32cb in read () from /lib64/libpthread.so.0
#1  0x00000000004777ce in read_nbytes (bsock=0xf3fdaf8, ptr=0x4a07f018 "", 
nbytes=4) at bnet.c:80
#2  0x0000000000479f8c in BSOCK::recv (this=0xf3fdaf8) at bsock.c:418
#3  0x00000000004555bd in handle_UA_client_request (arg=0xf3fdaf8) at 
ua_server.c:139
#4  0x00000000004a0cb4 in workq_server (arg=0x888900) at workq.c:346
#5  0x00002aea5ebcc367 in start_thread () from /lib64/libpthread.so.0
#6  0x00002aea5fbe1f7d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2aea60f8d930 (LWP 22594)):
#0  0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000004762ca in bmicrosleep (sec=60, usec=0) at bsys.c:71
#2  0x0000000000437844 in wait_for_next_job (one_shot_job_to_run=0x0) at 
scheduler.c:178
#3  0x000000000040fbaa in main (argc=0, argv=0x7fff4c0ff588) at dird.c:322
#0  0x00002aea5ebd3b71 in nanosleep () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000004762ca in bmicrosleep (sec=60, usec=0) at bsys.c:71
71         stat = nanosleep(&timeout, NULL);
Current language:  auto; currently c++
timeout = {tv_sec = 60, tv_nsec = 0}
tv = {tv_sec = 140734469501792, tv_usec = 1276115906}
tz = {tz_minuteswest = 6, tz_dsttime = 0}
stat = 10986
#2  0x0000000000437844 in wait_for_next_job (one_shot_job_to_run=0x0) at 
scheduler.c:178
178           bmicrosleep((next_check_secs < twait)?next_check_secs:twait, 0);
twait = 4198
jcr = (JCR *) 0x7fff4c0ff3e0
job = (JOB *) 0x429123
run = (RUN *) 0x125890f515ca8
now = 1240584182
prev = 1240584182
next_job = (job_item *) 0x2aaab002b768
first = false
#3  0x000000000040fbaa in main (argc=0, argv=0x7fff4c0ff588) at dird.c:322
322        while ( (jcr = wait_for_next_job(runjob)) ) {
ch = -1
jcr = (JCR *) 0x2aaab0024348
no_signals = false
test_config = false
uid = 0x7fff4c0ffbab "bacula"
gid = 0x7fff4c0ffbb5 "bacula"
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#0  0x0000000000000000 in ?? ()
No symbol table info available.

------------------------------------------------------------------------------
Crystal Reports &#45; New Free Runtime and 30 Day Trial
Check out the new simplified licensign option that enables unlimited
royalty&#45;free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>
  • [Bacula-users] bacula storage daemon hang issue, Bob Hetzel <=