Bacula-users

[Bacula-users] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive

2009-01-06 10:02:02
Subject: [Bacula-users] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive
From: Bob Hetzel <beh AT case DOT edu>
To: bacula-devel AT lists.sourceforge DOT net, bacula-users AT lists.sourceforge DOT net
Date: Tue, 06 Jan 2009 09:59:13 -0500
Hi all,

I've had a problem for a while whereby bacula hangs waiting for storage.

Here's the message I posted to the bacula-users list previously.
http://marc.info/?l=bacula-users&m=123004380923706&w=2

Yesterday I upgraded to 2.4.4 and I think I've still got that problem--bacula 
still stops processing backups, but the message output is different.  Here's 
the 
last messages from the console this time.

05-Jan 18:28 mxg86: VSS Writer (BackupComplete): "WMI Writer", State: 0x1 
(VSS_WS_STABLE)
05-Jan 18:28 mxg86: VSS Writer (BackupComplete): "MSDEWriter", State: 0x1 
(VSS_WS_STABLE)
05-Jan 18:28 mxg86: VSS Writer (BackupComplete): "Microsoft Writer (Bootable 
State)", State: 0x1 (VSS_WS_STABLE)
05-Jan 18:28 mxg86: VSS Writer (BackupComplete): "Microsoft Writer (Service 
State)", State: 0x1 (VSS_WS_STABLE)
05-Jan 18:28 gyrus-sd JobId 53263: Job write elapsed time = 01:25:13, Transfer 
rate = 1.977 M bytes/second
05-Jan 18:28 gyrus-sd JobId 53263: Committing spooled data to Volume 
"LTO295L2". 
Despooling 10,120,678,805 bytes ...
05-Jan 19:30 gyrus-dir JobId 53310: Prior failed job found in catalog. 
Upgrading 
to Full.
05-Jan 19:30 gyrus-dir JobId 53311: Prior failed job found in catalog. 
Upgrading 
to Full.
05-Jan 19:30 gyrus-dir JobId 53313: Prior failed job found in catalog. 
Upgrading 
to Full.
05-Jan 19:30 gyrus-dir JobId 53323: Prior failed job found in catalog. 
Upgrading 
to Full.
06-Jan 08:55 gyrus-dir JobId 53280: Fatal error: Network error with FD during 
Backup: ERR=Connection timed out
06-Jan 08:55 gyrus-sd JobId 53280: Job regcomm-gx280.2009-01-05_11.53.02.12 
marked to be canceled.
06-Jan 08:55 gyrus-dir JobId 53280: Fatal error: No Job status returned from FD.
06-Jan 08:55 gyrus-dir JobId 53280: Error: Bacula gyrus-dir 2.4.4 (28Dec08): 
06-Jan-2009 08:55:07
   Build OS:               i686-pc-linux-gnu suse 10.2
   JobId:                  53280
   Job:                    regcomm-gx280.2009-01-05_11.53.02.12
   Backup Level:           Incremental, since=2009-01-04 12:00:41
   Client:                 "regcomm-gx280" 2.4.0 (04Jun08) 
Linux,Cross-compile,Win32
   FileSet:                "cd-drive-dirs" 2007-11-05 19:00:00
   Pool:                   "Default" (From Job resource)
   Storage:                "Dell-PV136T" (From Job resource)
   Scheduled time:         05-Jan-2009 11:53:02
   Start time:             05-Jan-2009 17:13:48
   End time:               06-Jan-2009 08:55:07
   Elapsed time:           15 hours 41 mins 19 secs
   Priority:               15
   FD Files Written:       0
   SD Files Written:       0
   FD Bytes Written:       0 (0 B)
   SD Bytes Written:       0 (0 B)
   Rate:                   0.0 KB/s
   Software Compression:   None
   VSS:                    no
   Storage Encryption:     no
   Volume name(s):
   Volume Session Id:      92
   Volume Session Time:    1231165805
   Last Volume Bytes:      281,632,167,936 (281.6 GB)
   Non-fatal FD errors:    0
   SD Errors:              0
   FD termination status:  Error
   SD termination status:  Error
   Termination:            *** Backup Error ***


I then ran a status director and saw that jobs have been stalled for more than 
12 hours.
Upon running a status storage, the bconsole program became unresponsive.
*status storage
Automatically selected Storage: Dell-PV136T
Connecting to Storage daemon Dell-PV136T at gyrus:9103

gyrus-sd Version: 2.4.4 (28 December 2008) i686-pc-linux-gnu suse 10.2
Daemon started 05-Jan-09 09:30, 79 Jobs run since started.
  Heap: heap=3,608,576 smbytes=3,262,825 max_bytes=3,328,738 bufs=567 
max_bufs=569
Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8

Running Jobs:
Writing: Full Backup job krr6-d830.2009-01-05_11 JobId=53240 Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=22,666 Bytes=16,473,267,441 Bytes/sec=259,119
     FDReadSeqNo=437,933 in_msg=373469 out_msg=9 fd=29
Writing: Incremental Backup job lcc3-o755.2009-01-05_11 JobId=53243 
Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=0 Bytes=0 Bytes/sec=0
     FDReadSeqNo=6 in_msg=6 out_msg=4 fd=49
Writing: Full Backup job lsk2.2009-01-05_11 JobId=53248 Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=1
     Files=51,118 Bytes=6,367,622,723 Bytes/sec=107,850
     FDReadSeqNo=528,490 in_msg=378838 out_msg=9 fd=50
Writing: Full Backup job mes179-d630.2009-01-05_11 JobId=53254 Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=0 Bytes=0 Bytes/sec=0
     FDReadSeqNo=6 in_msg=6 out_msg=5 fd=16
Writing: Full Backup job mje42-gx280.2009-01-05_11 JobId=53256 Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=1
     Files=26,703 Bytes=4,115,465,652 Bytes/sec=69,705
     FDReadSeqNo=285,444 in_msg=209770 out_msg=9 fd=23
Writing: Full Backup job mlc10-d830.2009-01-05_11 JobId=53257 Volume="LTO295L2"
     pool="Default" device="IBMLTO2-2" (/dev/nst1)
     spooling=0 despooling=0 despool_wait=0
     Files=4,705 Bytes=7,970,089,786 Bytes/sec=135,003
     FDReadSeqNo=158,333 in_msg=145785 out_msg=5 fd=6
Writing: Full Backup job mxg86.2009-01-05_11 JobId=53263 Volume="LTO295L2"
     pool="Default" device="IBMLTO2-2" (/dev/nst1)
     spooling=0 despooling=0 despool_wait=1
     Files=48,131 Bytes=10,109,816,521 Bytes/sec=171,248
     FDReadSeqNo=557,760 in_msg=417784 out_msg=9 fd=27
Writing: Full Backup job nma9.2009-01-05_11 JobId=53267 Volume="LTO295L2"
     pool="Default" device="IBMLTO2-2" (/dev/nst1)
     spooling=0 despooling=0 despool_wait=0
     Files=8,203 Bytes=8,329,933,955 Bytes/sec=141,283
     FDReadSeqNo=194,492 in_msg=171633 out_msg=5 fd=37
Writing: Full Backup job ody.2009-01-05_11 JobId=53271 Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=0 Bytes=0 Bytes/sec=0
     FDReadSeqNo=6 in_msg=6 out_msg=4 fd=41
Writing: Full Backup job pyromancer1.2009-01-05_11 JobId=53276 Volume="LTO295L2"
     pool="Default" device="IBMLTO2-2" (/dev/nst1)
     spooling=0 despooling=0 despool_wait=1
     Files=61,700 Bytes=33,704,216,119 Bytes/sec=573,718
     FDReadSeqNo=1,037,077 in_msg=857720 out_msg=9 fd=12
Writing: Full Backup job rbd2-gx280.2009-01-05_11 JobId=53279 Volume="LTO295L2"
     pool="Default" device="IBMLTO2-2" (/dev/nst1)
     spooling=0 despooling=0 despool_wait=1
     Files=37,744 Bytes=8,917,776,587 Bytes/sec=151,820
     FDReadSeqNo=457,402 in_msg=346240 out_msg=9 fd=43
Writing: Incremental Backup job regcomm-gx280.2009-01-05_11 JobId=53280 
Volume="LTO298L2"
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=0 Bytes=0 Bytes/sec=0
     FDReadSeqNo=6 in_msg=6 out_msg=5 fd=46
Writing: Incremental Backup job rxm23.2009-01-05_11 JobId=53285 Volume=""
     pool="Default" device="IBMLTO2-1" (/dev/nst0)
     spooling=0 despooling=0 despool_wait=0
     Files=0 Bytes=0 Bytes/sec=0
     FDSocket closed
====

Jobs waiting to reserve a drive:
    3605 JobId=53285 wants free drive but device "IBMLTO2-1" (/dev/nst0) is 
busy.
====

Terminated Jobs:
  JobId  Level    Files      Bytes   Status   Finished        Name
===================================================================
  53230  Full     24,670    3.577 G  OK       05-Jan-09 16:44 
jxc37-gx280.2009-01-05_11
  53242  Full     13,891    7.358 G  OK       05-Jan-09 16:48 
kxs45-o755.2009-01-05_11
  53219  Full     24,002    16.59 G  OK       05-Jan-09 16:59 
jc-gx620.2009-01-05_11
  53251  Incr         67    1.157 G  OK       05-Jan-09 16:59 
lxm12.2009-01-05_11
  53237  Full     60,759    20.41 G  OK       05-Jan-09 17:02 
klh16-o755.2009-01-05_11
  53252  Incr         68    9.517 M  OK       05-Jan-09 17:02 
marlon.2009-01-05_11
  53250  Full     21,339    4.143 G  OK       05-Jan-09 17:03 
lxh68-gx620.2009-01-05_11
  53262  Incr         61    204.7 M  OK       05-Jan-09 17:04 
msj2-gx260.2009-01-05_11
  53273  Incr        101    323.2 M  OK       05-Jan-09 17:07 
plp5-o755.2009-01-05_11
  53258  Incr      1,147    107.3 M  OK       05-Jan-09 17:07 
mls7-gx620.2009-01-05_11
====

Device status:
Autochanger "Dell-PV136T" with devices:
    "IBMLTO2-1" (/dev/nst0)
    "IBMLTO2-2" (/dev/nst1)
Device "IBMLTO2-1" (/dev/nst0) is mounted with:
     Volume:      LTO298L2
     Pool:        Default
     Media type:  LTO-2
     Slot 43 is loaded in drive 0.
     Total Bytes=282,273,481,728 Blocks=4,375,518 Bytes/block=64,512
     Positioned at File=284 Block=9,941
Device "IBMLTO2-2" (/dev/nst1) is mounted with:
     Volume:      LTO295L2
     Pool:        Default
     Media type:  LTO-2
     Slot 40 is loaded in drive 1.
     Total Bytes=87,572,201,472 Blocks=1,357,455 Bytes/block=64,512
     Positioned at File=87 Block=8,956
====

Used Volume status:

[output ends suddenly here before getting back to a * prompt]

I can leave it like this for the next couple hours just in case somebody has 
any 
ideas for something else they'd like me to try or send to assist in solving 
this 
problem.

Here's the traceback file...

Using host libthread_db library "/lib/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread -1214118192 (LWP 21853)]
[New Thread -1426953328 (LWP 26587)]
[New Thread -1393382512 (LWP 26575)]
[New Thread -1367471216 (LWP 26496)]
[New Thread -1401775216 (LWP 26488)]
[New Thread -1435346032 (LWP 26405)]
[New Thread -1375863920 (LWP 26262)]
[New Thread -1384256624 (LWP 26044)]
[New Thread -1418560624 (LWP 25949)]
[New Thread -1341043824 (LWP 25672)]
[New Thread -1332544624 (LWP 22749)]
[New Thread -1324143728 (LWP 22748)]
[New Thread -1315751024 (LWP 22747)]
[New Thread -1307350128 (LWP 22746)]
[New Thread -1298949232 (LWP 22745)]
[New Thread -1290556528 (LWP 22744)]
[New Thread -1282159728 (LWP 22743)]
[New Thread -1273767024 (LWP 22742)]
[New Thread -1265374320 (LWP 22741)]
[New Thread -1256977520 (LWP 22740)]
[New Thread -1248584816 (LWP 22737)]
[New Thread -1231582320 (LWP 22736)]
[New Thread -1239975024 (LWP 22223)]
[New Thread -1223189616 (LWP 21856)]
[New Thread -1214796912 (LWP 21855)]
0xb7fb7410 in __kernel_vsyscall ()
$1 = "gyrus-dir", '\0' <repeats 20 times>
$2 = 0x80f8e00 "bacula-dir"
$3 = 0x80f90b0 "/usr/sbin/"
$4 = 0x80f91e0 "MySQL"
$5 = 0x80ed38b "2.4.4 (28 December 2008)"
$6 = 0x80ed3a4 "i686-pc-linux-gnu"
$7 = 0x80ed3b6 "suse"
$8 = 0x80ed3bb "10.2"
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc0876 in __nanosleep_nocancel () from /lib/libpthread.so.0
#2  0x080a8347 in bmicrosleep (sec=60, usec=0) at bsys.c:71
#3  0x08071be4 in wait_for_next_job (one_shot_job_to_run=0x0) at scheduler.c:130
#4  0x0804de85 in main (argc=0, argv=0xbfa58da4) at dird.c:288

Thread 25 (Thread -1214796912 (LWP 21855)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7b28a41 in ___newselect_nocancel () from /lib/libc.so.6
#2  0x080a9a85 in bnet_thread_server (addrs=0x80f9860, max_clients=20, 
client_wq=0x80f64e0,
     handle_client_request=0x808d5fe <handle_UA_client_request>) at 
bnet_server.c:161
#3  0x0808d5f6 in connect_thread (arg=0x80f9860) at ua_server.c:84
#4  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#5  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 24 (Thread -1223189616 (LWP 21856)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x080ccff3 in watchdog_thread (arg=0x0) at watchdog.c:307
#3  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#4  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 23 (Thread -1239975024 (LWP 22223)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x82d3b28, ptr=0xb61771b4 
";\r\016\b�q\027�\217#", nbytes=4) at bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x82d3b28) at bsock.c:381
#4  0x080a90ed in bnet_recv (bsock=0x82d3b28) at bnet.c:187
#5  0x0808efef in do_storage_status (ua=0x8172fc0, store=0x823b5a0) at 
ua_status.c:325
#6  0x0808f776 in status_cmd (ua=0x8172fc0, cmd=0x8172c98 "status storage") at 
ua_status.c:134
#7  0x08076e8c in do_a_command (ua=0x8172fc0, cmd=0x8172c98 "status storage") 
at 
ua_cmds.c:180
#8  0x0808d70f in handle_UA_client_request (arg=0x816e4c8) at ua_server.c:147
#9  0x080cda1e in workq_server (arg=0x80f64e0) at workq.c:357
#10 0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#11 0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 22 (Thread -1231582320 (LWP 22736)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0x8302638) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0x8302638) at backup.c:304
#4  0x08051abd in do_backup (jcr=0x8302638) at backup.c:235
#5  0x08063e29 in job_thread (arg=0x8302638) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 21 (Thread -1248584816 (LWP 22737)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0x832a4c8) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0x832a4c8) at backup.c:304
#4  0x08051abd in do_backup (jcr=0x832a4c8) at backup.c:235
#5  0x08063e29 in job_thread (arg=0x832a4c8) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 20 (Thread -1256977520 (LWP 22740)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x82704b8, ptr=0xb513ff44 "��˷(", 
nbytes=4) 
at bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x82704b8) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x82704b8) at getmsg.c:109
#5  0x0806c14b in start_storage_daemon_job (jcr=0xaf819f50, rstore=0x0, 
wstore=0xaf81ae00) at msgchan.c:285
#6  0x080517e3 in do_backup (jcr=0xaf819f50) at backup.c:142
#7  0x08063e29 in job_thread (arg=0xaf819f50) at job.c:290
#8  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#9  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#10 0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 19 (Thread -1265374320 (LWP 22741)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0xaf80f3e0) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0xaf80f3e0) at backup.c:304
#4  0x08051abd in do_backup (jcr=0xaf80f3e0) at backup.c:235
#5  0x08063e29 in job_thread (arg=0xaf80f3e0) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 18 (Thread -1273767024 (LWP 22742)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0x8310518) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0x8310518) at backup.c:304
#4  0x08051abd in do_backup (jcr=0x8310518) at backup.c:235
#5  0x08063e29 in job_thread (arg=0x8310518) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 17 (Thread -1282159728 (LWP 22743)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0xaf842440,
     ptr=0xb393bf44 
"��˷(�\223�h.3\b�*3\b\030\0222\b��\223�\201�\005\b@$\204���˷H�\223�|\222\n\b@$\204�\224�1\b\004",
 

     nbytes=4) at bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0xaf842440) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0xaf842440) at getmsg.c:109
#5  0x0806c14b in start_storage_daemon_job (jcr=0xaf81b7b0, rstore=0x0, 
wstore=0xaf81c660) at msgchan.c:285
#6  0x080517e3 in do_backup (jcr=0xaf81b7b0) at backup.c:142
#7  0x08063e29 in job_thread (arg=0xaf81b7b0) at job.c:290
#8  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#9  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#10 0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 16 (Thread -1290556528 (LWP 22744)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0x8335af0) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0x8335af0) at backup.c:304
#4  0x08051abd in do_backup (jcr=0x8335af0) at backup.c:235
#5  0x08063e29 in job_thread (arg=0x8335af0) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 15 (Thread -1298949232 (LWP 22745)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x827e880, ptr=0xb2939024 
"`\231\017\b\202", nbytes=4) at bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x827e880) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x827e880) at getmsg.c:109
#5  0x080513f8 in wait_for_job_termination (jcr=0xaf8016d0) at backup.c:274
#6  0x08051abd in do_backup (jcr=0xaf8016d0) at backup.c:235
#7  0x08063e29 in job_thread (arg=0xaf8016d0) at job.c:290
#8  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#9  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#10 0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 14 (Thread -1307350128 (LWP 22746)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0xaf809e38) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0xaf809e38) at backup.c:304
#4  0x08051abd in do_backup (jcr=0xaf809e38) at backup.c:235
#5  0x08063e29 in job_thread (arg=0xaf809e38) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 13 (Thread -1315751024 (LWP 22747)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc0876 in __nanosleep_nocancel () from /lib/libpthread.so.0
#2  0x080a8347 in bmicrosleep (sec=2, usec=0) at bsys.c:71
#3  0x08066122 in jobq_server (arg=0x80f6340) at jobq.c:590
#4  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#5  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 12 (Thread -1324143728 (LWP 22748)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cbd7dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x0806b094 in wait_for_storage_daemon_termination (jcr=0x831e3b8) at 
msgchan.c:409
#3  0x080514e2 in wait_for_job_termination (jcr=0x831e3b8) at backup.c:304
#4  0x08051abd in do_backup (jcr=0x831e3b8) at backup.c:235
#5  0x08063e29 in job_thread (arg=0x831e3b8) at job.c:290
#6  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#7  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#8  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 11 (Thread -1332544624 (LWP 22749)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0xaf8425b8, ptr=0xb092f024 
"`\231\017\b\202", nbytes=4) at bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0xaf8425b8) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0xaf8425b8) at getmsg.c:109
#5  0x080513f8 in wait_for_job_termination (jcr=0x8307878) at backup.c:274
#6  0x08051abd in do_backup (jcr=0x8307878) at backup.c:235
#7  0x08063e29 in job_thread (arg=0x8307878) at job.c:290
#8  0x08065a5f in jobq_server (arg=0x80f6340) at jobq.c:466
#9  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#10 0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 10 (Thread -1341043824 (LWP 25672)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x8334e10, ptr=0xb0114114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x8334e10) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x8334e10) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x8302638) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 9 (Thread -1418560624 (LWP 25949)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0xaf843388, ptr=0xab727114 "", nbytes=4) 
at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0xaf843388) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0xaf843388) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x8307878) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 8 (Thread -1384256624 (LWP 26044)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x816acb8, ptr=0xad7de114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x816acb8) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x816acb8) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x8310518) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 7 (Thread -1375863920 (LWP 26262)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0xaf801128, ptr=0xadfdf114 "", nbytes=4) 
at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0xaf801128) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0xaf801128) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x831e3b8) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 6 (Thread -1435346032 (LWP 26405)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x82935e0, ptr=0xaa725114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x82935e0) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x82935e0) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x832a4c8) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 5 (Thread -1401775216 (LWP 26488)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0xaf843798, ptr=0xac729114 "", nbytes=4) 
at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0xaf843798) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0xaf843798) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0x8335af0) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 4 (Thread -1367471216 (LWP 26496)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x82cc080, ptr=0xae7e0114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x82cc080) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x82cc080) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0xaf8016d0) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 3 (Thread -1393382512 (LWP 26575)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x8186008, ptr=0xacf2a114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x8186008) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x8186008) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0xaf809e38) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 2 (Thread -1426953328 (LWP 26587)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc002b in __read_nocancel () from /lib/libpthread.so.0
#2  0x080a93cc in read_nbytes (bsock=0x82a0090, ptr=0xaaf26114 "", nbytes=4) at 
bnet.c:82
#3  0x080abae8 in BSOCK::recv (this=0x82a0090) at bsock.c:381
#4  0x0805de81 in bget_dirmsg (bs=0x82a0090) at getmsg.c:109
#5  0x0806b3b1 in msg_thread (arg=0xaf80f3e0) at msgchan.c:374
#6  0xb7cb9112 in start_thread () from /lib/libpthread.so.0
#7  0xb7b2f2ee in clone () from /lib/libc.so.6

Thread 1 (Thread -1214118192 (LWP 21853)):
#0  0xb7fb7410 in __kernel_vsyscall ()
#1  0xb7cc0876 in __nanosleep_nocancel () from /lib/libpthread.so.0
#2  0x080a8347 in bmicrosleep (sec=60, usec=0) at bsys.c:71
#3  0x08071be4 in wait_for_next_job (one_shot_job_to_run=0x0) at scheduler.c:130
#4  0x0804de85 in main (argc=0, argv=0xbfa58da4) at dird.c:288
#0  0xb7fb7410 in __kernel_vsyscall ()
#0  0xb7fb7410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7cc0876 in __nanosleep_nocancel () from /lib/libpthread.so.0
No symbol table info available.
#2  0x080a8347 in bmicrosleep (sec=60, usec=0) at bsys.c:71
71         stat = nanosleep(&timeout, NULL);
Current language:  auto; currently c++
timeout = {tv_sec = 60, tv_nsec = 0}
tv = {tv_sec = 0, tv_usec = 2}
tz = {tz_minuteswest = 2, tz_dsttime = 0}
stat = 0
#3  0x08071be4 in wait_for_next_job (one_shot_job_to_run=0x0) at scheduler.c:130
130           bmicrosleep(next_check_secs, 0); /* recheck once per minute */
jcr = (JCR *) 0xaf85cef8
job = (JOB *) 0x820acb8
run = (RUN *) 0x8241a70
now = 53354
prev = 1231214400
next_job = (job_item *) 0x0
first = false
#4  0x0804de85 in main (argc=0, argv=0xbfa58da4) at dird.c:288
288        while ( (jcr = wait_for_next_job(runjob)) ) {
ch = -1
jcr = (JCR *) 0xaf85cef8
no_signals = false
test_config = false
uid = 0xbfa5a4fe "bacula"
gid = 0xbfa5a508 "bacula"
#0  0x00000000 in ?? ()
No symbol table info available.
#0  0x00000000 in ?? ()
No symbol table info available.
#0  0x00000000 in ?? ()
No symbol table info available.

------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>