Bacula-users

Re: [Bacula-users] [Bacula-devel] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive

2009-01-27 10:32:45
Subject: Re: [Bacula-users] [Bacula-devel] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive
From: Silver Salonen <silver AT ultrasoft DOT ee>
To: Kern Sibbald <kern AT sibbald DOT com>
Date: Tue, 27 Jan 2009 17:29:46 +0200
Hi.

So you're saying there was nothing wrong with bacula-sd behaviour? Or you're 
saying that I've misconfigured bacula-sd somehow (eg. too many devices or 
smth), so that bacula-sd's behaviour in this case was OK?

Because to my mind I'm not asking for support, but rather notifying developers 
about a possible bug..

PS. I've included list back in discussion as only files attached had some 
sensitive data in them..

-- 
Silver

On Tuesday 27 January 2009 17:22:55 you wrote:
> Hello Silver,
> 
> I am sorry, but at this point, it comes back to a support request, as it 
looks 
> to me like an operational problem rather than a bug.  
> 
> If you want to continue, I can give you support through Bacula Systems, but 
it 
> will cost €3,200/day. You can get much more reasonably priced support via 
> Bacula Systems with other experienced support people at €1,200/day, or you 
> can get a support contract.  In addition, there are Bacula Systems partners 
> and other professional support companies listed on the Bacula web site.  
> Finally, as I have suggested to you several time, you can probably get quite 
> decent free support on via the bacula-users email list.
> 
> If you want to do something with my help (very expensive) or via Bacula 
> Systems (normal commercial rates), feel free to let me know.  Otherwise, I 
am 
> sorry, and good luck.
> 
> Best regards,
> 
> Kern
> 
> On Monday 26 January 2009 23:46:32 Silver Salonen wrote:
> > Hello.
> >
> > I'm contacting you privately as I'm a bit security paranoid and I'm not
> > sure if sending all my servers' names and bacula-sd log to public mailing
> > list is a good idea ;)
> >
> > Anyway, I think I've done all the suggested now ;)
> >
> > First of all.. the storage daemon is not hung.. it just won't process any
> > more jobs. When I check its status, I see:
> > ==========
> > backup-sd Version: 2.4.4 (28 December 2008) i386-portbld-freebsd6.3
> > freebsd 6.3-RELEASE-p1
> > Daemon started 25-Jan-09 18:14, 67 Jobs run since started.
> >  Heap: heap=3,817,472 smbytes=1,903,201 max_bytes=2,427,485 bufs=687
> > max_bufs=721
> > Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8
> >
> > Running Jobs:
> > Writing: Incremental Backup job aidamees-log.2009-01-26_23 JobId=40383
> > Volume="kasparov-sqldump-incr-0008"
> >     pool="pool-aidamees-log-incr" device="device-aidamees"
> > (/backup/bacula/aidamees)
> >     spooling=0 despooling=0 despool_wait=0
> >     Files=0 Bytes=0 Bytes/sec=0
> >     FDSocket closed
> > Writing: Incremental Backup job aidamees-userdata.2009-01-26_23
> > JobId=40384 Volume="aidamees-userdata-incr-0005"
> >     pool="pool-aidamees-userdata-incr" device="device-aidamees"
> > (/backup/bacula/aidamees)
> >     spooling=0 despooling=0 despool_wait=0
> >     Files=0 Bytes=0 Bytes/sec=0
> >     FDReadSeqNo=6 in_msg=6 out_msg=4 fd=18
> > ...
> > [4 more such entries]
> > ...
> > ====
> >
> > Jobs waiting to reserve a drive:
> >    3608 JobId=40383 wants Pool="pool-aidamees-log-incr" but have
> > Pool="pool-aidamees-userdata-incr" nreserve=1 on drive
> > "device-aidamees" (/backup/bacula/aidamees).
> >    3608 JobId=40382 wants Pool="pool-aidamees-conf-incr" but have
> > Pool="pool-aidamees-userdata-incr" nreserve=1 on drive
> > "device-aidamees" (/backup/bacula/aidamees).
> >    3608 JobId=40389 wants Pool="pool-columbus-coltar-log-full" but have
> > Pool="pool-columbus-coltar-conf-incr" nreserve=1 on drive
> > "device-columbus-coltar" (/backup/bacula/columbus-coltar).
> >    3608 JobId=40393 wants Pool="pool-knoke-gw-log-incr" but have
> > Pool="pool-knoke-gw-conf-incr" nreserve=1 on drive "device-knoke-gw"
> > (/backup/bacula/knoke-gw).
> > ====
> >
> > Terminated Jobs:
> > ...
> > ====
> >
> > Device status:
> > Device "device-default" (/backup/bacula) is not open.
> > Configured device capabilities:
> > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS ALWAYSOPEN
> > Device state:
> > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL !SHORT
> > !MOUNTED
> > num_writers=0 block=0
> >
> > Device parameters:
> > Archive name: /backup/bacula Device name: device-default
> > File=0 block=0
> > Min block=0 Max block=0
> > Device "device-aidamees" (/backup/bacula/aidamees) is not open.
> >     Device is being initialized.
> > Configured device capabilities:
> > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS ALWAYSOPEN
> > Device state:
> > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL !SHORT
> > !MOUNTED
> > num_writers=0 block=3
> >
> > Device parameters:
> > Archive name: /backup/bacula/aidamees Device name: device-aidamees
> > File=0 block=0
> > Min block=0 Max block=0
> > Device "device-aidamees-webdav" (/backup/bacula/aidamees) is not open.
> > Configured device capabilities:
> > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS ALWAYSOPEN
> > Device state:
> > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL !SHORT
> > !MOUNTED
> > num_writers=0 block=0
> > ...
> > [32 more such entries]
> >
> > Device parameters:
> > Archive name: /backup/bacula/varukas Device name: device-varukas
> > File=0 block=0
> > Min block=0 Max block=0
> > ====
> >
> > Used Volume status:
> > ==========
> >
> > That's it.. so it just keeps sitting on this "Used Volume status".
> >
> > Bacula-sd output and gdb log is attached..
> >
> > --
> > Silver
> >
> > On Sun, January 25, 2009 00:43, Kern Sibbald wrote:
> > > On Saturday 24 January 2009 23:18:44 Silver Salonen wrote:
> > >> Anyway, I compiled bacula-sd with --enable-smartalloc and ran in the
> > >> correct way. Is the attached traceback more usable?
> > >
> > > Yes, you are making good progress.  The dump is much better but still
> > > lacks important information.  I suggest two things:
> > >
> > > 1. Make sure you compile with the -g option.  You might check to
> > > ensure that:
> > >
> > > #define DEVELOPER 1
> > >
> > >
> > > is defined in <bacula-source>/src/version.h
> > >
> > > Then completely rebuild the code:
> > >
> > >
> > > make clean make make install
> > >
> > > Then instead of sending a signal to the SD, when you are *sure* it is
> > > hung, while running it under the debugger, simply enter a ctl-c in the
> > > debugger shell window and then the rest of the commands.
> > >
> > > Hopefully, that will get a complete traceback.
> > >
> > >
> > > Finally, you should use -d 100 on the execution line and direct the
> > > output of Bacula to a file so that in addition to the traceback we have 
a
> > > debug listing.
> > >
> > > Regards,
> > >
> > >
> > > Kern
> > >
> > >> --
> > >> Silver
> > >>
> > >> On Sat, January 24, 2009 21:19, Silver Salonen wrote:
> > >>> OK, I'll try.. but what does "built with debug information turned on
> > >>> and not stripped of debugging symbols" mean? The only thing I found
> > >>> about debugging in configure script was --enable-smartalloc - that's
> > >>> it?
> > >>>
> > >>> --
> > >>> Silver
> > >>>
> > >>> On Sat, January 24, 2009 19:29, Kern Sibbald wrote:
> > >>>> Hello,
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> Sorry, but the backtrace is not usable.  Please make sure you have
> > >>>> build the SD with the debug symbols left in (i.e. do not strip it).
> > >>>> I
> > >>>> suggest you read the Kaboom chapter of the manual that explains how
> > >>>> to get a backtrace.
> > >>>>
> > >>>> Kern
> > >>>>
> > >>>> On Saturday 24 January 2009 17:54:05 Silver Salonen wrote:
> > >>>>> Hi.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> It seems I'm experiencing the same problem on FreeBSD 6.3. I ran
> > >>>>> bacula-sd in gdb and when the backups started running, a few of
> > >>>>> them ran and completed successfully, but stayed in "terminated"
> > >>>>> status afterwards. Other jobs just didn't start running. When I
> > >>>>> sent just ordinary kill to the process, gdb said the program
> > >>>>> terminated. The output of gdb:
> > >>>>>
> > >>>>> (gdb) run -f -c /usr/local/etc/bacula-sd.conf
> > >>>>> Starting program: /usr/local/sbin/bacula-sd -f -c
> > >>>>> /usr/local/etc/bacula-
> > >>>>> sd.conf (no debugging symbols found)...(no debugging symbols
> > >>>>> found)...warning:
> > >>>>> Unable to get location for thread creation breakpoint: generic
> > >>>>> error [New LWP 100405]
> > >>>>> (no debugging symbols found)...(no debugging symbols found)...(no
> > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > >>>>> debugging symbols found)...[New Thread 0x80c0200 (LWP 100057)]
> > >>>>>
> > >>>>>
> > >>>>> Program received signal SIGTERM, Terminated.
> > >>>>> [Switching to Thread 0x80c0200 (LWP 100057)]
> > >>>>> 0x281075db in pthread_testcancel () from /lib/libpthread.so.2
> > >>>>> (gdb) backtrace
> > >>>>> #0  0x281075db in pthread_testcancel () from /lib/libpthread.so.2
> > >>>>> #1  0x280f4c25 in sigaction () from /lib/libpthread.so.2
> > >>>>> #2  0x280f4f11 in sigaction () from /lib/libpthread.so.2
> > >>>>> #3  0x280f56f0 in sigaction () from /lib/libpthread.so.2
> > >>>>> #4  0x280f589c in sigaction () from /lib/libpthread.so.2
> > >>>>> #5  0x280ffeec in pthread_mutexattr_init () from
> > >>>>> /lib/libpthread.so.2
> > >>>>> #6  0x280d8450 in ?? ()
> > >>>>> (gdb) quit
> > >>>>> The program is running.  Exit anyway? (y or n) y
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> PS. Sorry if I used gdb incorrectly, I'm not very experienced
> > >>>>> with it.. let me know what to do better next time ;)

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>