Bacula-users

Re: [Bacula-users] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive

2009-02-05 06:09:00
Subject: Re: [Bacula-users] bacula hang issue. was: bacula sometimes gets stuck when volume wanted is already in a different drive
From: Kern Sibbald <kern AT sibbald DOT com>
To: Silver Salonen <silver AT ultrasoft DOT ee>
Date: Thu, 5 Feb 2009 12:07:46 +0100
Hello,

The problem is very clear: as much as I would like, I am not able to provide 
support any more.  I have already said that.  In addition the Bacula users 
list which I monitor from time to time has some extremely qualified users who 
are providing impressive and excellent free support.

I think my response below on the various ways that you can get support was 
also clear.  If this is indeed a bug, then it will need clear proof, and it 
should be filed as a bug report.  However, with the current information, I do 
not consider it a bug, so without some new clear proof (job reports, valid 
tracebacks with debugging symbols, debug output from the SD), we will have no 
choice bug to close any bug report.

I think that all of this is and has been stated very clearly for some time, so 
I would appreciate it if you would stop emailing me directly and instead use 
the mechanisms that the Bacula project has setup for helping people or you 
should seek professional help.

Regards,

Kern

On Thursday 05 February 2009 09:17:55 Silver Salonen wrote:
> Hello.
>
> Can you please tell in what way was my problem a support request?


On Tuesday 27 January 2009 17:29:46 Silver Salonen wrote:
> Hi.
> 
> So you're saying there was nothing wrong with bacula-sd behaviour? Or you're 
> saying that I've misconfigured bacula-sd somehow (eg. too many devices or 
> smth), so that bacula-sd's behaviour in this case was OK?
> 
> Because to my mind I'm not asking for support, but rather notifying 
developers 
> about a possible bug..
> 
> PS. I've included list back in discussion as only files attached had some 
> sensitive data in them..
> 
> -- 
> Silver
> 
> On Tuesday 27 January 2009 17:22:55 you wrote:
> > Hello Silver,
> > 
> > I am sorry, but at this point, it comes back to a support request, as it 
> looks 
> > to me like an operational problem rather than a bug.  
> > 
> > If you want to continue, I can give you support through Bacula Systems, 
but 
> it 
> > will cost €3,200/day. You can get much more reasonably priced support via 
> > Bacula Systems with other experienced support people at €1,200/day, or you 
> > can get a support contract.  In addition, there are Bacula Systems 
partners 
> > and other professional support companies listed on the Bacula web site.  
> > Finally, as I have suggested to you several time, you can probably get 
quite 
> > decent free support on via the bacula-users email list.
> > 
> > If you want to do something with my help (very expensive) or via Bacula 
> > Systems (normal commercial rates), feel free to let me know.  Otherwise, I 
> am 
> > sorry, and good luck.
> > 
> > Best regards,
> > 
> > Kern
> > 
> > On Monday 26 January 2009 23:46:32 Silver Salonen wrote:
> > > Hello.
> > >
> > > I'm contacting you privately as I'm a bit security paranoid and I'm not
> > > sure if sending all my servers' names and bacula-sd log to public 
mailing
> > > list is a good idea ;)
> > >
> > > Anyway, I think I've done all the suggested now ;)
> > >
> > > First of all.. the storage daemon is not hung.. it just won't process 
any
> > > more jobs. When I check its status, I see:
> > > ==========
> > > backup-sd Version: 2.4.4 (28 December 2008) i386-portbld-freebsd6.3
> > > freebsd 6.3-RELEASE-p1
> > > Daemon started 25-Jan-09 18:14, 67 Jobs run since started.
> > >  Heap: heap=3,817,472 smbytes=1,903,201 max_bytes=2,427,485 bufs=687
> > > max_bufs=721
> > > Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8
> > >
> > > Running Jobs:
> > > Writing: Incremental Backup job aidamees-log.2009-01-26_23 JobId=40383
> > > Volume="kasparov-sqldump-incr-0008"
> > >     pool="pool-aidamees-log-incr" device="device-aidamees"
> > > (/backup/bacula/aidamees)
> > >     spooling=0 despooling=0 despool_wait=0
> > >     Files=0 Bytes=0 Bytes/sec=0
> > >     FDSocket closed
> > > Writing: Incremental Backup job aidamees-userdata.2009-01-26_23
> > > JobId=40384 Volume="aidamees-userdata-incr-0005"
> > >     pool="pool-aidamees-userdata-incr" device="device-aidamees"
> > > (/backup/bacula/aidamees)
> > >     spooling=0 despooling=0 despool_wait=0
> > >     Files=0 Bytes=0 Bytes/sec=0
> > >     FDReadSeqNo=6 in_msg=6 out_msg=4 fd=18
> > > ...
> > > [4 more such entries]
> > > ...
> > > ====
> > >
> > > Jobs waiting to reserve a drive:
> > >    3608 JobId=40383 wants Pool="pool-aidamees-log-incr" but have
> > > Pool="pool-aidamees-userdata-incr" nreserve=1 on drive
> > > "device-aidamees" (/backup/bacula/aidamees).
> > >    3608 JobId=40382 wants Pool="pool-aidamees-conf-incr" but have
> > > Pool="pool-aidamees-userdata-incr" nreserve=1 on drive
> > > "device-aidamees" (/backup/bacula/aidamees).
> > >    3608 JobId=40389 wants Pool="pool-columbus-coltar-log-full" but have
> > > Pool="pool-columbus-coltar-conf-incr" nreserve=1 on drive
> > > "device-columbus-coltar" (/backup/bacula/columbus-coltar).
> > >    3608 JobId=40393 wants Pool="pool-knoke-gw-log-incr" but have
> > > Pool="pool-knoke-gw-conf-incr" nreserve=1 on drive "device-knoke-gw"
> > > (/backup/bacula/knoke-gw).
> > > ====
> > >
> > > Terminated Jobs:
> > > ...
> > > ====
> > >
> > > Device status:
> > > Device "device-default" (/backup/bacula) is not open.
> > > Configured device capabilities:
> > > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS 
ALWAYSOPEN
> > > Device state:
> > > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL 
!SHORT
> > > !MOUNTED
> > > num_writers=0 block=0
> > >
> > > Device parameters:
> > > Archive name: /backup/bacula Device name: device-default
> > > File=0 block=0
> > > Min block=0 Max block=0
> > > Device "device-aidamees" (/backup/bacula/aidamees) is not open.
> > >     Device is being initialized.
> > > Configured device capabilities:
> > > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS 
ALWAYSOPEN
> > > Device state:
> > > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL 
!SHORT
> > > !MOUNTED
> > > num_writers=0 block=3
> > >
> > > Device parameters:
> > > Archive name: /backup/bacula/aidamees Device name: device-aidamees
> > > File=0 block=0
> > > Min block=0 Max block=0
> > > Device "device-aidamees-webdav" (/backup/bacula/aidamees) is not open.
> > > Configured device capabilities:
> > > EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS 
ALWAYSOPEN
> > > Device state:
> > > !OPENED !TAPE !LABEL !MALLOC !APPEND !READ !EOT !WEOT !EOF !NEXTVOL 
!SHORT
> > > !MOUNTED
> > > num_writers=0 block=0
> > > ...
> > > [32 more such entries]
> > >
> > > Device parameters:
> > > Archive name: /backup/bacula/varukas Device name: device-varukas
> > > File=0 block=0
> > > Min block=0 Max block=0
> > > ====
> > >
> > > Used Volume status:
> > > ==========
> > >
> > > That's it.. so it just keeps sitting on this "Used Volume status".
> > >
> > > Bacula-sd output and gdb log is attached..
> > >
> > > --
> > > Silver
> > >
> > > On Sun, January 25, 2009 00:43, Kern Sibbald wrote:
> > > > On Saturday 24 January 2009 23:18:44 Silver Salonen wrote:
> > > >> Anyway, I compiled bacula-sd with --enable-smartalloc and ran in the
> > > >> correct way. Is the attached traceback more usable?
> > > >
> > > > Yes, you are making good progress.  The dump is much better but still
> > > > lacks important information.  I suggest two things:
> > > >
> > > > 1. Make sure you compile with the -g option.  You might check to
> > > > ensure that:
> > > >
> > > > #define DEVELOPER 1
> > > >
> > > >
> > > > is defined in <bacula-source>/src/version.h
> > > >
> > > > Then completely rebuild the code:
> > > >
> > > >
> > > > make clean make make install
> > > >
> > > > Then instead of sending a signal to the SD, when you are *sure* it is
> > > > hung, while running it under the debugger, simply enter a ctl-c in the
> > > > debugger shell window and then the rest of the commands.
> > > >
> > > > Hopefully, that will get a complete traceback.
> > > >
> > > >
> > > > Finally, you should use -d 100 on the execution line and direct the
> > > > output of Bacula to a file so that in addition to the traceback we 
have 
> a
> > > > debug listing.
> > > >
> > > > Regards,
> > > >
> > > >
> > > > Kern
> > > >
> > > >> --
> > > >> Silver
> > > >>
> > > >> On Sat, January 24, 2009 21:19, Silver Salonen wrote:
> > > >>> OK, I'll try.. but what does "built with debug information turned on
> > > >>> and not stripped of debugging symbols" mean? The only thing I found
> > > >>> about debugging in configure script was --enable-smartalloc - that's
> > > >>> it?
> > > >>>
> > > >>> --
> > > >>> Silver
> > > >>>
> > > >>> On Sat, January 24, 2009 19:29, Kern Sibbald wrote:
> > > >>>> Hello,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Sorry, but the backtrace is not usable.  Please make sure you have
> > > >>>> build the SD with the debug symbols left in (i.e. do not strip it).
> > > >>>> I
> > > >>>> suggest you read the Kaboom chapter of the manual that explains how
> > > >>>> to get a backtrace.
> > > >>>>
> > > >>>> Kern
> > > >>>>
> > > >>>> On Saturday 24 January 2009 17:54:05 Silver Salonen wrote:
> > > >>>>> Hi.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> It seems I'm experiencing the same problem on FreeBSD 6.3. I ran
> > > >>>>> bacula-sd in gdb and when the backups started running, a few of
> > > >>>>> them ran and completed successfully, but stayed in "terminated"
> > > >>>>> status afterwards. Other jobs just didn't start running. When I
> > > >>>>> sent just ordinary kill to the process, gdb said the program
> > > >>>>> terminated. The output of gdb:
> > > >>>>>
> > > >>>>> (gdb) run -f -c /usr/local/etc/bacula-sd.conf
> > > >>>>> Starting program: /usr/local/sbin/bacula-sd -f -c
> > > >>>>> /usr/local/etc/bacula-
> > > >>>>> sd.conf (no debugging symbols found)...(no debugging symbols
> > > >>>>> found)...warning:
> > > >>>>> Unable to get location for thread creation breakpoint: generic
> > > >>>>> error [New LWP 100405]
> > > >>>>> (no debugging symbols found)...(no debugging symbols found)...(no
> > > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > > >>>>> debugging symbols found)...(no debugging symbols found)...(no
> > > >>>>> debugging symbols found)...[New Thread 0x80c0200 (LWP 100057)]
> > > >>>>>
> > > >>>>>
> > > >>>>> Program received signal SIGTERM, Terminated.
> > > >>>>> [Switching to Thread 0x80c0200 (LWP 100057)]
> > > >>>>> 0x281075db in pthread_testcancel () from /lib/libpthread.so.2
> > > >>>>> (gdb) backtrace
> > > >>>>> #0  0x281075db in pthread_testcancel () from /lib/libpthread.so.2
> > > >>>>> #1  0x280f4c25 in sigaction () from /lib/libpthread.so.2
> > > >>>>> #2  0x280f4f11 in sigaction () from /lib/libpthread.so.2
> > > >>>>> #3  0x280f56f0 in sigaction () from /lib/libpthread.so.2
> > > >>>>> #4  0x280f589c in sigaction () from /lib/libpthread.so.2
> > > >>>>> #5  0x280ffeec in pthread_mutexattr_init () from
> > > >>>>> /lib/libpthread.so.2
> > > >>>>> #6  0x280d8450 in ?? ()
> > > >>>>> (gdb) quit
> > > >>>>> The program is running.  Exit anyway? (y or n) y
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> PS. Sorry if I used gdb incorrectly, I'm not very experienced
> > > >>>>> with it.. let me know what to do better next time ;)

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users