Bacula-users

Re: [Bacula-users] bacula hang waiting for storage

2008-11-27 09:13:06
Subject: Re: [Bacula-users] bacula hang waiting for storage
From: Pasi Kärkkäinen <pasik AT iki DOT fi>
To: Arno Lehmann <al AT its-lehmann DOT de>
Date: Thu, 27 Nov 2008 16:10:34 +0200
On Thu, Nov 27, 2008 at 08:14:45AM +0100, Arno Lehmann wrote:
> Hi,
> 
> 26.11.2008 21:22, Bob Hetzel wrote:
> > I've got bacula currently in a hung state with the following interesting 
> > info.  When I run a status storage produces the following...
> 
> Is your Bacula still stuck? If so, and you have gdb installed, and a 
> Bacula with debug symbols, now might be a good time to see what it's 
> doing...
> 
> > Automatically selected Storage: Dell-PV136T
> > Connecting to Storage daemon Dell-PV136T at gyrus:9103
> > 
> > gyrus-sd Version: 2.4.3 (10 October 2008) i686-pc-linux-gnu suse 10.2
> > Daemon started 25-Nov-08 19:20, 59 Jobs run since started.
> >   Heap: heap=3,756,032 smbytes=3,519,564 max_bytes=3,684,397 bufs=555 
> > max_bufs=557
> > Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8
> > 
> > Running Jobs:
> > Writing: Incremental Backup job axh93-gx270 JobId=45634 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=1
> >      Files=78 Bytes=21,123,239 Bytes/sec=2,337
> >      FDReadSeqNo=970 in_msg=750 out_msg=9 fd=20
> > Writing: Incremental Backup job bxn4-gx280 JobId=45641 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=1
> >      Files=155 Bytes=2,925,138,595 Bytes/sec=323,648
> >      FDReadSeqNo=45,916 in_msg=45480 out_msg=9 fd=35
> > Writing: Incremental Backup job cdking JobId=45646 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=1
> >      Files=88 Bytes=11,846,912 Bytes/sec=1,310
> >      FDReadSeqNo=920 in_msg=672 out_msg=9 fd=23
> > Writing: Incremental Backup job ceg3-d810 JobId=45648 Volume="LTO253L2"
> >      pool="Default" device="IBMLTO2-2" (/dev/nst1)
> >      spooling=0 despooling=1 despool_wait=0
> >      Files=35 Bytes=1,391,695,993 Bytes/sec=176,588
> >      FDReadSeqNo=21,542 in_msg=21439 out_msg=9 fd=36
> > Writing: Incremental Backup job clifford3 JobId=45651 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=0
> >      Files=0 Bytes=0 Bytes/sec=0
> >      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=32
> > Writing: Incremental Backup job cxj57-gx270 JobId=45657 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=0
> >      Files=0 Bytes=0 Bytes/sec=0
> >      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=33
> > Writing: Incremental Backup job dxa2-d630 JobId=45665 Volume="LTO261L2"
> >      pool="Default" device="IBMLTO2-3" (/dev/nst2)
> >      spooling=0 despooling=0 despool_wait=0
> >      Files=0 Bytes=0 Bytes/sec=0
> >      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=17
> > Writing: Incremental Backup job educationdean JobId=45667 Volume=""
> >      pool="Default" device="IBMLTO2-1" (/dev/nst0)
> >      spooling=0 despooling=0 despool_wait=0
> >      Files=0 Bytes=0 Bytes/sec=0
> >      FDSocket closed
> > ====
> > 
> > Jobs waiting to reserve a drive:
> >     3605 JobId=45667 wants free drive but device "IBMLTO2-1" (/dev/nst0) 
> > is busy.
> > ====
> > [terminated jobs info snipped out]
> > Device status:
> > Autochanger "Dell-PV136T" with devices:
> >     "IBMLTO2-1" (/dev/nst0)
> >     "IBMLTO2-2" (/dev/nst1)
> >     "IBMLTO2-3" (/dev/nst2)
> > Device "IBMLTO2-1" (/dev/nst0) is mounted with:
> >      Volume:      LTO342L2
> >      Pool:        Default
> >      Media type:  LTO-2
> >      Slot 32 is loaded in drive 0.
> >      Total Bytes=11,991,168,000 Blocks=185,874 Bytes/block=64,512
> >      Positioned at File=14 Block=0
> > Device "IBMLTO2-2" (/dev/nst1) is mounted with:
> >      Volume:      LTO253L2
> >      Pool:        Default
> >      Media type:  LTO-2
> >      Slot 48 is loaded in drive 1.
> >      Total Bytes=2,193,408 Blocks=33 Bytes/block=66,466
> >      Positioned at File=1 Block=0
> > Device "IBMLTO2-3" (/dev/nst2) is not open.
> >      Device is being initialized.
> >      Drive 2 status unknown.
> > ====
> > 
> > Used Volume status:
> > [nothing further and the bconsole program hangs here]
> 
> That alone would be a bug, I guess...
> 
> > Note that the last Writing line has no volume listed.  The odd thing is 
> > that there actually is a tape in IBMLTO2-1.  There's no tape in drive 
> > IBMLTO2-3.  The pool apparently needs another appendable volume and 
> > there are several available in the Scratch pool but bacula is stuck.
> > 
> > I tried to mount a volume into the empty drive and got back the following...
> > *mount slot=61 drive=2
> > Automatically selected Storage: Dell-PV136T
> > 3001 Device "IBMLTO2-3" (/dev/nst2) is doing acquire.
> > 
> > Does anybody have any idea what to do to further troubleshoot this?  I 
> > have had some other instances of bacula getting hung up and so I have 
> > already previously applied the 2.4.3-orphaned-jobs.patch
> 
> Sounds like it's worth a bug report - especially if you can re-create 
> the problem. I cc'ed this to Eric, who - I believe - has been working 
> on this sort of problems recently.
> 

I have also seen this lately.. but that was with Bacula 2.5.18.

I could make that hang happen multiple times, but I'm not totally sure what
caused that..

-- Pasi

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users