Bacula-users

Re: [Bacula-users] bacula hang waiting for storage

2008-11-27 02:17:39
Subject: Re: [Bacula-users] bacula hang waiting for storage
From: Arno Lehmann <al AT its-lehmann DOT de>
To: bacula-users AT lists.sourceforge DOT net
Date: Thu, 27 Nov 2008 08:14:45 +0100
Hi,

26.11.2008 21:22, Bob Hetzel wrote:
> I've got bacula currently in a hung state with the following interesting 
> info.  When I run a status storage produces the following...

Is your Bacula still stuck? If so, and you have gdb installed, and a 
Bacula with debug symbols, now might be a good time to see what it's 
doing...

> Automatically selected Storage: Dell-PV136T
> Connecting to Storage daemon Dell-PV136T at gyrus:9103
> 
> gyrus-sd Version: 2.4.3 (10 October 2008) i686-pc-linux-gnu suse 10.2
> Daemon started 25-Nov-08 19:20, 59 Jobs run since started.
>   Heap: heap=3,756,032 smbytes=3,519,564 max_bytes=3,684,397 bufs=555 
> max_bufs=557
> Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8
> 
> Running Jobs:
> Writing: Incremental Backup job axh93-gx270 JobId=45634 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=1
>      Files=78 Bytes=21,123,239 Bytes/sec=2,337
>      FDReadSeqNo=970 in_msg=750 out_msg=9 fd=20
> Writing: Incremental Backup job bxn4-gx280 JobId=45641 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=1
>      Files=155 Bytes=2,925,138,595 Bytes/sec=323,648
>      FDReadSeqNo=45,916 in_msg=45480 out_msg=9 fd=35
> Writing: Incremental Backup job cdking JobId=45646 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=1
>      Files=88 Bytes=11,846,912 Bytes/sec=1,310
>      FDReadSeqNo=920 in_msg=672 out_msg=9 fd=23
> Writing: Incremental Backup job ceg3-d810 JobId=45648 Volume="LTO253L2"
>      pool="Default" device="IBMLTO2-2" (/dev/nst1)
>      spooling=0 despooling=1 despool_wait=0
>      Files=35 Bytes=1,391,695,993 Bytes/sec=176,588
>      FDReadSeqNo=21,542 in_msg=21439 out_msg=9 fd=36
> Writing: Incremental Backup job clifford3 JobId=45651 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=0
>      Files=0 Bytes=0 Bytes/sec=0
>      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=32
> Writing: Incremental Backup job cxj57-gx270 JobId=45657 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=0
>      Files=0 Bytes=0 Bytes/sec=0
>      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=33
> Writing: Incremental Backup job dxa2-d630 JobId=45665 Volume="LTO261L2"
>      pool="Default" device="IBMLTO2-3" (/dev/nst2)
>      spooling=0 despooling=0 despool_wait=0
>      Files=0 Bytes=0 Bytes/sec=0
>      FDReadSeqNo=6 in_msg=6 out_msg=4 fd=17
> Writing: Incremental Backup job educationdean JobId=45667 Volume=""
>      pool="Default" device="IBMLTO2-1" (/dev/nst0)
>      spooling=0 despooling=0 despool_wait=0
>      Files=0 Bytes=0 Bytes/sec=0
>      FDSocket closed
> ====
> 
> Jobs waiting to reserve a drive:
>     3605 JobId=45667 wants free drive but device "IBMLTO2-1" (/dev/nst0) 
> is busy.
> ====
> [terminated jobs info snipped out]
> Device status:
> Autochanger "Dell-PV136T" with devices:
>     "IBMLTO2-1" (/dev/nst0)
>     "IBMLTO2-2" (/dev/nst1)
>     "IBMLTO2-3" (/dev/nst2)
> Device "IBMLTO2-1" (/dev/nst0) is mounted with:
>      Volume:      LTO342L2
>      Pool:        Default
>      Media type:  LTO-2
>      Slot 32 is loaded in drive 0.
>      Total Bytes=11,991,168,000 Blocks=185,874 Bytes/block=64,512
>      Positioned at File=14 Block=0
> Device "IBMLTO2-2" (/dev/nst1) is mounted with:
>      Volume:      LTO253L2
>      Pool:        Default
>      Media type:  LTO-2
>      Slot 48 is loaded in drive 1.
>      Total Bytes=2,193,408 Blocks=33 Bytes/block=66,466
>      Positioned at File=1 Block=0
> Device "IBMLTO2-3" (/dev/nst2) is not open.
>      Device is being initialized.
>      Drive 2 status unknown.
> ====
> 
> Used Volume status:
> [nothing further and the bconsole program hangs here]

That alone would be a bug, I guess...

> Note that the last Writing line has no volume listed.  The odd thing is 
> that there actually is a tape in IBMLTO2-1.  There's no tape in drive 
> IBMLTO2-3.  The pool apparently needs another appendable volume and 
> there are several available in the Scratch pool but bacula is stuck.
> 
> I tried to mount a volume into the empty drive and got back the following...
> *mount slot=61 drive=2
> Automatically selected Storage: Dell-PV136T
> 3001 Device "IBMLTO2-3" (/dev/nst2) is doing acquire.
> 
> Does anybody have any idea what to do to further troubleshoot this?  I 
> have had some other instances of bacula getting hung up and so I have 
> already previously applied the 2.4.3-orphaned-jobs.patch

Sounds like it's worth a bug report - especially if you can re-create 
the problem. I cc'ed this to Eric, who - I believe - has been working 
on this sort of problems recently.

Arno

>     Bob
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 

-- 
Arno Lehmann
IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück
www.its-lehmann.de

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users