On Sun, Apr 04, 2010 at 01:20:49PM -0600, Robert LeBlanc wrote:
> I'm having problems with our SD and tapes being locked in the
> drive occasionally.

How does it manifest exactly ? bconsole umount command returns error,
or remains in some state (check with "status storage") ? Which state
and/or error ?

Have you tried shutting down bacula-sd and ejecting tape with "mt
eject" and/or "mt offline" ? Do they succeed (and the drive ejects)
or do they return error (and which one) ? Double check that bacula-sd
is down before you try those (they won't work if bacula-sd is still
having the drive open).

library eject function and/or pressing hardware eject button on the
drive itself (depending on the library type...) ?

If mt works but bacula-sd doesn't, than you can rule out hardware and
kernel -- it is bacula problem (and usually "status storage" will
show it -- it can happen sometimes if you have more than one drive
that it deadlocks by waiting for a tape that is in the other drive).

> At first I thought this might be a problem with our tape
> library.

That still looks like the most probable cause to me - like a drive in
the library is having problems. We've had a similar issue with one of
several LTO2 drives in our library; it would (sometimes) take the
tape and refuse to give it back (on "mt eject" and even physical
button touch). Needed power cycling and long (half a minute?) button
press to make it give the tape back.

After it happened third time (always the same drive) we kicked it out
of the library. Other drives worked OK all the time.

If the hardware button always works but software commands don't, it
could be fiber cables and/or GBIC/SPF (which we refused to believe at
one time because drives were always detected OK and worked, albeit
sometimes much slower than normal, without any errors in kernel logs,
and would also lock up). You can try cleaning tape also.

> Then I saw these errors in the syslog. I switched out the Qlogic FC
> adapter thinking that maybe it was just losing all the paths to the drive.

AFAIR you would get different errors if it loses path completely (but
it is possible for drive to behave erratically even if it doesn't
lose path)

> I'm still getting the errors, so I'm not sure where the hangup is. I can't
> tell if it's a bug in the kernel module, mt or bacula. Can someone give me
> some pointers to narrowing this down? This has been happening for over a
> year and through several kernel and bacula versions.
>
> This is Debian Squeeze
>
> Linux lsddomainsd 2.6.32-trunk-686 #1 SMP Sun Jan 10 06:32:16 UTC 2010 i686
> GNU/Linux

The "INFO:" messages themselves are just "normal" feature of newer
2.6.x kernels, they are informational message only (See "INFO:") that
tells you some system call (like open(2) or write(2) or read(2)) is
taking longer than 120 seconds to complete. They didn't exist in
older kernels.

It is there to catch problems with I/O schedulers and problematic
hardware issues -- but sometime it needs to be increased for tape
drives (it is quite possible for open(2) or lseek(2) on tape to have
to rewind it, and that sometimes can take more than two minutes).

you can raise the current kernel limit with:
echo 300 > /proc/sys/kernel/hung_task_timeout_secs

or (to survive reboot) by putting:
kernel.hung_task_timeout_secs=300
in /etc/sysctl.conf (or a file in /etc/sysctl.d directory)

But as I say, those will not help your lockup problems, just make the
spurious messages go away when they are to be expected.


Try the other things in the mail to narrow the problem down to
bacula, kernel or hardware.