Bacula-users

[Bacula-users] A strange BAT anomaly, and a Bacula/MySQL/ZFS PSA

2011-03-08 20:25:15
Subject: [Bacula-users] A strange BAT anomaly, and a Bacula/MySQL/ZFS PSA
From: Phil Stracchino <alaric AT metrocast DOT net>
To: bacula-users <bacula-users AT lists.sourceforge DOT net>, bacula-devel <bacula-devel AT lists.sourceforge DOT net>
Date: Tue, 08 Mar 2011 20:22:25 -0500
I just had a very odd (and rather alarming, for a few minutes) event happen.

Five days ago, I received a new box of blank LTO2 tapes.
On Sunday, I deleted the first six old LTO1 tapes from my FULL-TAPE pool
and labelled six of the new LTO2 tapes, ARCH-0001 through ARCH-0006, in
preparation for the full backups scheduled to begin at 0430 yesterday.
The backups ran rather slowly, writing three tapes, ARCH-0001 through
ARCH-0003, before eventually stalling at the writing-attributes stage.
(I think I know why that happened.  I'll come back to that later.)

Late this morning, I eventually killed the stalled backups, deleted the
jobs, purged the three tapes used, and marked them RECYCLE.  Then I
restarted MySQL and Bacula (more on that later), then restarted all the
backups, by hand.  The smaller backups ran to completion fine; the main
server backup filled the first LTO2 tape not long ago and requested a
second tape.  It requested ARCH-0004, as ARCH-0004 is the first
completely unused appendable volume.  I have it set to ACCEPT ANY
VOLUME, so I loaded the correct tape, ARCH-0002, and clicked Mount in BAT.

This is where I had an adrenaline-rush moment, because after mounting
the tape BAT storage daemon status reported the storage daemon as
mounted WITH ARCH-0004.  I double-checked to ensure that I had
physically inserted ARCH-0002 into the drive, and sure enough, it was.
The display did not update to correctly show the mounted volume as
ARCH-0002 until Bacula wrote the first 5GB "file" to the tape.  That
gave me a few bad minutes.  I did NOT want to have to kill the full
backup and start it over.

(The first tape always seems to be slow to fill, because many small
files are written to it.  It took seven hours to write 226GB of data to
the first tape, an average of about half a gigabyte per minute.  By the
time Bacula gets into the second tape, it's into the big storage array
which mostly contains multi-megabyte to multi-gigabyte files, and it's
now writing about 1.3GB/minute.)


Now, I said I'd get back to the MySQL issues.  I was reading through
some documents about mySQL on ZFS, and came across (again) a
recommendation from one MySQL tester that reported the best MySQL
performance from setting the InnoDB buffer pool small, 100MB or so, and
allowing ZFS to do the data caching.  And I thought, "You know, I don't
believe I've ever tried this.  I'll give it a shot and see how it goes."

Well, after trying this, I can now definitively state:  DO NOT DO THIS
IF YOUR BACULA CATALOG IS ON MYSQL.  It may well work well for general
usage, but for Bacula, this configuration trick DOES NOT WORK.  If you
are using Bacula with a MySQL catalog database on ZFS, configure MySQL
as you would if it were NOT on ZFS.  Do NOT rely on the "let ZFS cache
the data" trick, because it won't work with Bacula.



-- 
  Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
  alaric AT caerllewys DOT net   alaric AT metrocast DOT net   phil AT 
co.ordinate DOT org
  Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater
                 It's not the years, it's the mileage.

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users