Amanda-Users

2.6.2alpha-20090206 snapshot, total failure

2009-02-07 06:13:01
Subject: 2.6.2alpha-20090206 snapshot, total failure
From: Gene Heskett <gene.heskett AT verizon DOT net>
To: amanda-users AT amanda DOT org
Date: Sat, 07 Feb 2009 06:05:07 -0500
Greetings;

I had a total failure after changing my configure input to do away with 
the --with-tapedev line that I reported as a build warning yesterday.

In going through amanda.conf, I found a note

tpchanger      "chg-disk"      # deprecated, use amanda_changer now.
#amanda_changer  "chg-disk"

and that we should be using the latter format now.  I have changed that, and 
will fire off a run by hand as it didn't leave anything in the holding disk 
either.

which would be nice if it worked, but amcheck says otherwise:
"/usr/local/etc/amanda/Daily/amanda.conf", line 110: configuration keyword 
expected
amcheck: errors processing config file

The point is, that an invocation of amcheck did not warn about this 
miss-configuration, returning no errors.  Has the canary died?

Also, looking through the ./configure --help output, it appears I should be 
using this syntax:

--with-tape-device=DEVICE
instead of the older
        --with-tapedev="FILE:/amandatapes" \
So, should I rebuild it again, using this syntax
--with-tape-device="FILE:/amandatapes" \

I think I will before I run it again.  That was not reported as an error.

Amcheck SHOULD report this and didn't.  The canary died before the cage was 
carried into the mine.

And with all my editing, this is turning into a disaster.  Without further 
adieu, the gh.cf I use to build it, and the amanda.conf that is supposed to 
tell it what to do:
------------------------------
#!/bin/sh
# since I'm always forgetting to su amanda...
if [ `whoami` != 'amanda' ]; then
        echo
        echo "!!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!!"
        echo "Amanda needs to be configured and built by the"
        echo "user amanda, but must be installed by user root."
        echo
        exit 1
fi
make clean
rm -f config.status config.cache
./configure --with-user=amanda \
        --with-group=disk \
        --with-owner=amanda \
        --with-gnu-ld \
        --prefix=/usr/local \
        --with-debugging=/tmp/amanda-dbg/ \
        --with-tape-device="FILE:/amandatapes" \
        --with-tape-server=coyote \
        --with-bsdtcp-security --with-amandahosts \
        --with-configdir=/usr/local/etc/amanda \
        --with-config=Daily \
        --with-gnutar=/bin/tar
echo "sleeping 5 seconds for reading configures warnings"
echo "a make as amanda will continue after..."
sleep 5
make
--------------------------
amanda.conf
--------------------------
org "The Coyote Den"            # your organization name for reports
mailto "gene AT coyote.coyote DOT den"  # space separated list of operators
dumpuser "amanda"       # the user to run dumps under

inparallel 2            # maximum dumpers that will run in parallel (max 63)
                        # this maximum can be increased at compile-time,
                        # modifying MAX_DUMPERS in server-src/driverio.h

maxdumps 2              # allowing one per disk make sense to me

netusage  800 Kbps      # maximum net bandwidth for Amanda, in KB per sec

dumpcycle 7             # The number of days in the normal dump cycle
runspercycle 7          # the number of amdump runs in dumpcycle days
                        # (4 weeks * 5 amdump runs per week -- just weekdays)

tapecycle 30 tapes      # the number of tapes in rotation
                        # 4 weeks (dumpcycle) times 5 tapes per week (just
                        # the weekdays) plus a few to handle errors that
                        # need amflush and so we do not overwrite the full
                        # backups performed at the beginning of the previous
                        # cycle

dumporder "STSTSTST" # specify the priority order of each dumper
                        #   s -> smallest size
                        #   S -> biggest size
                        #   t -> smallest time
                        #   T -> biggest time
                        #   b -> smallest bandwidth
                        #   B -> biggest bandwidth
                        # try "BTBTBTBTBTBT" if you are not holding
                        # disk constrained
                        # BUT, if you want streaming, start with the
                        # big ones and work down

taperalgo largest       # The algorithm used to choose dump image to send
                        # to the taper.
                        #
                        # Possible values:
                        # [first|firstfit|largest|largestfit|smallest|last]
                        # Default: first.
                        # first         First in - first out.
                        # firstfit      The first dump image that will fit
                        #               on the current tape.
                        # largest       The largest dump image.
                        # largestfit    The largest dump image that will fit
                        #               on the current tape.
                        # smallest      The smallest dump image.
                        # last          Last in - first out.
                        #

bumpsize 10 Mb          # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1              # minimum days at each level
bumpmult 2              # threshold = bumpsize * bumpmult^(level-1)

etimeout 900            # number of seconds per filesystem for estimates.

dtimeout 2800           # number of idle seconds before a dump is aborted.

ctimeout 8              # maximum number of seconds that amcheck waits
                        # for each client host
#estimate "client"      # also is default, forces amanda to use the 
                                # same dump program to do the estimates
                        # other options are calcsize and server which aren't as 
accurate

# tapebufs 320          # deprecated, use:
device_output_buffer_size 2560k
                       # amount of buffer space to use when writing to devices

# A positive integer telling taper how many 32k buffers to allocate.
# WARNING! If this is set too high, taper will not be able to allocate
# the memory and will die.  The default is 20 (640k).
# I wonder what the limit might be, its been running at 60 for months
# so try 80.

# Specify tape device and/or tape changer.  If you don't have a tape
# changer, and you don't want to use more than one tape per run of
# amdump, just comment out the definition of tpchanger.

# Some tape changers require tapedev to be defined; others will use
# their own tape device selection mechanism.  Some use a separate tape
# changer device (changerdev), others will simply ignore this
# parameter.  Some rely on a configuration file (changerfile) to
# obtain more information about tape devices, number of slots, etc;
# others just need to store some data in files, whose names will start
# with changerfile.  For more information about individual tape
# changers, read docs/TAPE.CHANGERS.

# At most one changerfile entry must be defined; select the most
# appropriate one for your configuration.  If you select man-changer,
# keep the first one; if you decide not to use a tape changer, you may
# comment them all out.

runtapes        1               # number of tapes to be used in a single run of 
amdump
amanda_changer  "chg-disk"
tapedev         "file:/amandatapes/Dailys/"     # the no-rewind tape device to 
be used
changerdev      "/dev/null"
changerfile     "/usr/local/etc/amanda/Daily/chg-disk"
tapetype        HARD-DISK
labelstr "^Dailys-[0-9][0-9]*$" # label constraint regex: all tapes must match

#Some boolean variables
autoflush yes
amrecover_do_fsf yes            # amrecover will call amrestore with the
                                # -f flag for faster positioning of the tape.
amrecover_check_label yes       # amrecover will call amrestore with the
                                # -l flag to check the label.
#amrecover_changer "chg-scsi"   # amrecover will use the changer if you 
restore
amrecover_changer "chg-disk"
#amrecover_changer  "chg-mtx"    # from this device.

# By default, Amanda can only track at most one run per calendar day. When
# the usetimestamps option is enabled, however, Amanda can track as many
# runs as you care to make.
# WARNING: This option is not backward-compatible. Do not enable it if you
#          intend to downgrade your server installation to Amanda community
#          edition 2.5
usetimestamps yes

# Specify holding disks.  These are used as a temporary staging area for
# dumps before they are written to tape and are recommended for most sites.
# The advantages include: tape drive is more likely to operate in streaming
# mode (which reduces tape and drive wear, reduces total dump time); multiple
# dumps can be done in parallel (which can dramatically reduce total dump 
time.
# The main disadvantage is that dumps on the holding disk need to be flushed
# (with amflush) to tape after an operating system crash or a tape failure.
# If no holding disks are specified then all dumps will be written directly
# to tape.  If a dump is too big to fit on the holding disk than it will be
# written directly to tape.  If more than one holding disk is specified then
# they will all be used based on activity and available space.

holdingdisk hd1 {
    comment "main holding disk"
    directory "/dumps"  # where the holding disk is
    use -500 Mb         # how much space can we use on it
                        # a non-positive value means:
                        #        use all space but that value
    chunksize 2000Mb    # size of chunk if you want big dump to be
                        # dumped on multiple files on holding disks
                        #  N Kb/Mb/Gb split images in chunks of size N
                        #             The maximum value should be
                        #             (MAX_FILE_SIZE - 1Mb)
                        #  0          same as INT_MAX bytes
    }
#holdingdisk hd2 {
#    directory "/dumps2/amanda"
#    use 1000 Mb
#    }
#holdingdisk hd3 {
#    directory "/mnt/disk4"
#    use 1000 Mb
#    }


# If amanda cannot find a tape on which to store backups, it will run
# as many backups as it can to the holding disks.  In order to save
# space for unattended backups, by default, amanda will only perform
# incremental backups in this case, i.e., it will reserve 100% of the
# holding disk space for the so-called degraded mode backups.
# However, if you specify a different value for the `reserve'
# parameter, amanda will not degrade backups if they will fit in the
# non-reserved portion of the holding disk.

reserve 25 # percent
# This means save at least 25% of the holding disk space for degraded
# mode backups.

# The format for a ColumnSpec is a ',' seperated list of triples.
# Each triple consists of
#   + the name of the column (as in ColumnNameStrings)
#   + prefix before the column
#   + the width of the column, if set to -1 it will be recalculated
#     to the maximum length of a line to print.
# Example:
#       "Disk=1:17,HostName=1:10,OutKB=1:7"
# or
#       "Disk=1:-1,HostName=1:10,OutKB=1:7"
#
# You need only specify those colums that should be changed from
# the default. If nothing is specified in the configfile, the
# above compiled in values will be in effect, resulting in an
# output as it was all the time.
# The names of the columns are:
# HostName, Disk, Level, OrigKB, OutKB, Compress, DumpTime, DumpRate,
# TapeTime and TapeRate.
#                                                       ElB, 1999-02-24.
columnspec 
"HostName=0:7,Disk=1:26,Level=1:2,OrigKB=1:7,OutKB=1:7,Compress=1:5,DumpTime=1:5,DumpRate=1:7,TapeTime=1:5,TapeRate=1:7"

# This is a recently added item, "displayunit".  The default has always been K 
till now
# One should set
#displayunit "m"
# to make it use megabytes instead

# Amanda needs a few Mb of diskspace for the log and debug files,
# as well as a database.  This stuff can grow large, so the conf directory
# isn't usually appropriate.  Some sites use /usr/local/var and some /usr/adm.
# Create an amanda directory under there.  You need a separate infofile and
# logdir for each configuration, so create subdirectories for each conf and
# put the files there.  Specify the locations below.

# Note that, although the keyword below is infofile, it is only so for
# historic reasons, since now it is supposed to be a directory (unless
# you have selected some database format other than the `text' default)
infofile "/usr/local/var/amanda/Daily/curinfo"  # database DIRECTORY
logdir   "/usr/local/var/amanda/Daily"          # log directory
indexdir "/usr/local/var/amanda/Daily/index"    # index directory
tapelist "/usr/local/etc/amanda/Daily/tapelist" # list of used tapes
# tapelist is stored, by default, in the directory that contains amanda.conf


# tapetypes

# Define the type of tape you use here, and use it in "tapetype"
# above.  Some typical types of tapes are included here.  The tapetype
# tells amanda how many MB will fit on the tape, how big the filemarks
# are, and how fast the tape device is.

# A filemark is the amount of wasted space every time a tape section
# ends.  If you run `make tapetype' in tape-src, you'll get a program
# that generates tapetype entries, but it is slow as hell, use it only
# if you really must and, if you do, make sure you post the data to
# the amanda mailing list, so that others can use what you found out
# by searching the archives.

# For completeness Amanda should calculate the inter-record gaps too,
# but it doesn't.  For EXABYTE and DAT tapes this is ok.  Anyone using
# 9 tracks for amanda and need IRG calculations?  Drop me a note if
# so.

# If you want amanda to print postscript paper tape labels
# add a line after the comment in the tapetype of the form
#    lbl-templ "/path/to/postscript/template/label.ps"

# if you want the label to go to a printer other than the default
# for your system, you can also add a line above for a different
# printer. (i usually add that line after the dumpuser specification)

# dumpuser "operator"     # the user to run dumps under
# printer "mypostscript"  # printer to print paper label on

# here is an example of my definition for an EXB-8500

# define tapetype EXB-8500 {
# ...
#     lbl-templ "/usr/local/amanda/config/lbl.exabyte.ps"
# }

define application-tool app_amgtar {
     comment "amgtar"
     plugin  "amgtar"
     #property "GNUTAR-PATH" "/path/to/gtar"
     #property "GNUTAR-LISTDIR" "/path/to/gnutar_list_dir"
     #default from gnutar_list_dir setting in amanda-client.conf
     #property "ONE-FILE-SYSTEM" "yes"  #use '--one-file-system' option
     #property "SPARSE" "yes"           #use '--sparse' option
     #property "ATIME-PRESERVE" "yes"   #use '--atime-preserve=system' option
     property "CHECK-DEVICE" "no"       #use '--no-check-device' if set to "no"
}

define tapetype HARD-DISK {
        comment "Hitachi 400GB sata drive"
        lbl-templ "/usr/local/etc/amanda/Daily/3hole.ps"
        length 13800 mbytes # leave 190 megs of room for saved configs & indices
}
[snip unused tapetypes]

# dumptypes
#
# These are referred to by the disklist file.  The dumptype specifies
# certain parameters for dumping including:
#   auth        - authentication scheme to use between server and client.
#                 Valid values are "bsd" and "krb4".  Default: [auth bsd]
#   comment     - just a comment string
#   comprate    - set default compression rate.  Should be followed by one or
#                 two numbers, optionally separated by a comma.  The 1st is
#                 the full compression rate; the 2nd is the incremental rate.
#                 If the second is omitted, it is assumed equal to the first.
#                 The numbers represent the amount of the original file the
#                 compressed file is expected to take up.
#                 Default: [comprate 0.50, 0.50]
#   compress    - specify compression of the backed up data.  Valid values are:
#                 "none"        - don't compress the dump output.
#                 "client best" - compress on the client using the best (and
#                                 probably slowest) algorithm.
#                 "client fast" - compress on the client using fast algorithm.
#                 "server best" - compress on the tape host using the best (and
#                                 probably slowest) algorithm.
#                 "server fast" - compress on the tape host using a fast
#                                 algorithm.  This may be useful when a fast
#                                 tape host is backing up slow clients.
#                 Default: [compress client fast]
#   dumpcycle   - set the number of days in the dump cycle, ie, set how often a
#                 full dump should be performed.  Default: from DUMPCYCLE above
#   exclude     - specify files and directories to be excluded from the dump.
#                 Useful with gnutar only; silently ignored by dump and samba.
#                 Valid values are:
#                 "pattern"       - a shell glob pattern defining which files
#                                   to exclude.
#                                   gnutar gets --exclude="pattern"
#                 list "filename" - a file (on the client!) containing patterns
#                                   re's (1 per line) defining which files to
#                                   exclude.
#                                   gnutar gets --exclude-from="filename"
#                 Note that the `full pathname' of a file within its
#                 filesystem starts with `./', because of the way amanda runs
#                 gnutar: `tar -C $mountpoint -cf - --lots-of-options .' (note
#                 the final dot!)  Thus, if you're backing up `/usr' with a
#                 diskfile entry like ``host /usr gnutar-root', but you don't
#                 want to backup /usr/tmp, your exclude list should contain
#                 the pattern `./tmp', as this is relative to the `/usr' above.
#                 Please refer to the man-page of gnutar for more information.
#                 If a relative pathname is specified as the exclude list,
#                 it is searched from within the directory that is
#                 going to be backed up.
#                 Default: include all files
#   holdingdisk - should the holding disk be used for this dump.  Useful for
#                 dumping the holding disk itself.  Default: [holdingdisk yes]
#   ignore      - do not back this filesystem up.  Useful for sharing a single
#                 disklist in several configurations.
#   index       - keep an index of the files backed up.  Default: [index no]
#   kencrypt    - encrypt the data stream between the client and server.
#                 Default: [kencrypt no]
#   maxdumps    - max number of concurrent dumps to run on the client.
#                 Default: [maxdumps 1]
#   priority    - priority level of the dump.  Valid levels are "low", "medium"
#                 or "high".  These are really only used when Amanda has no
#                 tape to write to because of some error.  In that "degraded
#                 mode", as many incrementals as will fit on the holding disk
#                 are done, higher priority first, to insure the important
#                 disks are at least dumped.  Default: [priority medium]
#   program     - specify the dump system to use.  Valid values are "DUMP" and
#                 "GNUTAR".  Default: [program "DUMP"].
#   record      - record the dump in /etc/dumpdates.  Default: [record yes]
#   skip-full   - skip the disk when a level 0 is due, to allow full backups
#                 outside Amanda, eg when the machine is in single-user mode.
#   skip-incr   - skip the disk when the level 0 is NOT due.  This is used in
#                 archive configurations, where only full dumps are done and
#                 the tapes saved.
#   starttime   - delay the start of the dump?  Default: no delay
#   strategy    - set the dump strategy.  Valid strategies are currently:
#                 "standard" - the standard one.
#                 "nofull"   - do level 1 dumps every time.  This can be used,
#                              for example, for small root filesystems that
#                              only change slightly relative to a site-wide
#                              prototype.  Amanda then backs up just the
#                              changes.
#                 "noinc"    - do level 0 dumps every time.
#                              Unfortunately, this is not currently
#                              implemented.  Use `dumpcycle 0'
#                              instead.
#                 "skip"     - skip all dumps.  Useful for sharing a single
#                              disklist in several configurations.
#                 "incronly" - do only incremental dumps. This is similar
#                              to strategy 'nofull', but will increase
#                              the dump level as usual. Full dumps will
#                              only be performed when an 'amadmin force'
#                              has been issued
#                 Default: [strategy standard]
#
# Note that you may specify previously defined dumptypes as a shorthand way
# of defining parameters.

define dumptype dt_amgtar {
        program "APPLICATION"
        application "app_amgtar"
        index yes
}

define dumptype global {
    comment "Global definitions"
    # This is quite useful for setting global parameters, so you don't have
    # to type them everywhere.  All dumptype definitions in this sample file
    # do include these definitions, either directly or indirectly.
    # There's nothing special about the name `global'; if you create any
    # dumptype that does not contain the word `global' or the name of any
    # other dumptype that contains it, these definitions won't apply.
    # Note that these definitions may be overridden in other
    # dumptypes, if the redefinitions appear *after* the `global'
    # dumptype name.
    # You may want to use this for globally enabling or disabling
    # indexing, recording, etc.  Some examples:
    index yes
    program "GNUTAR"

#    application "/usr/local/libexec/amanda/application/amgtar"
#   program "APPLICATION"
    # record no
##   tape_splitsize     1G
##   split_diskbuffer   "/dumps"
#   fallback_splitsize  64m
}

define dumptype always-full {
#    global
        dt_amgtar
        comment "Full dump of this filesystem always"
        compress none
        priority high
        dumpcycle 0
}

define dumptype always-full-comp {
#       global
        dt_amgtar
        always-full
        comment "Full, compressed dump of this filesystem"
        compress server best
}

define dumptype root-tar {
#       global
        dt_amgtar
        comment "root partitions dumped with tar"
        compress none
        priority low
}

define dumptype coyote-tar {
        root-tar
        exclude list "/GenesAmandaHelper-0.6/excludes"
}

define dumptype coyote-tar-iso {
        root-tar
}

define dumptype gene-tar {
        root-tar
}

define dumptype user-tar {
    root-tar
    comment "user partitions dumped with tar"
    priority medium
}

define dumptype high-tar {
    root-tar
    comment "partitions dumped with tar"
    priority high
}

define dumptype comp-root-tar {
    root-tar
    comment "Root partitions with compression"
    compress client best
}

define dumptype comp-coyote-tar {
        comp-root-tar
}

define dumptype comp-gene-tar {
        comp-root-tar
}

define dumptype comp-shop-tar {
        comp-root-tar
}

define dumptype comp-user-tar {
    user-tar
    compress client best
}

define dumptype tmp-comp-coyote-tar   {
        comp-root-tar
        comment "/tmp dir w/o amanda stuff, its re-entrant!"
        exclude file "/GenesAmandaHelper-0.6/excludes"
}

define dumptype holding-disk {
#       global
        dt_amgtar
        comment "The master-host holding disk itself"
        holdingdisk no # do not use the holding disk
        priority medium
}

define dumptype comp-user {
#       global
        dt_amgtar
        comment "Non-root partitions on reasonably fast machines"
        compress client fast
        priority medium
}

define dumptype nocomp-user {
    comp-user
    comment "Non-root partitions on slow machines"
    compress none
}

define dumptype comp-root {
#       global
        comment "Root partitions with compression"
        compress client fast
        priority low
}

define dumptype nocomp-root {
        comp-root
        comment "Root partitions without compression"
        compress none
}

define dumptype comp-high {
#       global
        dt_amgtar
        comment "very important partitions on fast machines"
        compress client best
        priority high
}

define dumptype nocomp-high {
        comp-high
        comment "very important partitions on slow machines"
        compress none
}

define dumptype nocomp-test {
#       global
        dt_amgtar
        comment "test dump without compression, no /etc/dumpdates recording"
        compress none
        record no
        priority medium
}

define dumptype comp-test {
        nocomp-test
        comment "test dump with compression, no /etc/dumpdates recording"
        compress client best
}

# network interfaces
#
# These are referred to by the disklist file.  They define the attributes
# of the network interface that the remote machine is accessed through.
# Notes: - netusage above defines the attributes that are used when the
#          disklist entry doesn't specify otherwise.
#        - the values below are only samples.
#        - specifying an interface does not force the traffic to pass
#          through that interface.  Your OS routing tables do that.  This
#          is just a mechanism to stop Amanda trashing your network.
# Attributes are:
#       use             - bandwidth above which amanda won't start
#                         backups using this interface.  Note that if
#                         a single backup will take more than that,
#                         amanda won't try to make it run slower!

define interface local {
    comment "a local disk"
    use 1000 kbps
}

define interface le0 {
    comment "100 Mbps ethernet"
    use 5000 kbps
}

# You may include other amanda configuration files, so you can share
# dumptypes, tapetypes and interface definitions among several
# configurations.

#includefile "/usr/local/etc/amanda/Daily/amanda.conf.main"
-----------------------

What am I doing wrong now?

Thanks.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Trying to get Windows to run on the hardware that Linux typically runs on
is like pushing an elephant through a keyhole.

   -- Forbes Magazine