Amanda-Users

Re: backup just on holding disks makes many level 0

2006-03-15 10:59:46
Subject: Re: backup just on holding disks makes many level 0
From: Gene Heskett <gene.heskett AT verizon DOT net>
To: amanda-users AT amanda DOT org
Date: Wed, 15 Mar 2006 10:55:58 -0500
On Wednesday 15 March 2006 07:03, Thomas Widhalm wrote:
>On Tue, 2006-03-14 at 22:49 +0100, Stefan G. Weichinger wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Thomas Widhalm schrieb:
>> >> Have you tried resetting some parameters like bumpsize, bumpmult,
>> >> and bumpdays?  They may have an affect on your situation.
>> >
>> > This could really be it. bumpsize was set 100Mb. Maybe, because
>> > this setup was used to backup on a tape just once in a week.
>> > Unfortunately I had to take over the whole machine without having
>> > time to look through the configuration too much.
>>
>> If that didn't solve your issues you maybe share your config with
>> us. This would help to get the picture.
>
>Here are the 2 parts of the config:
>
>first file:

But what is this files name?

>#
># amanda.conf - sample Amanda configuration file.  This started off
> life as
>#               the actual config file in use at CS.UMD.EDU.
>#
># If your configuration is called, say, "csd", then this file normally
>goes
># in /etc/amanda/csd/amanda.conf.
>#
>
>mailto "root"           # space separated list of operators at your
> site dumpuser "amanda"       # the user to run dumps under
>
>netusage  100000 Kbps   # maximum net bandwidth for Amanda, in KB per
>sec
>
>                        # 4 weeks (dumpcycle) times 5 tapes per week
>(just
>                        # the weekdays) plus a few to handle errors
> that # need amflush and so we do not overwrite the full
>                        # backups performed at the beginning of the
>previous
>                        # cycle
>maxdumps 1
>
>### ### ###
># WARNING: don't use `inf' for tapecycle, it's broken!
>### ### ###
>
>bumpsize 10 Mb          # minimum savings (threshold) to bump level 1
> -> 2
>bumpdays 1              # minimum days at each level
>bumpmult 4              # threshold = bumpsize * bumpmult^(level-1)
>
>etimeout 1800           # number of seconds per filesystem for
>estimates.
>#etimeout -600          # total number of seconds for estimates.
># a positive number will be multiplied by the number of filesystems on
># each host; a negative number will be taken as an absolute total
> time- out.
># The default is 5 minutes per filesystem.
>
>
># Specify tape device and/or tape changer.  If you don't have a tape
># changer, and you don't want to use more than one tape per run of
># amdump, just comment out the definition of tpchanger.
>
># Some tape changers require tapedev to be defined; others will use
># their own tape device selection mechanism.  Some use a separate tape
># changer device (changerdev), others will simply ignore this
># parameter.  Some rely on a configuration file (changerfile) to
># obtain more information about tape devices, number of slots, etc;
># others just need to store some data in files, whose names will start
># with changerfile.  For more information about individual tape
># changers, read docs/TAPE.CHANGERS.
>
># At most one changerfile entry must be defined; select the most
># appropriate one for your configuration.  If you select man-changer,
># keep the first one; if you decide not to use a tape changer, you may
># comment them all out.
>
>runtapes 1              # number of tapes to be used in a single run
> of amdump
>#tpchanger "chg-manual" # the tape-changer glue script
>#tapedev "file:/data/amanda/"   # the no-rewind tape device to be used
>#tapedev "no-such-device"       # the no-rewind tape device to be used
>#tapedev "/dev/null"    # the no-rewind tape device to be used
>#rawtapedev "/dev/null" # the raw device to be used (ftape only)
>#rawtapedev "no-such-device"    # the raw device to be used (ftape
> only) #changerfile "/var/lib/amanda/DailySet1/changer"
>#changerfile "/var/lib/amanda/DailySet1/changer-status"
>#changerfile "/etc/amanda/DailySet1/changer.conf"
>#changerdev "/dev/null"
>#changerdev "no-such-device"
>
>#tapetype HP-DAT                # what kind of tape it is (see
> tapetypes below)
>#labelstr "^DailySet1[0-9][0-9]*$"      # label constraint regex: all
>tapes must match
>
>#tapetype CISOS-notape
>#labelstr "^ONYX[0-9][0-9]*$"   # label constraint regex: all tapes
> must match
>
># Specify holding disks.  These are used as a temporary staging area
> for # dumps before they are written to tape and are recommended for
> most sites.
># The advantages include: tape drive is more likely to operate in
>streaming
># mode (which reduces tape and drive wear, reduces total dump time);
>multiple
># dumps can be done in parallel (which can dramatically reduce total
>dump time.
># The main disadvantage is that dumps on the holding disk need to be
>flushed
># (with amflush) to tape after an operating system crash or a tape
>failure.
># If no holding disks are specified then all dumps will be written
>directly
># to tape.  If a dump is too big to fit on the holding disk than it
> will be
># written directly to tape.  If more than one holding disk is
> specified then
># they will all be used round-robin.
>
>#holdingdisk hd2 {
>#    directory "/dumps2/amanda"
>#    use 1000 Mb
>#    }
>#holdingdisk hd3 {
>#    directory "/mnt/disk4"
>#    use 1000 Mb
>#    }
>
>
># If amanda cannot find a tape on which to store backups, it will run
># as many backups as it can to the holding disks.  In order to save
># space for unattended backups, by default, amanda will only perform
># incremental backups in this case, i.e., it will reserve 100% of the
># holding disk space for the so-called degraded mode backups.
># However, if you specify a different value for the `reserve'
># parameter, amanda will not degrade backups if they will fit in the
># non-reserved portion of the holding disk.
>
># reserve 30 # percent
>reserve 0
>
># This means save at least 30% of the holding disk space for degraded
># mode backups.
>
># Amanda needs a few Mb of diskspace for the log and debug files,
># as well as a database.  This stuff can grow large, so the conf
>directory
># isn't usually appropriate.  Some sites use /usr/local/var and
>some /usr/adm.
># Create an amanda directory under there.  You need a separate
> infofile and
># logdir for each configuration, so create subdirectories for each
> conf and
># put the files there.  Specify the locations below.
>
>#tapelist "/var/lib/amanda/DailySet1/tapelist"  # list of used tapes
># tapelist is stored, by default, in the directory that contains
>amanda.conf
>
>
># tapetypes
>
># Define the type of tape you use here, and use it in "tapetype"
># above.  Some typical types of tapes are included here.  The tapetype
># tells amanda how many MB will fit on the tape, how big the filemarks
># are, and how fast the tape device is.
>
># A filemark is the amount of wasted space every time a tape section
># ends.  If you run `make tapetype' in tape-src, you'll get a program
># that generates tapetype entries, but it is slow as hell, use it only
># if you really must and, if you do, make sure you post the data to
># the amanda mailing list, so that others can use what you found out
># by searching the archives.
>
># For completeness Amanda should calculate the inter-record gaps too,
># but it doesn't.  For EXABYTE and DAT tapes this is ok.  Anyone using
># 9 tracks for amanda and need IRG calculations?  Drop me a note if
># so.
>
># If you want amanda to print postscript paper tape labels
># add a line after the comment in the tapetype of the form
>#    lbl-templ "/path/to/postscript/template/label.ps"
>
># if you want the label to go to a printer other than the default
># for your system, you can also add a line above for a different
># printer. (i usually add that line after the dumpuser specification)
>
># dumpuser "operator"     # the user to run dumps under
># printer "mypostscript"  # printer to print paper label on
>
># here is an example of my definition for an EXB-8500
>
># define tapetype EXB-8500 {
># ...
>#     lbl-templ "/usr/local/amanda/config/lbl.exabyte.ps"
># }
>
>define tapetype Quantum-DLT7000 {
>  comment "Quantum DLT7000 (hardware compression on: Xmn)"
>  length 29206 mbytes
>  filemark 30 kbytes
>  speed 1670 kps
>}
>
>define tapetype DEC-TZ89 {
>  comment "DEC TZ89 (hardware compression on: Xmn)"
>  length 29414 mbytes
>  filemark 24 kbytes
>  speed 1634 kps
>}
>
>define tapetype CISOS-notape {
>    comment "ONYX dummy tape"
>    length 1000 gbytes
>    filemark 0 kbytes         # don't know a better value
>    speed 6000 kbytes            # dito
>}
>
>define tapetype QIC-60 {
>    comment "Archive Viper"
>    length 60 mbytes
>    filemark 100 kbytes         # don't know a better value
>    speed 100 kbytes            # dito
>}
>
>define tapetype DEC-DLT2000 {
>    comment "DEC Differential Digital Linear Tape 2000"
>    length 15000 mbytes
>    filemark 8 kbytes
>    speed 1250 kbytes
>}
>
># goluboff AT butch.Colorado DOT EDU
># in amanda-users (Thu Dec 26 01:55:38 MEZ 1996)
>define tapetype DLT {
>    comment "DLT tape drives"
>    length 20000 mbytes         # 20 Gig tapes
>    filemark 2000 kbytes        # I don't know what this means
>    speed 1536 kbytes           # 1.5 Mb/s
>}
>
>define tapetype SURESTORE-1200E {
>    comment "HP AutoLoader"
>    length 3900 mbytes
>    filemark 100 kbytes
>    speed 500 kbytes
>}
>
>define tapetype EXB-8500 {
>    comment "Exabyte EXB-8500 drive on decent machine"
>    length 4200 mbytes
>    filemark 48 kbytes
>    speed 474 kbytes
>}
>
>define tapetype EXB-8200 {
>    comment "Exabyte EXB-8200 drive on decent machine"
>    length 2200 mbytes
>    filemark 2130 kbytes
>    speed 240 kbytes
>}
>
>define tapetype HP-DAT {
>    comment "DAT tape drives"
>    # data provided by Rob Browning <rlb AT cs.utexas DOT edu>
>    length 1930 mbytes
>    filemark 111 kbytes
>    speed 468 kbytes
>}
>
>define tapetype DAT {
>    comment "DAT tape drives"
>    length 1000 mbytes          # these numbers are not accurate
>    filemark 100 kbytes         # but you get the idea
>    speed 100 kbytes
>}
>
>define tapetype MIMSY-MEGATAPE {
>    comment "Megatape (Exabyte based) drive through Emulex on Vax
> 8600" length 2200 mbytes
>    filemark 2130 kbytes
>    speed 170 kbytes            # limited by the Emulex bus interface,
>ugh
>}
>
>
># dumptypes
>#
># These are referred to by the disklist file.  The dumptype specifies
># certain parameters for dumping including:
>#   auth        - authentication scheme to use between server and
>client.
>#                 Valid values are "bsd" and "krb4".  Default: [auth
>bsd]
>#   comment     - just a comment string
>#   comprate    - set default compression rate.  Should be followed by
>one or
>#                 two numbers, optionally separated by a comma.  The
> 1st is
>#                 the full compression rate; the 2nd is the
> incremental rate.
>#                 If the second is omitted, it is assumed equal to the
>first.
>#                 The numbers represent the amount of the original
> file the
>#                 compressed file is expected to take up.
>#                 Default: [comprate 0.50, 0.50]
>#   compress    - specify compression of the backed up data.  Valid
>values are:
>#                 "none"        - don't compress the dump output.
>#                 "client best" - compress on the client using the
> best (and
>#                                 probably slowest) algorithm.
>#                 "client fast" - compress on the client using fast
>algorithm.
>#                 "server best" - compress on the tape host using the
>best (and
>#                                 probably slowest) algorithm.
>#                 "server fast" - compress on the tape host using a
> fast #                                 algorithm.  This may be useful
> when a fast
>#                                 tape host is backing up slow
> clients. #                 Default: [compress client fast]
>#   dumpcycle   - set the number of days in the dump cycle, ie, set
> how often a
>#                 full dump should be performed.  Default: from
>DUMPCYCLE above
>#   exclude     - specify files and directories to be excluded from
> the dump.
>#                 Useful with gnutar only; silently ignored by dump
> and samba.
>#                 Valid values are:
>#                 "pattern"       - a shell glob pattern defining
> which files
>#                                   to exclude.
>#                                   gnutar gets --exclude="pattern"
>#                 list "filename" - a file (on the client!) containing
>patterns
>#                                   re's (1 per line) defining which
>files to
>#                                   exclude.
>#                                   gnutar gets --exclude-
>from="filename"
>#                 Note that the `full pathname' of a file within its
>#                 filesystem starts with `./', because of the way
> amanda runs
>#                 gnutar: `tar -C $mountpoint -cf - --lots-of-
>options .' (note
>#                 the final dot!)  Thus, if you're backing up `/usr'
>with a
>#                 diskfile entry like ``host /usr gnutar-root', but
> you don't
>#                 want to backup /usr/tmp, your exclude list should
>contain
>#                 the pattern `./tmp', as this is relative to the
> `/usr' above.
>#                 Please refer to the man-page of gnutar for more
>information.
>#                 Default: include all files
>#   holdingdisk - should the holding disk be used for this dump. 
> Useful for
>#                 dumping the holding disk itself.  Default:
>[holdingdisk yes]
>#   ignore      - do not back this filesystem up.  Useful for sharing
> a single
>#                 disklist in several configurations.
>#   index       - keep an index of the files backed up.  Default:
> [index no]
>#   kencrypt    - encrypt the data stream between the client and
> server. #                 Default: [kencrypt no]
>#   maxdumps    - max number of concurrent dumps to run on the client.
>#                 Default: [maxdumps 1]
>#   priority    - priority level of the dump.  Valid levels are "low",
>"medium"
>#                 or "high".  These are really only used when Amanda
> has no
>#                 tape to write to because of some error.  In that
>"degraded
>#                 mode", as many incrementals as will fit on the
> holding disk
>#                 are done, higher priority first, to insure the
>important
>#                 disks are at least dumped.  Default: [priority
> medium] #   program     - specify the dump system to use.  Valid
> values are "DUMP" and
>#                 "GNUTAR".  Default: [program "DUMP"].
>#   record      - record the dump in /etc/dumpdates.  Default: [record
>yes]
>#   skip-full   - skip the disk when a level 0 is due, to allow full
>backups
>#                 outside Amanda, eg when the machine is in
> single-user mode.
>#   skip-incr   - skip the disk when the level 0 is NOT due.  This is
>used in
>#                 archive configurations, where only full dumps are
> done and
>#                 the tapes saved.
>#   starttime   - delay the start of the dump?  Default: no delay
>#   strategy    - set the dump strategy.  Valid strategies are
>currently:
>#                 "standard" - the standard one.
>#                 "nofull"   - do level 1 dumps every time.  This can
> be used,
>#                              for example, for small root filesystems
>that
>#                              only change slightly relative to a
> site- wide
>#                              prototype.  Amanda then backs up just
> the #                              changes.
>#                 "noinc"    - do level 0 dumps every time.
>#                              Unfortunately, this is not currently
>#                              implemented.  Use `dumpcycle 0'
>#                              instead.
>#                 "skip"     - skip all dumps.  Useful for sharing a
>single
>#                              disklist in several configurations.
>#                 Default: [strategy standard]
>#
># Note that you may specify previously defined dumptypes as a
> shorthand way
># of defining parameters.
>
>####
>#### BEGIN USEFUL DUMPTYPES
>####
>
>define dumptype unix {
>    comment "Global definitions UNIX LINUX"
>    index yes
>    #exclude list ".amanda.excludes"
>}
>
>define dumptype unix-tar {
>    unix
>    program "GNUTAR"
>}
>
>define dumptype unix-tar-d0-c0-t1 {
>    unix-tar
>    comment "partitions full dumped with tar not zipped"
>    compress none
>    dumpcycle 0
>}
>
>define dumptype unix-tar-d1-c0-t1 {
>    unix-tar
>    comment "partitions incr dumped with tar not zipped"
>    compress none
>    strategy incronly
>}
>
>define dumptype unix-tar-d0-c1-t1 {
>    unix-tar
>    comment "partitions full dumped with tar zipped"
>    compress client fast
>    # dumpcycle 0 #changed by widhalmt 20060103
>}
>
>define dumptype unix-tar-d1-c1-t1 {
>    unix-tar
>    comment "partitions incr dumped with tar zipped"
>    #compress client fast
>    compress client best #changed by widhalmt 20060310
>    # strategy incronly #changed by widhalmt 20060103
>}
>
>define dumptype unix-dump {
>    unix
>}
>
>define dumptype unix-dump-d0-c0-t1 {
>    unix-dump
>    comment "partitions full dumped"
>    compress none
>    #dumpcycle 0 #changed by widhalmt 20060111
>}
>
>define dumptype unix-dump-d1-c0-t1 {
>    unix-dump
>    comment "partitions incr dumped"
>    compress none
>    strategy incronly
>}
>
>define dumptype unix-dump-d0-c1-t1 {
>    unix-dump
>    comment "partitions full dumped"
>    compress none
>    #dumpcycle 0 #changed by widhalmt 20060111
>}
>
>define dumptype unix-dump-d1-c1-t1 {
>    unix-dump
>    comment "partitions incr dumped"
>    compress none
>    strategy incronly
>}
>
>####
>#### END USEFUL DUMPTYPES
>####
>
>define dumptype linux-holding-disk {
>    unix-tar
>    comment "The master-host holding disk itself"
>    holdingdisk no # do not use the holding disk
>    priority medium
>}
>
>define dumptype irix {
>    comment "Global definitions IRIX"
>    exclude list optional ".amanda_no"
>}
>
>define dumptype irix-root {
>    irix
>    comment "irix root partitions"
>    compress none
>    index
>    priority low
>}
>
>define dumptype irix-high-tar {
>    irix-root
>    comment "irix partitions dumped with tar"
>    program "GNUTAR"
>    compress client fast
>    priority high
>}
>
>define dumptype global {
>    comment "Global definitions"
>    # This is quite useful for setting global parameters, so you don't
>have
>    # to type them everywhere.  All dumptype definitions in this
> sample file
>    # do include these definitions, either directly or indirectly.
>    # There's nothing special about the name `global'; if you create
> any # dumptype that does not contain the word `global' or the name of
> any
>    # other dumptype that contains it, these definitions won't apply.
>    # Note that these definitions may be overridden in other
>    # dumptypes, if the redefinitions appear *after* the `global'
>    # dumptype name.
>    # You may want to use this for globally enabling or disabling
>    # indexing, recording, etc.  Some examples:
>    # index yes
>    # record no
>}
>
>define dumptype always-full {
>    global
>    comment "Full dump of this filesystem always"
>    compress none
>    priority high
>    dumpcycle 0
>}
>
>define dumptype root-tar {
>    global
>    program "GNUTAR"
>    comment "root partitions dumped with tar"
>    compress none
>    index
>    exclude list optional ".amanda_no"
>    priority low
>}
>
>define dumptype user-tar {
>    root-tar
>    comment "user partitions dumped with tar"
>    priority medium
>}
>
>define dumptype high-tar {
>    root-tar
>    comment "partitions dumped with tar"
>    priority high
>}
>
>define dumptype comp-root-tar {
>    root-tar
>    comment "Root partitions with compression"
>    compress client fast
>}
>
>define dumptype comp-user-tar {
>    user-tar
>    compress client fast
>}
>
>define dumptype holding-disk {
>    global
>    comment "The master-host holding disk itself"
>    holdingdisk no # do not use the holding disk
>    priority medium
>}
>
>define dumptype comp-user {
>    global
>    comment "Non-root partitions on reasonably fast machines"
>    compress client fast
>    priority medium
>}
>
>define dumptype nocomp-user {
>    comp-user
>    comment "Non-root partitions on slow machines"
>    compress none
>}
>
>define dumptype comp-root {
>    global
>    comment "Root partitions with compression"
>    compress client fast
>    priority low
>}
>
>define dumptype nocomp-root {
>    comp-root
>    comment "Root partitions without compression"
>    compress none
>}
>
>define dumptype comp-high {
>    global
>    comment "very important partitions on fast machines"
>    compress client best
>    priority high
>}
>
>define dumptype nocomp-high {
>    comp-high
>    comment "very important partitions on slow machines"
>    compress none
>}
>
>define dumptype nocomp-test {
>    global
>    comment "test dump without compression, no /etc/dumpdates
> recording" compress none
>    record no
>    priority medium
>}
>
>define dumptype comp-test {
>    nocomp-test
>    comment "test dump with compression, no /etc/dumpdates recording"
>    compress client fast
>}
>
># network interfaces
>#
># These are referred to by the disklist file.  They define the
>attributes
># of the network interface that the remote machine is accessed
> through. # Notes: - netusage above defines the attributes that are
> used when the #          disklist entry doesn't specify otherwise.
>#        - the values below are only samples.
>#        - specifying an interface does not force the traffic to pass
>#          through that interface.  Your OS routing tables do that.
>This
>#          is just a mechanism to stop Amanda trashing your network.
># Attributes are:
>#       use             - bandwidth above which amanda won't start
>#                         backups using this interface.  Note that if
>#                         a single backup will take more than that,
>#                         amanda won't try to make it run slower!
>
>define interface local {
>    comment "a local disk"
>    use 10000 kbps
>}
>
>define interface hme0 {
>    comment "100 Mbps ethernet"
>    use 100000 kbps
>}
>
># You may include other amanda configuration files, so you can share
># dumptypes, tapetypes and interface definitions among several
># configurations.
>
>#includefile "/usr/local/amanda.conf.main"
>
>
>and the other:

And what is this files name?

>org "IS"     # your organization name for reports
>
>inparallel 1           # maximum dumpers that will run in parallel
>
>dumpcycle 8             # the number of days in the normal dump cycle
>runspercycle 0          # the number of amdump runs in dumpcycle days
>tapecycle 4 tapes       # the number of tapes in rotation
>                        # 4 weeks (dumpcycle) times 5 tapes per week
>(just
>                        # the weekdays) plus a few to handle errors
> that # need amflush and so we do not overwrite the full
>                        # backups performed at the beginning of the
>previous
>                        # cycle
>
>holdingdisk hd1 {
>    comment "main holding disk for IS"
>    directory "/data/amanda/IS"        # where the holding disk is
>    use -1              # how much space can we use on it
>#    use 70 Mb           # how much space can we use on it
>                        # a negative value mean:
>                        #        use all space except that value
>    chunksize 2 Gb      # size of chunk if you want big dump to be
>                        # dumped on multiple files on holding disks
>                        #  N Kb/Mb/Gb split disks in chunks of size N
>                        #  0          split disks in INT_MAX/1024 Kb
>chunks
>                        # -N Kb/Mb/Gb dont split, dump larger
>                        #             filesystems directly to tape
>                        #             (example: -2 Gb)
>    }
>
>
>#tapedev "/dev/rmt/0mn"        # the no-rewind tape device to be used
>#da 2.band defekt... #tapedev "/dev/nst1"        # the no-rewind tape
>device to be used
>#tapedev "/dev/nst0"        # the no-rewind tape device to be used
>tapedev "/dev/null"
>tapetype DEC-TZ89             # see tapetype in amanda-shared.conf
>labelstr "^IS[0-9][0-9]*$"    # label constraint regex: all tapes must
>match
>
>infofile "/var/lib/amanda/IS/curinfo"        # database filename
>logdir   "/var/lib/amanda/IS"                # log directory
>indexdir "/var/lib/amanda/IS/index"  # index directory
>
>includefile "/etc/amanda/amanda-shared.conf"

So can we assume the first file you included is amanda-shared.conf?
>
>the first file is used by all configs. The second one is one specific
>config.
>
>> You mentioned you had to take over that server:
>> Do you have any experience with Amanda already or not?
>
>My only experience with amanda comes from experimenting with this
>server.
>
>Thanks,
>Thomas
>
>> Stefan.
>>

-- 
Cheers, Gene
People having trouble with vz bouncing email to me should add the word
'online' between the 'verizon', and the dot which bypasses vz's
stupid bounce rules.  I do use spamassassin too. :-)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2006 by Maurice Eugene Heskett, all rights reserved.