On Thursday 09 February 2006 05:14, Paul Bijnens wrote:
>uwe.kaufmann AT infoconsult DOT nu wrote:
>> I have a general question concerning the vtapes mechanisms and vtape
>> length.
>>
>> I configured an amanda server ("minerva", amanda-2.4.5-2, SuSE 10.0)
>> with "chg-disk" and 14 slots.
>>
>> In the beginning there are a lot of "lev 0 failed", obviously,
>> because I am dealing with >100G data and each slot has only 18G (to
>> fit on tape later on).
>>
>> But where does the "lev 0 FAILED [out of tape]" comes from?. I
>> thought that amanda would calculate properly the relation between
>> data and tapelength.
>
>Amanda does an estimate. And one of the main reasons that initial
>estimates are wrong, is the compression.
>When you have a dumptype with compression enabled, the estimate phase
> does: - Run the dump program that gives an esimate of the
> UNcompressed backup - multiply that value with the expected
> compression that was learned from history
>
>For a DLE that has history (i.e. not a new DLE), you can see that
> value with the command "amadmin Config info", giving an output like:
>
> $ amadmin test info katastrov /space
>
> Current info for katastrov /space:
> Stats: dump rates (kps), Full: 588.0, 603.0, 575.0
> Incremental: 386.0, 230.0, 195.0
> compressed size, Full: 41.6%, 41.6%, 39.1%
> Incremental: 20.6%, 9.1%, 20.1%
> Dumps: lev datestmp tape file origK compK secs
> 0 20060202 Daily-05 114 6985920 2903984 4931
> 1 20060207 Daily-08 95 1415860 284916 1455
> 2 20060209 Daily-10 99 1375710 283200 733
>
>The above example has 3 historical values for compression ratio for a
>full backup and 3 other for incremental backup.
>The new expected compression ratio is calculated as an weighted
> average of those three values, with the newest ratio having the most
> weight. That prediction usually is very good, unless the contents of
> the DLE suddenly change in nature: e.g. zip-files replacing very
> compressable subdirectory trees (what people tend to do when they are
> told to clean up :-) ).
>
>When there is no historical data, amanda assumes a 50% compression.
>That may be tuned by the amanda.conf parameter "comprate".
>
>Now, let's see what happened in your config, below.
>
>> Thank you for any enlightenment of my poor spirit.
>>
>> BTW: Great wiki and docs, thanks to all who spent effort, really
>> helpful!
>>
>> Cheers
>> Uwe
>>
>>
>> Additional info
>> --------------------
>> The report:
>>
>> These dumps were to tape slot3.
>> *** A TAPE ERROR OCCURRED: [[writing file: No space left on
>> device]]. Some dumps may have been left in the holding disk.
>> Run amflush to flush them to tape.
>> The next tape Amanda expects to use is: a new tape.
>> The next new tape already labelled is: slot4.
>>
>> FAILURE AND STRANGE DUMP SUMMARY:
>> hercules /samba/LAGERVERSAND lev 0 FAILED [dumps too big, 109335
>> KB, but cannot incremental dump new disk]
>> hercules //Edi/TC4000/IBDasi lev 0 FAILED [dumps too big, 111909
>> KB, but cannot incremental dump new disk]
>> hercules /KT_EIN_PRO lev 0 FAILED [dumps too big, 576825 KB, but
>> cannot incremental dump new disk]
>> hercules /US_LE lev 0 FAILED [dumps too big, 584480 KB, but
>> cannot incremental dump new disk]
>> hercules /K_EIN_NO4 lev 0 FAILED [dumps too big, 596835 KB, but
>> cannot incremental dump new disk]
>
>The above errors are to be expected indeed, as you already explained.
>
>> hercules /KONST lev 0 FAILED [out of tape]
>
>And one DLE did not fit on the tape.
>(I'll explain below, how we even improve on this, by letting a smaller
>DLE fail instead of this one!)
>
>> STATISTICS:
>> Total Full Incr.
>> -------- -------- --------
>> Estimate Time (hrs:min) 0:01
>> Run Time (hrs:min) 4:14
>> Dump Time (hrs:min) 4:09 4:09 0:00
>> Output Size (meg) 18852.6 18851.7 0.9
>> Original Size (meg) 36786.0 36780.9 5.1
>> Avg Compressed Size (%) 51.2 51.3 16.7
>> (level:#disks ...)
>
>So the Original size (= before compression) was estimated
>and turned out to be 36786 Mbytes.
>Assuming a default compression rate, because these were all
>new DLE's) of 50%, Amanda expected to generate only 18393 Mbytes
>of output. (We should actually use the values of the estimate
>run, which can again be a little bit different than the backup run;
>you find those values in the "amdump.1" file).
>
>Your tapetype was defined as
>
> > define tapetype HARD-DISK {
> > comment "Dump onto hard disk vtapes"
> > length 18432 mbytes
> > }
>
>So, Amanda scheduled up to 18393 Mbytes. The rest was left for the
>next run ("dumps too big... but cannot incremental dump new disk").
>Actually a pretty good approximation.
>
>In real life the compression turns out to be only 51.2%, (comparing
>100 * after / before % -- beware that many compression programs
>report the complement of this: what was saved, instead of what is
> left).
>
>> Filesystems Dumped 24 16 8 (1:8)
>> Avg Dump Rate (k/s) 1290.8 1291.0 412.0
>>
>> Tape Time (hrs:min) 0:10 0:10 0:00
>> Tape Size (meg) 16074.0 16073.1 0.9
>> Tape Used (%) 87.3 87.3 0.0
>> (level:#disks ...) Filesystems Taped 23 15
>> 8)
>> Avg Tp Write Rate (k/s) 28260.4 28271.6 3385.5
>>
>> USAGE BY TAPE:
>> Label Time Size % Nb
>> slot3 0:10 16459733k 87.3 23
>
>This is the size of all the usable backup images. Amanda did
>try to write one more, but while doing that, it bumped into end of
>tape and that backup image is not usable for a restore, so not counted
>in the percentage above.
>
>> NOTES:
>> planner: Adding new disk hercules:/samba/CONTROLLING.
>
>[...]
>
>> planner: Adding new disk hercules:/K_EIN_REST.
>> taper: tape slot3 kb 18874336 fm 24 writing file: No space left on
>> device
>
>Here you see when Amanda hit the end of tape really: at 18874336 kb
>or 18431 Mbyte (compared this to length 18432 Mbyte in the tapetype).
>
>> driver: going into degraded mode because of tape error.
>>
>> DUMP SUMMARY:
>> DUMPER STATS
>> TAPER STATS
>> HOSTNAME DISK L ORIG-kB OUT-kB COMP% MMM:SS KB/s
>> MMM:SS KB/s
>> ----------------------- ------------------------------------------
>> --------------
>> hercules -00/IBDasi 0 FAILED
>> --------------------------------------------------
>> hercules /KONST 0 7004350 2845316 40.6 48:12 984.0
>> FAILED ------
>
>This DLE did make it to holdingdisk, but did not make it to tape.
>As you can see, it was 2.8 Gbyte.
>
>[...]
>
>> (brought to you by Amanda version 2.4.5)
>>
>>
>> --------------------
>> My amanda.conf:
>>
>> org "normal" # your organization name for reports
>> dumpuser "amanda" # the user to run dumps under
>
>[...]
>
>> --------------------
>> The slots so far:
>>
>> minerva:/amandatapes/normal # du -hx --max-depth=1 .
>> 19G ./slot1 <= first amdump
>> 8.5G ./slot2 <= amflush
>> 19G ./slot3 <= second amdump
>> 2.8G ./slot4 <= amflush
>
>So you had to flush 2.8 Gbyte (the one DLE above) to the fourth tape.
>
>The reason was that Amanda's estimate was a few KBytes off: 50%
>instead of 51.2%. Next time amanda will estimate better.
>But there is always a little chance of errors, of course.
>
>But, as I said, we can improve the setup a little bit more.
>Note that the last failed image was 2.8 Gbyte. And that whole image
>must be put on tape again on the next tape. It would be better if
>Amanda put the larger dumps not at the end of the tape, if possible.
>
>Just add the amanda.conf parameter "taperalgo largestfit" to
>the setup. For a complete explanation, also read:
>
>http://wiki.zmanda.com/index.php/Filling_a_tape_to_100%25
--
Cheers, Gene
People having trouble with vz bouncing email to me should add the word
'online' between the 'verizon', and the dot which bypasses vz's
stupid bounce rules. I do use spamassassin too. :-)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
|
