On 2007-08-01 10:12, fedora wrote:

When you believe the estimate is taking too long for some DLE's (e.g.
because the filesystem is one of those bad cases like containing lots
of small files) than think about switching to a faster, but less
accurate, estimate methode, like calcsize (see "man amanda.conf").

Many thanks. By the way, what does it mean by "less accurate"? In term of
what?

In terms of the estimate numbers produced :-)

levels of all the DLE's, and deciding for a mix of full (level 0)
backups, and incremental backups (some one level higher than last run).
For each DLE, Amanda can do estimates for level 0, level N (the level
from the previous run), and level N+1.  Which estimates are needed
is also planned -- Amanda does not always need 3 estimates for each DLE.

Amanda has several options to get those estimates.
The default method is running the backup program on the client
trowing away the backup data, and scanning the stderr of the program
when it says now much data that is.
For "dump" it starts a backup, redirecting backup data to /dev/null
and waits until dump outputs the estimated size on stderr, and then
kills the dump. This usually takes only a few minutes at most.
For "gnutar" backups, it runs gnutar with backup data output redirected

because it does need to read all the data in each file.
However gnutar does detect the output redirection to /dev/null and
makes some optimisation like not really writing any bytes at all, making
it still faster than a real backup.

Calcsize is a program that traverses the tree, but essentially only
stat()s every file, adding the filesizes of each, plus some guess about
how much gnutar would take to store the filename and attributes, and
rounding to 512 block sizes, as gnutar does.
It does not read the file contents.  The resulting number is usually
pretty close to what gnutar would generate.  But "sparse files" (files
with holes in them, having a large filesize, but occupying much
less disk blocks) and the small errors in filename/attribute and
rounding calculations result in less accurate numbers.

For completeness, there is also a third method, using only numbers
from previous Amanda runs and using statistics to predict this run.
If the filesystem is reasonably stable, these numbers are good enough
to use, but if the directory tree had a major reorganisation that day
Amanda does not know about that, and still plans as if nothing
has changed.

The accuracy of those estimates impact how good Amanda can plan
the backup.  They need to be more accurate when it is planning
for a run that fills your tape to 100%.  It could result in
a real backup run having 101% of the estimated data, and that means
that the last backup image will not fit on the tape, and be left
in holdingdisk if possible.  Experience learns that it is usually
a very large image that is written as last one on tape.
If the tapes on the other hand are usually only 50% full, then
Amanda may plan with figures that are half the real size, and
still have no big impact on that run.

(to be really complete, the estimate numbers generated by dump,
gnutar or calcsize are multiplied by a statically computed compression
ratio for that DLE;  this also adds some inacurracy for the DLE's
that use software compression).

--
Paul Bijnens, xplanation Technology Services        Tel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM    Fax  +32 16 397.512
http://www.xplanation.com/          email:  Paul.Bijnens AT xplanation DOT com
***********************************************************************
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out          *
***********************************************************************