I've just been caught out by something which is either

(a) a limitation of AMANDA and/or tar/dump;
(b) a bug.

In summary: when a directory is *renamed* the files underneath it are
not considered to have changed: this means that a non-level-0 backup
following the directory rename only backs up the new directory entry,
not the files inside the directory.  This means that, if the disk fails
following such a non-level-0 backup and requires a recovery, the full
recovery will *not* recover the files inside the directory: the new
directory will be recreated by amrecover, but will be empty.

Let me demonstrate by means of a simple example:

On Friday 03.03.2006, I do the following:

$ ls /home/davee/stata/w
total 24
-rw-r--r--  1 davee users    68 2006-02-28 16:25 Makefile
-rw-r--r--  1 davee users   479 2006-03-03 15:44 w.csv
-rw-r--r--  1 davee users   224 2006-01-19 19:21 w.do
-rw-r--r--  1 davee users 11147 2006-03-03 15:44 w.log

These are files last edited on 03.03.2006 and on this night a level 0
backup took place.  The index for this directory contains the directory
and all its files as expected:

# zgrep 'davee/stata' 20060303_0.gz
/davee/stata/
/davee/stata/w/
/davee/stata/w/Makefile
/davee/stata/w/w.csv
/davee/stata/w/w.do
/davee/stata/w/w.log

On Monday 06.03.2006, no further changes are made to the files and a
level 1 backup tapes place that evening.  The index for that backup
includes the directory entry as expected, but none of the files:

# zgrep 'davee/stata' 20060306_1.gz
/davee/stata/
/davee/stata/w/

On Tuesday 07.03.2006, I rename the directory 'w' to 'w2', but make no
changes to the files inside the directory.  The file structure now looks
like this:

$ mv w w2
$ ls -l /home/davee/stata/w2/
total 24
-rw-r--r--  1 davee users    68 2006-02-28 16:25 Makefile
-rw-r--r--  1 davee users   479 2006-03-03 15:44 w.csv
-rw-r--r--  1 davee users   224 2006-01-19 19:21 w.do
-rw-r--r--  1 davee users 11147 2006-03-03 15:44 w.log

On the evening of 07.03.2006, a level 2 backup takes place and the index
contains the new directory entry:

# zgrep 'davee/stata' 20060307_2.gz
/davee/stata/
/davee/stata/w2/

The files in the directory are not changed which is presumably why the
files are not included in this index.  (Importantly, though, the 
directory entry for davee/stata/w has now gone: this will matter during
recovery.)

SO ... (and this is big issue here) ... if the disk on which these
files are stored fails at this point, a recovery DOES NOT RECOVER ALL
FILES.

I guessing that what happens is that the recovery process (which prompts
for the tapes from 20060303, 20060306 and 20060307 in turn) does the
following:

- restores the contents of the level 0 backup from 20060303 which would
  leave the filesystem as follows:
  $ ls /home/davee/stata/w
  total 24
  -rw-r--r--  1 davee users    68 2006-02-28 16:25 Makefile
  -rw-r--r--  1 davee users   479 2006-03-03 15:44 w.csv
  -rw-r--r--  1 davee users   224 2006-01-19 19:21 w.do
  -rw-r--r--  1 davee users 11147 2006-03-03 15:44 w.log

- restores the contents of the level 1 backup from 20060306 which won't
  make any further changes to this part of the filesystem;

- restores the contents of the level 2 backup from 20060307 which
  introduces an emtpy directory davee/stata/w2 and REMOVES davee/stata/w
  because it DOESN'T EXIST ANYMORE, leaving the filesystem as follows:
  $ ls /home/davee/stata
  drwxr-xr-x  2 davee users    4096 2006-03-03 15:44 w2
  $ ls /home/davee/stata/w2
  (nothing)

Clearly the solution in this case is to perform a separate recovery of
the level 0 backup and manually put the files back in place.  However,
in the context of a massive recovery of many thousands of files
following a disk failure, events such as the one described above (which
is merely a small worked example) will not be noticed.

The point of my illustration is that even though one has all the 
appropriate tapes for the recovery process, running an amrecover and
feeding in the tapes one by one will NOT RESTORE ALL FILES, if both of
the following circumstances are true:

- the most recent backup prior to disk failure was not a level 0 backup;

- directories were renamed since the last level 0 backup.

Thoughts/opinions here?  BTW the AMANDA server runs Debian/Woody
(2.4.2p2-4) and the client being backed up above runs Debian/Sarge AMD64
(2.4.4p3-3).

Cheers,

Dave.
-- 
Dave Ewart
davee AT ceu.ox.ac DOT uk
Computing Manager, Cancer Epidemiology Unit
Cancer Research UK / Oxford University
PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370
Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc
N 51.7518, W 1.2016

signature.asc
Description: Digital signature