Amanda-Users

Re: More Debian Amanda package upgrade issues (this time with amdump)

2006-11-15 12:04:25
Subject: Re: More Debian Amanda package upgrade issues (this time with amdump)
From: Jean-Louis Martineau <martineau AT zmanda DOT com>
To: Frank Smith <fsmith AT hoovers DOT com>
Date: Wed, 15 Nov 2006 11:52:58 -0500
Frank Smith wrote:
Thanks to Jean-Louis for the reply on my amcheck issues.

I got some really strange results from my first amdump run last
night after my upgrade.

I updated my home Amanda server and clients (Debian etch
packages) from 2.5.0p2-2.1 to 2.5.1p1-2, and tar from 1.15.91-2 to
1.16-1.

FAILURE AND STRANGE DUMP SUMMARY:
  server   /  lev 2  STRANGE
  clientA  /  lev 0  FAILED [/bin/tar returned 1]
  clientA  /  lev 0  FAILED [/bin/tar returned 1]
...

Why is clientA / reported twice? Probably because it failed twice,
see comments farther down.

FAILED AND STRANGE DUMP DETAILS:

/--  server / lev 2 STRANGE
sendbackup: start [k6:/ level 2]
sendbackup: info BACKUP=/bin/tar
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/tar -f - ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
? gtar: ./hdb/holdingdisk/daily/20061031053001: Cannot savedir: No such file or 
directory
? gtar: ./hdb/holdingdisk/daily/20061031053001/: Warning: Cannot stat: No such 
file or directory
? gtar: ./hdb/holdingdisk/daily/20061101053002: Cannot savedir: No such file or 
directory
? gtar: ./hdb/holdingdisk/daily/20061101053002/: Warning: Cannot stat: No such 
file or directory
? gtar: ./hdb/holdingdisk/daily/20061102053001: Cannot savedir: No such file or 
directory
....
many more lines like these.
For one thing, /hdb is is a mount point for a second hard disk, so
tar has been escaping the root filesystem, The other thing is that
the directories were all flushed to tape the day before, so they
ceased to exist before the time of last night's backup.  Is Amanda
trying to check everything in the level 1 index file instead of
just backing up what currently exists?
It's a tar bug that it cross filesystem boundaries, you will have to use exclude as workaround.
OK, why did it try backing up / on clientA twice?  It didn't just
report it twice, it ran it twice (notice the slightly different
sizes and a changed temp file during one run).
Because it failed.
Now the really odd part:
NOTES:
  planner: Last full dump of clientA:/home/user1 on tape  overwritten in 1 run.
  planner: Last full dump of clientA:/home/user2 on tape  overwritten in 1 run.
  planner: Last full dump of clientA:/home/user3 on tape  overwritten in 1 run.
  planner: Last full dump of clientA:/home/user4 on tape  overwritten in 1 run.
  planner: Last full dump of clientA:/home/user5 on tape  overwritten in 1 run.
  planner: Last full dump of clientB:/ on tape  overwritten in 1 run.
  taper: tape_rdlabel: tape open: /dev/nst0: Input/output error
Bug, your last full is on holding disk, it should not report this mesage, try the overwrite.diff attached patch.
I've got a tapecycle of 45, and ususally flush once a week, so
why does Amanda thing I'm about to overwrite most of my level 0s?
And why are the tape names blank?  The taper error is expected, I
didn't have a tape in the drive.

  big estimate: clientA /home/user8 1
                est: 64k    out 1k
  big estimate: clientA /home/user2 1
                est: 1056k    out 107k
  big estimate: clientA /home/user4 1
                est: 64k    out 18k
  big estimate: clientA /home/user5 1
                est: 64k    out 5k
  big estimate: clientA /home/user6 1
                est: 64k    out 1k
  big estimate: clientA /home/user7 1
                est: 64k    out 1k
  small estimate: server / 2
                  est: 224128k    out 249679k
  small estimate: clientB / 1
                  est: 1856k    out 3375k

I've never seen this estimate section before, guess I need to go read
up on it and see what it means.
It only report if the estimate is far form the real size, in general, nothing to worry about.
I think it should be tuned for small backup.
DUMP SUMMARY:
                                               DUMPER STATS                 
TAPER STATS
HOSTNAME     DISK               L  ORIG-kB   OUT-kB  COMP%  MMM:SS    KB/s 
MMM:SS   KB/s
--------------------------------- ---------------------------------------- 
-------------
clientA  /                  0      N/A  2775998    --     N/A     N/A    N/A    
N/A  PARTIAL


What does 'PARTIAL' mean?  ClientA / was reported as about 4.1GB
during both its failed runs above, so is the 2.7GB reported in the
dump summary the compressed size (the dumptype uses compress-fast)
or was it in reality a partial dump?
PARTIAL is when amanda think the backup is not complete (dumper reported FAIL because tar exited with 1).
I've really been surprised by the number of issues I've encountered
making a minor version change (2.5.0->2.5.1), especially since the
upgrade from 2.4.2 to 2.5 went fairly smoothly.  I'm sure part of
the problems are a result of tar 1.16.  One relevant changelog entry
for tar (1.15.92) is:
* After creating an archive, tar exits with code 1 if some files were
  changed while being read.  Previous versions exited with code 2 (fatal
  error), and only if some files were truncated while being archived.

Does this mean that many backups will now be marked as FAILED instead
of STRANGE?
Yes
Could you try the amanda-tar-1.1.6.diff attached patch?
Since older releaser never returned 1, I think it is safe to ignore return status of 1.

Jean-Louis

diff -u -r --show-c-function --exclude-from=amanda.diff 
amanda-2.5.2alpha.new/client-src/sendbackup.c 
amanda-2.5.2alpha.new.tar/client-src/sendbackup.c
--- amanda-2.5.2alpha.new/client-src/sendbackup.c       2006-10-20 
07:55:19.000000000 -0400
+++ amanda-2.5.2alpha.new.tar/client-src/sendbackup.c   2006-11-15 
11:25:29.000000000 -0500
@@ -605,6 +605,11 @@ check_status(
     }
 #endif
 
+    if(pid == tarpid) {
+       if(ret == 1) {
+           rc = 0;
+       }
+    }
 #ifdef IGNORE_TAR_ERRORS
     if(pid == tarpid) {
        /*
diff -u -r --show-c-function --exclude-from=amanda.diff 
amanda-2.5.2alpha.new/server-src/planner.c 
amanda-2.5.2alpha.new.overwrite/server-src/planner.c
--- amanda-2.5.2alpha.new/server-src/planner.c  2006-11-08 07:03:31.000000000 
-0500
+++ amanda-2.5.2alpha.new.overwrite/server-src/planner.c        2006-11-15 
11:48:45.000000000 -0500
@@ -782,7 +782,7 @@ setup_estimate(
     /* adjust priority levels */
 
     /* warn if dump will be overwritten */
-    if(ep->last_level > -1) {
+    if(ep->last_level > -1 && info.inf[0].label[0] != '\0') {
        overwrite_runs = when_overwrite(info.inf[0].label);
        if(overwrite_runs == 0) {
            log_add(L_WARNING, "Last full dump of %s:%s "