Amanda-Users

Re: incremental with gnutar bogusly dumping old files

2007-09-27 19:29:31
Subject: Re: incremental with gnutar bogusly dumping old files
From: John E Hein <jhein AT timing DOT com>
To: "Dustin J. Mitchell" <dustin AT zmanda DOT com>
Date: Thu, 27 Sep 2007 17:29:33 -0600
Dustin J. Mitchell wrote at 16:15 -0500 on Sep 27, 2007:
 > On 9/27/07, John E Hein <jhein AT timing DOT com> wrote:
 > > Indeed.  I was looking for the format documented in the .info
 > > pages for gnutar, but couldn't find it.

[by the way, somewhere between gtar 1.15.1 and 1.18.1, the info
page added this format information]


 > Not to toot my own horn, but there's a tool to display/edit these
 > numbers in the tar scripts/ directory.  It's available via CVS at
 > http://cvs.savannah.gnu.org/viewvc/*checkout*/tar/scripts/tar-snapshot-edit?revision=1.2&root=tar
 > The hand- (or sed-)editing of these files works fine, but the script
 > handles the various flavors, including those which use embedded NUL
 > bytes, etc.

Nice.  I'll take a look at it the next time I need to do this.


 > > Supporting a per-DLE mode to [temporarily] ignore a change in devno in
 > > amanda should be easy.
 > 
 > See http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html
 > -- it's actually not so easy, unfortunately.

Why not?... when amanda copies the old level N - 1 snapshot file to
level N, it can convert using sed or your tool or C.  Then when gtar
uses the new level N file for --listed-incremental, it will be
comparing with the new devno that amanda sed'd in.


 > > And maybe adding a "devno changed" check in amcheck would be nice -
 > > the client has the gnutar-list files, so it would require some work in
 > > the client side of amcheck.
 > 
 > This might be an interesting addition to the GNU Tar application, when
 > it's complete.  I'm not sure if there's a good way to stuff it into
 > the current implementation.

I think a check is higher level than gtar.  What would you do?  Have
gtar grow yet another -- option, say
--check-for-incremental-devno-change?  Or perhaps spew a warning if
the devno for an otherwise identical file is different?  Unfortunately
it's harder to generically code a check into gtar if not using
--one-file-system (which allows you to assume devno is the same
throughout the file).  If it's got a mix of devnos, you might have to
check the entire snapshot file to see if any device numbers have
changed.

On the other hand, gtar is where all the knowledge about the
gnutar-list file is kept.  If the check is done in amanda, you can
make the assumption that all devno's are the same since we know we use
--one-file-system.  But now you have add specific knowledge of the
gtar file format to amanda, and I agree that's not so good (and this
point also applies to the sed above).

Maybe the gtar distribution could grow a utility (optionally
installed) to do such a check.  That way it can be maintained
alongside gtar and not pollute the utility with arcane options only
needed by things like amcheck.  Beside peace in our time, what the
world needs is a gtar-utils package ;)


I could definitely see adding a gtar option that could be used to
ignore devno changes, say --ignore-devno-change, that is used with
--listed-incremental.  Then you could avoid having to do the sed
described above.  And that should be a tiny code change to gtar (in
theory - </me says without looking at the source>).


 > > For us, this was a large raid, so suffering through a couple days of
 > > huge incremental dumps was painful (particularly while getting 'dumps
 > > way too big' for some DLEs) and threw the balance out of whack.
 > 
 > Gene had similar problems -- it really throws Amanda off its stride.

Indeed.