Re: Virtual tape to Real tape cycles

Quoting "Dustin J. Mitchell" <dustin AT zmanda DOT com>:

On Fri, Jun 15, 2007 at 04:23:57PM -0400, Charlie Reitsma wrote:
As I read through the man pages I am finding hints that it has beenthought of. In the amdd man page "This may be used for debugging orto convert from one format to another."
I began looking at the contents of various directories and filesbased on comments found in Gene's Amanda Helper scripts. The indexis just a list of files contained in the imagescontained on the the tape. It wouldn't change at all as the imagesaren't being modified. I couldn't find anything that maps an imagefile to a particular location on a tape. Itappears that the contents of a virtual tape need no furtherprocessing before moving them to a real tape. Copying those imagesonto a tape and moving the index, curinfo andtapelist date to another cycle seems like it might bestraightforward. Since all the image sizes are known in advancethey might even be reordered more optimally to fit on tape. Ididn't like Gene's suggestion that the virtual tape size and thereal tape size should be kept the same. We might even choose to putdifferent file systems on different tapes as
they may go to different offsite locations for security reasons.
Restores appear to collect the images associated with a particularfile system and give them back to the appropriate program toextract the file(s) requested. That process wouldn't
change just because the media changed.

I'll take a look at the Developer Documentation to continue my exploration.
Charlie --

This sounds like a great project, and I'm excited you're working on it.
Please feel free to consult amanda-hackers with any further technical
questions or musings you might have.

The functionality you're thinking about is a form of hierarchical
storage management, with data migrating between storage media based on
enterprise policies and needs.  It's a kind of functionality that would
be welcome in Amanda, and that has been in the back of most developers'
minds for some time.

The new Device API will be an important step in this direction, since
(among other benefits) it will allow us to treat all storage media the
same way.  With a little reworking, taper can then become a generic data
migration tool.  The major unresolved issues are:
 - policy (how does Amanda know when to migrate what, and where)
 - metadata updates (which you mentioned above)

Device API: http://wiki.zmanda.com/index.php/Device_API

Once the Device API is released, the immediate next step is to treat
holding disk as a device.  This represents Amanda's existing
functionality -- dumping to holding, then spooling to tape -- as a data
migration, opening the door to more complex migrations down the road.

Dustin

--
        Dustin J. Mitchell
        Storage Software Engineer, Zmanda, Inc.
        http://www.zmanda.com/

I can report that my basic expectation of success has been met. I canachieve what I want "manually" as follows:

1) copy the image files (new terminology: collection) from the tape tothe holding directory. I achieved this by manually copying it from thevirtual tape to a directory I created on the holding disk named as thedate of the original backup. I had to strip off the leading filenumfrom each file name. I copied the most recent level 0, level 1 andlevel 2 this way. amrestore -r is expected to provide the same result.

2) I created a new config with new virtual tapes and told amflush touse this config to write the image files to tape.

3) I changed the stats lines in the curinfo file of the new config tomatch the history lines for those image files from the originalconfig. I think if I had copied these history lines to the newconfig's curinfo file before the flush then amflush would have writtenthe stats lines properly.

4) I copied the index information from the original config for each ofthe image files to the new config's index.

End result is amrecover is able to recover from any of the threeimages on the new tape.


So, it can be done. Now to make it happen in a less manual way.

Charlie Reitsma
x6642