> > In the old tape days, I used to force a level-0 dump when I wanted to do
> > that. This was a pain and never very satisfying. So I wrote a perl script
> > that will extract the most recent full dump for every disk partition out
> > of the vtapes.
>
> What a neat idea, and one that works best with vtape as the daily backup.
>
> Would you consider writing up your experiences and techniques?
Sure, although maybe the script itself says it best? :-) I'll append it
below.
> I'm guessing that for recovery from the archived dumps you would not
> use amanda's indexing features.
Yes, that's right. The script builds a table-of-contents file for each
dump and puts that on the disk as well (this is the part which assumes
my dumps are done with gtar). There might be a smarter way to extract
that info from amanda's existing indices instead.
I also liked someone's idea of copying amanda's indices to the off-site
media and having them avaiable for the restore. If someone would like
to extend this to have that ability, that would be neat.
> Have you had to do any recoveries from the archived dumps rather
> than the vtapes?
I've done a lot of test restores, and they work fine (using
"dd bs=32k skip=1 ..."). I haven't had a call to use the offsites
to restore anything under pressure yet. If I never ever get to use
my offsite backups and this is all a waste of time, I'll be a happy
man :-)
Mark
--
Mark Costlow | Southwest Cyberport | Fax: +1-505-232-7975
cheeks AT swcp DOT com | Web: www.swcp.com | Voice: +1-505-232-7992
"Education is never a waste" - Viscount du Valmont
--------------------- amoffsite ------------------------------------------
#!/usr/local/bin/perl
#
# $Id: amoffsite,v 1.3 2005/01/12 06:31:02 cheeks Exp $
#
# Program: amoffsite
# Author: Mark Costlow <cheeks AT swcp DOT com>
# Date: Jan, 2005
#
# This program prepares an offsite dump. It finds the most recent
# level-0 dump for each disk partition in an Amanda config. These are in the
# "virtual tapes" of the large RAID that we use for nightly backups. It
# copies those files to the "offsite" disk.
#
# The idea is that someone will run this script once a month to copy the
# offsite dumps to a disk, then pull that disk out of the disk array and take
# it home with them. If something happens to our disk array, or if we need to
# restore a file older than what we have on-site, the offsite disk(s) should
# save us.
#
#
# This is the first working version of this program -- there's plenty of room
# for improvement. If you have suggestions or fixes, please send them to
# cheeks AT swcp DOT com.
#
# As with the rest of amanda, THIS SOFTWARE IS BEING MADE AVAILABLE ``AS-IS''.
# It might work for what you want but it might also delete every backup you
# ever made, cause your computer to melt, and your hair to catch fire.
#
#
# Usage: offsite-dump configname
#
# configname is the name of an amanda config.
#
$| = 1; # No STDOUT buffer
$gzip = "/usr/bin/gzip";
$tar = "/usr/bin/gtar";
require 'getopts.pl';
Getopts('hzivd:');
$defroot = "/amanda/offsite";
$offroot = $opt_d ? $opt_d : $defroot;
$docomp = $opt_z;
$doidx = $opt_i;
$verbose = $opt_v;
$cat = $docomp ? "/usr/bin/zcat" : "/bin/cat";
$config = shift;
if ($config eq '' || $opt_h) {
print "Usage: amoffsite [-d dir] configname\n\n";
print "configname is the name of an amanda config.\n\n";
print "-h This help message.\n";
print "-d dir Use dir instead of default target directory [$defroot].\n";
print "-z Compress the dump files with gzip.\n";
print "-i Generate an index for each dump file.\n";
print "-v Be verbose about progress.\n";
exit 1;
}
%disklist = &read_disklist($config);
%tapelist = &find_tapes($config, \%disklist);
$vtape_dir = &get_vtape_dir($config);
%slots = &find_slots($vtape_dir, \%tapelist);
©_files($config, $vtape_dir, $offroot, \%tapelist, \%slots);
exit 0;
sub read_disklist {
local($config) = @_;
local(@lines, $line, $host, $disk, $lnum, $dspec, %dlist);
print "Reading disk list ..." if $verbose;
$/ = ""; # paragraph input mode
open(CF, "amadmin $config disklist |") || die "'amadmin $config disklist:
$!\n";
while (<CF>) {
@lines = split(/\n/, $_);
$host = $disk = '';
foreach $line (@lines) {
if ($line =~ /host (\S+):$/) {
$host = $1;
} elsif ($line =~ /disk (\S+):$/) {
$disk = $1;
} elsif ($line =~ /^line (\d+)/) {
$lnum = $1;
}
}
if ($host eq '' || $disk eq '') {
print STDERR "ERROR processing line '$lnum'. host=$host disk=$disk\n";
exit 1;
}
$dspec = "${host}:${disk}";
$dlist{$dspec} = 1;
}
close(CF);
print " done\n" if $verbose;
$/ = "\n"; # back to line input mode
return %dlist;
}
# $dlist is a hashref
sub find_tapes {
local($config,$dlist) = @_;
local(@lines, %tlist, $line, $host, $disk, $dspec, $tspec);
local($level, $tape, $fnum, $status, $h, $d, $date);
local(%zdates, @dtmp, $nd);
$nd = scalar(keys %$dlist);
print "Finding tapes for $nd disks ..." if $verbose;
foreach $dspec (sort keys %$dlist) {
print "." if $verbose;
($host,$disk) = split(/:/, $dspec);
%zdates = ();
open(AM, "amadmin $config find $host $disk |")
|| die "'amadmin $config find $host $disk: $!\n";
while (<AM>) {
($date, $h, $d, $level, $tape, $fnum, $status) = split;
next if ($date !~ /^\d\d\d\d-\d\d-\d\d/);
next if ($status ne 'OK');
next if ($level ne '0');
# If we're still here, then this is a successful level-0 dump
$tspec = "${tape}:${fnum}";
$zdates{$date} = $tspec;
}
close(AM);
if (scalar(keys %zdates) == 0) {
print STDERR "ERROR: no level-0 for $dspec -- Continue? [n] ";
chomp($ans = <STDIN>);
if ($ans !~ /y/i) {
exit 1;
} else {
$tlist{$dspec} = 'SKIP';
next;
}
}
# Sort the list of level-0 dates, and then take the last one, which
# should be the most recent.
@dtmp = sort keys %zdates;
$date = pop(@dtmp);
$tlist{$dspec} = $zdates{$date};
$dumpdate{$dspec} = $date;
}
print " done\n" if $verbose;
# Return the modified disklist
return %tlist;
}
# $tlist is a hashref
sub find_slots {
local($tdir, $tlist) = @_;
local(%slots, $slot, $tape, $tspec, $fnum);
local($files, @files, $nf, $sdir);
chdir ($tdir) || die "Can't cd to $tdir: $!\n";
foreach $tspec (sort values %$tlist) {
next if ($tspec eq 'SKIP');
($tape,$fnum) = split(/:/, $tspec);
next if (defined $slots{$tape});
# There should be a file called slotN/00000.TAPENAME. This tells us what
# directory to find the dump files in for this "tape".
$files = `find . -name 00000.$tape -print`;
@files = split(/\n/, $files);
$nf = scalar(@files);
if ($nf != 1) {
print STDERR "ERROR: wanted exactly 1 tape with label $tape, but found
$nf. Abort.\n";
exit 1;
}
$sdir = $files[0];
$sdir =~ s|^\./||;
$sdir =~ s|/.*||;
$slots{$tape} = $sdir;
}
return %slots;
}
sub get_vtape_dir {
local($config) = @_;
local($str, $file, $dir);
chomp($str = `amgetconf $config tapedev`);
($file, $dir) = split(/:/, $str);
if ($file ne 'file') {
print STDERR "ERROR: failed to find a 'file' tape device in $config
config.\n";
exit 1;
}
return $dir;
}
#
# The %tlist and %slots hashes are passed by reference to this function
#
sub copy_files {
local($config, $src_root, $dst_root, $tlist, $slots) = @_;
local(@disks, $ndisks, $dnum, $ddir, $dfile, $sfile);
local($now,$datestamp,$target_dir);
local($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst);
# Make a target directory named after the config and today's date
$now = time();
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($now);
$datestamp = sprintf("%04d%02d%02d", $year+1900, $mon+1, $mday);
$target_dir = "${dst_root}/${config}-${datestamp}";
if (-e "$target_dir") {
print "$target_dir already exists. Continue? [n] ";
chomp($ans = <STDIN>);
if ($ans !~ /y/i) {
exit 1;
}
} else {
mkdir($target_dir,0700);
if (! -e "$target_dir") {
print STDERR "ERROR: Failed to create $target_dir -- abort.\n";
exit 1;
}
}
&init_dumpdate_file($target_dir);
@disks = sort keys %$tlist;
$ndisks = scalar(@disks);
$dnum = 0;
foreach $disk (@disks) {
$dnum++;
if ($tlist->{$disk} eq 'SKIP') {
print "Skipping $dnum/$ndisks $disk.\n";
next;
}
($host, $part) = split(/:/, $disk);
$part =~ s|/|_|g;
($tape, $fnum) = split(/:/, $tlist->{$disk});
$slot = $slots->{$tape};
&add_dumpdate($target_dir,$disk,$dumpdate{$disk});
$sfile = sprintf("%s/%s/%05d.%s.%s.0",
$src_root, $slot, $fnum, $host, $part);
$ddir = "${target_dir}/${host}";
if (! -d "$ddir") {
mkdir($ddir,0700);
}
$dfile = "${ddir}/${host}.${part}.0";
# printf("%5s %s\n", &getsize($sfile), $sfile);
$size_spec = &getsize($sfile);
print "Copying $size_spec file $dnum/$ndisks -> ${host}.${part}.0 ";
# system ("cp $sfile $dfile");
if ($docomp) {
$dfile = "${dfile}.gz";
system ("$gzip -v < $sfile > $dfile");
} else {
system ("cp $sfile $dfile");
print "\n";
}
if ($doidx) {
$ifile = "${ddir}/${host}.${part}.0.INDEX.gz";
print "Creating index file $ifile ...";
system ("$cat $dfile | dd bs=32k skip=1 | $tar tvf - | $gzip > $ifile");
print " done.\n";
}
}
}
#
sub getsize {
local($file) = @_;
local($b, $k, $m, $g, $t);
local($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks)
= stat($file);
$b = $size;
if ($b > 1024) {
$k = $b / 1024;
} else {
return sprintf("%.2f B", $b);
}
if ($k > 1024) {
$m = $k / 1024;
} else {
return sprintf("%.2f KB", $k);
}
if ($m > 1024) {
$g = $m / 1024;
} else {
return sprintf("%.2f MB", $m);
}
if ($g > 1024) {
$t = $g / 1024;
} else {
return sprintf("%.2f GB", $g);
}
# If we get this far, the file is over 1 TB. Wow.
return sprintf("%.2f TB", $t);
}
sub getbytes {
local($file) = @_;
local($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks)
= stat($file);
return $size;
}
sub init_dumpdate_file {
local($dir) = @_;
open (DDF, "> $dir/dumpdates.txt") || die "Can't open $dir/dumpdates.txt:
$!\n";
close(DDF);
}
sub add_dumpdate {
local($dir,$disk,$date) = @_;
open (DDF, ">> $dir/dumpdates.txt") || die "Can't open $dir/dumpdates.txt:
$!\n";
print DDF "$date\t$disk\n";
close(DDF);
}
|
