On 01/15/10 08:48, John Drescher wrote:
> I think others were looking for the more complicated task of deleting
> file volumes automatically after their retention period had expired.


Having now finished the OS reinstallation on my main workstation, I've
had time to look at this problem.  Here's my working example.

To start with, let's take a look at my Pool setup:

Pool {
  Name = Scratch
  Storage = babylon4-sd
  Pool Type = Backup
}

Pool {
  Name = Full-Disk
  Storage = babylon4-sd
  Pool Type = Backup
  Recycle = no
  Recycle Oldest Volume = no
  Recycle Current Volume = no
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 2 months
  Maximum Volume Jobs = 0
  Volume Use Duration = 23h
  Label Format =
"FULL-$Year${Month:p/2/0/r}${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
  RecyclePool = Scratch
}

Pool {
  Name = Diff-Disk
  Storage = babylon4-sd
  Pool Type = Backup
  Recycle = no
  Recycle Oldest Volume = no
  Recycle Current Volume = no
  AutoPrune = yes
  Volume Retention = 2 months
  Maximum Volume Jobs = 0
  Volume Use Duration = 23h
  Label Format =
"DIFF-$Year${Month:p/2/0/r}${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
  RecyclePool = Scratch
}

Pool {
  Name = Incr-Disk
  Storage = babylon4-sd
  Pool Type = Backup
  Recycle = no
  Recycle Oldest Volume = no
  Recycle Current Volume = no
  AutoPrune = yes
  Volume Retention = 1 month
  Maximum Volume Jobs = 0
  Volume Use Duration = 23h
  Label Format =
"INCR-$Year${Month:p/2/0/r}${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
  RecyclePool = Scratch
}

Pool {
  Name = Full-Tape
  Storage = babylon5-sd
  Pool Type = Backup
  Recycle = yes
  Autoprune = yes
  Volume Retention = 365d
  Recycle Oldest Volume = yes
  Recycle Current Volume = yes
  Label Format = "ARCH-"
  Maximum Volumes = 9
}


As you see, there are three separate Disk pools, divided by level, with
different retention periods, all of which recycle into the Scratch pool.
 Each disk volume is used for a single day's backups, then marked used,
enabling good granularity of disk usage.  There is also a separate tape
pool, volumes in which are recycled back into the same Pool ready for reuse.


Now, with that, on to the autoclean script:


#!/usr/bin/perl
use strict;
use Getopt::Long;
use IPC::Open2;
use IO::Handle;


my $bconsole = '/opt/bacula/bin/bconsole';
my (%opts, @purged, $pid);

GetOptions(\%opts,
           'verbose|v',
           'test');

my ($IN, $OUT) = (IO::Handle->new(), IO::Handle->new());

$pid = open2($OUT, $IN, $bconsole);

if (scalar (@purged = check_volumes()))
{
    printf("Bacula reports the following purged volumes:\n\t%s\n",
           join("\n\t", @purged)) if ($opts{verbose});
    my $deleted = delete_volumes(@purged);
    print "$deleted volumes deleted.\n" if ($opts{verbose});
}
elsif ($opts{verbose})
{
    print "No purged volumes found to delete.\n";
}

print $IN "exit\n";
waitpid($pid, 0);

exit (0);


sub check_volumes
{
    my $dividers = 0;
    my (@purged, @row);

    print $IN "list volumes pool=Scratch\n";
    for (;;)
    {
        my $resp = <$OUT>;
        last if ($resp =~ /No results to list./);
        $dividers++ if ($resp =~ /^[\+\-]+$/);
        last if ($dividers == 3);
        @row = split(/\s+/, $resp);
        push (@purged, $row[3]) if ($row[5] eq 'Purged');
    }

    return (@purged);
}


sub delete_volumes
{
    my $volume_dir = '/spool/bacula/';
    my $count = 0;

    foreach my $vol (@_)
    {
        my $l;
        my $file = $volume_dir.$vol;

        print "Deleting volume $vol from catalog ... " if ($opts{verbose});
        print $IN "delete volume=$vol yes\n";
        $l = <$OUT>;
        $l = <$OUT>;
        print "Done.\nDeleting volume $file from disk ... " if
($opts{verbose});
        if (-f $file)
        {
            $count++;
            unlink ($file);
        }
        print "Done.\n" if ($opts{verbose});
    }

    return ($count);
}



This script leaves volume expiration entirely to Bacula.  It asks Bacula
for a list of volumes that have been expired and moved into the Scratch
pool.  If there are none, it returns immediately; otherwise, it
double-checks to make sure that each of those volumes has been purged,
builds a list of purged volumes, then deletes the purged volumes first
from the Catalog, then from disk.


I haven't yet set this up as an Admin job, but I have tested it
thoroughly and verified that it works, including properly exiting if the
Scratch pool is empty.


-- 
  Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
  alaric AT caerllewys DOT net   alaric AT metrocast DOT net   phil AT 
co.ordinate DOT org
         Renaissance Man, Unix ronin, Perl hacker, Free Stater
                 It's not the years, it's the mileage.

------------------------------------------------------------------------------
Throughout its 18-year history, RSA Conference consistently attracts the
world's best and brightest in the field, creating opportunities for Conference
attendees to learn about information security's most important issues through
interactions with peers, luminaries and emerging and established companies.
http://p.sf.net/sfu/rsaconf-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users