Thanks Kjetilho,
Sounds ingenious!
I think it'd need some tweaking to delay putting each tablespace into backup
mode until Bacula is ready to back it up - one of the problems I have is that
we can't put all the tablespaces into backup mode at the same time because of
the volume of logs produced.
Perhaps I could run a script before the backups start to query the database for
a list of tablespaces, and build a set of Bacula job configs, one per
tablespace, then tell Bacula to reload its config. Each job has a pre-backup
script which dumps the tablespace (either to disk or to a FIFO), and the
fileset specifies that disk file or FIFO. If I'm dumping to disk, I'd need to
schedule them all with the same priority to allow the second job's dump to disk
to run in parallel with the first job's backup to tape, and so on down the
chain. If I use a FIFO, then they should be scheduled sequentially.
In fact, Would I need to dump the database to anything? Wouldn't I just have a
pre-backup job to put the tablespace into backup mode, bacula that tablespace's
file straight to tape, then a post-backup to put the tablespace back to normal?
Apart from the inherent ugliness of building backup configs dynamically on the
fly, and having a backup job and hence report for every single tablespace, it's
an intriguingly simple idea - thanks for that! I'll have a chat to the DBAs
about it.
DJ
-----Original Message-----
From: Kjetil Torgrim Homme [mailto:kjetilho AT linpro DOT no]
Sent: Thursday, 27 November 2008 14:17
To: bacula-users AT lists.sourceforge DOT net
Subject: Re: [Bacula-users] How to set up large database backup
I thought I'd get back to the original question :-)
"David Jurke" <David.Jurke AT tnzi DOT com> writes:
> The problem I have is with our large (expected to grow to several
> terabytes) [Oracle] server. I'm told by the DBAs that the size and
> amount of activity on this database is such that putting the whole
> database into hot backup mode for the several hours it takes to back
> it up is a Bad Idea, it generates far too many log(?) files. The
> method they recommend is to put a single tablespace into backup
> mode, back that up, put it back to normal, repeat for each
> tablespace. The backup takes the same time, but doesn't generate
> anything like the amount of log(?) files.
> [...]
> One way I can think to do it is if I can dive into Bacula and get
> hold of the "back up this file" program, and write a little script
> to do the database hot-backup change, back up that file (across the
> network via Bacula, straight to tape), change it back, rinse and
> repeat. It would need to be the Bacula component, not something like
> dd or tar (did I mention these are all Linux boxes), because it's
> going to be sharing a tape, in fact interleaved with, all the other
> backups. Pointers, anyone?
I think you need to use a FIFO. actually, many FIFO's, one for each
tablespace.
essentially, you set up each FIFO ready to go like this in a
pre-script:
mkfifo /var/backup/tablespace1
cat /var/oracle/tablespace1.data > /var/backup/tablespace1 &
cat will block, waiting for someone to open the FIFO and consume
bytes. so, when Bacula gets around to this file, cat will wake up and
transfer data.
you'll want to replace cat with a program which sets the correct locks
in Oracle, reads the data and dumps it into the FIFO, then releases
the locks.
the problem in your case is that you don't want to do the locking
until Bacula starts reading, and this calls for a little systems
programming. here's a rough outline: you'll need to dynamically
determine the list of tablespaces somehow, and you'll have to supply
the code to do the actual dumping :-)
#! /usr/bin/perl -w
use strict;
use POSIX;
my @tablespaces = ("ts01", "ts02", "exp01", "bal01", "bal02");
my $dumpdir = "/var/backup";
unless (-d $dumpdir) {
mkdir($dumpdir)
or die "$dumpdir: $!";
}
my %children;
for my $ts (@tablespaces) {
my $pid = fork();
if ($pid == 0) {
# the child
mkfifo("$dumpdir/$ts", 0600)
or die "$dumpdir/$ts: $!";
open(my $fifo, ">", "$dumpdir/$ts")
or die "$dumpdir/$ts: $!";
# open will block until there's a reader
print STDERR "Dumping $ts\n";
# ... do the work here. silly example just copies /etc/hosts
open(my $src, "/etc/hosts");
while (<$src>) {
print $fifo $_;
}
close($src);
close($fifo)
or warn "$ts: Closing FIFO failed: $!";
unlink("$dumpdir/$ts")
or warn "$dumpdir/$ts: $!";
exit(1);
} elsif ($pid > 0) {
$children{$pid} = $ts;
} else {
warn "$ts: fork failed: $!";
}
}
# Now we have started one process per tablespace in the background.
#
# You may want to skip this last bit, since the pre-script needs to
# exit before Bacula starts the backup. If you include it for better
# logging, you need to make sure the pre-script itself is started in
# the background.
my $pid;
do {
$pid = waitpid(-1, 0);
if ($pid > 0 && $? != 0) {
print STDERR "$children{$pid} (pid $pid) exited with code ", $?>>8,
"\n";
}
} while ($pid > 0);
--
hope this helps, | Redpill _
Kjetil T. Homme | Linpro (_)
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
