Bacula-users

Re: [Bacula-users] File based backup - Large number of Volumes

2010-03-08 02:46:25
Subject: Re: [Bacula-users] File based backup - Large number of Volumes
From: Henrik Johansen <henrik AT scannet DOT dk>
To: <bacula-users AT lists.sourceforge DOT net>
Date: Mon, 8 Mar 2010 08:43:45 +0100
Hi,

On 03/ 8/10 12:39 AM, Dan Langille wrote:
> On 3/7/2010 3:01 PM, Henrik Johansen wrote:
>> Hi,
>>
>> On 03/ 7/10 02:58 PM, Dan Langille wrote:
>>> I just set up a new SD for backing up to HDD.  The system can hande
>>> about 7TB of backup data.
>>>
>>> I don't see a good reason for putting all Volumes in the same directory,
>>> but I can see reasons for putting Volumes in different directories.
>>> This would require the creation of multiple Devices within the
>>> bacula-sd.conf file (one Device for each directory).
>>
>> Your new SD uses ZFS on FreeBSD, right ? If so - why bother with
>> different directories ?
>
> Well, sanity I suppose.  While I am a believer in letting Bacula worry
> about Volumes, I'm now about to have 3+years of backups in one
> directory.  I thought some structure might simplify things.

Simplify what ?

On a side note, I would prefer ZFS filesystems instead of directories 
since they give you a lot more control.

> Do not take the following as proven.  It is what I've been thinking
> about and testing with today.  I am not sure this is an ideal solution
> at all.
>
> But in order to achieve a hierarchy, it does complicate the
> configuration files.  I want something like this
>
> $ cd /zfs/bacula/volumes/
> $ ls
> alpha bravo charlie echo delta frank golf hotel india
>
> ... where each directory listed above represents a Client.  All backups
> for that client goes in there.
>
> Thinking about this: it requires a Device per client, as taken from
> bacula-sd.conf (vastly simplified to show the vital detail):
>
> Device {
>     Name = zfs-alpha
>     Media Type = File
>     Archive Device = /zfs/bacula/volumes/alpha
> }
>
> Repeat the above in bacula-sd.conf, once per each Client.
>
> We need similar things in bacula-dir.conf (once again, simplified and
> omitting vital settings):
>
> # Definiton of file storage device
> Storage {
>     Name       = MegaFile-alpha
>     Address    = kraken.example.org
>     Device     = zfs-alpha
> }
>
> Then, in the job[s] for Client = alpha, you need something like this:
>
> Job {
>     Name            = "alpha basic"
>     Client          = alpha
>     Storage         = MegaFile-alpha
> }
>
> But to do all this, you need a Pool per client, otherwise Bacula can't
> find them, and you'll see errors like this:
>
> kraken-sd JobId 33438: Warning: Volume "FileAuto-0272" not on device
> "MegaFile-beta" (/zfs/bacula/volumes/beta).
> kraken-sd JobId 33438: Marking Volume "FileAuto-0272" in Error in Catalog.
>
> A job for Client alpha created Volume FileAuto-0272.  But it's at
> /zfs/bacula/volumes/beta, where MegaFile-beta cannot see it.
>
> This leads me to thinking I need a Pool per client for this scenario to
> work properly.

I still think that this is *way* to much configuration purely for 
"knowing" where a given volume is located on disk ...

>>> How big is the disk array you're backing up to?  How do you have your
>>> Volumes arranged?
>>
>> We have 3 SD's with 50 TB capacity per SD. All our volumes live in the
>> same ZFS filesystem(s) on those boxes.
>>
>> We store multiple jobs per volume to allow for concurrent jobs with a
>> limit on how large each volume can grow.
>
> I allow any number of Jobs per Volume, with a maximum Volume size of 5GB.
>


-- 
Med venlig hilsen / Best Regards

Henrik Johansen
henrik AT scannet DOT dk
Tlf. 75 53 35 00

ScanNet Group
A/S ScanNet

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users