Re: [Bacula-users] File based backup - Large number of Volumes

On 3/7/2010 3:01 PM, Henrik Johansen wrote:
> Hi,
>
> On 03/ 7/10 02:58 PM, Dan Langille wrote:
>> I just set up a new SD for backing up to HDD.  The system can hande
>> about 7TB of backup data.
>>
>> I don't see a good reason for putting all Volumes in the same directory,
>> but I can see reasons for putting Volumes in different directories.
>> This would require the creation of multiple Devices within the
>> bacula-sd.conf file (one Device for each directory).
>
> Your new SD uses ZFS on FreeBSD, right ? If so - why bother with
> different directories ?

Well, sanity I suppose.  While I am a believer in letting Bacula worry 
about Volumes, I'm now about to have 3+years of backups in one 
directory.  I thought some structure might simplify things.

Do not take the following as proven.  It is what I've been thinking 
about and testing with today.  I am not sure this is an ideal solution 
at all.

But in order to achieve a hierarchy, it does complicate the 
configuration files.  I want something like this

$ cd /zfs/bacula/volumes/
$ ls
alpha bravo charlie echo delta frank golf hotel india

... where each directory listed above represents a Client.  All backups 
for that client goes in there.

Thinking about this: it requires a Device per client, as taken from 
bacula-sd.conf (vastly simplified to show the vital detail):

Device {
   Name = zfs-alpha
   Media Type = File
   Archive Device = /zfs/bacula/volumes/alpha
}

Repeat the above in bacula-sd.conf, once per each Client.

We need similar things in bacula-dir.conf (once again, simplified and 
omitting vital settings):

# Definiton of file storage device
Storage {
   Name       = MegaFile-alpha
   Address    = kraken.example.org
   Device     = zfs-alpha
}

Then, in the job[s] for Client = alpha, you need something like this:

Job {
   Name            = "alpha basic"
   Client          = alpha
   Storage         = MegaFile-alpha
}

But to do all this, you need a Pool per client, otherwise Bacula can't 
find them, and you'll see errors like this:

kraken-sd JobId 33438: Warning: Volume "FileAuto-0272" not on device 
"MegaFile-beta" (/zfs/bacula/volumes/beta).
kraken-sd JobId 33438: Marking Volume "FileAuto-0272" in Error in Catalog.

A job for Client alpha created Volume FileAuto-0272.  But it's at 
/zfs/bacula/volumes/beta, where MegaFile-beta cannot see it.

This leads me to thinking I need a Pool per client for this scenario to 
work properly.

>> How big is the disk array you're backing up to?  How do you have your
>> Volumes arranged?
>
> We have 3 SD's with 50 TB capacity per SD. All our volumes live in the
> same ZFS filesystem(s) on those boxes.
>
> We store multiple jobs per volume to allow for concurrent jobs with a
> limit on how large each volume can grow.

I allow any number of Jobs per Volume, with a maximum Volume size of 5GB.

-- 
Dan Langille - http://langille.org/

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users