Bacula-users

Re: [Bacula-users] Getting back to the basics; Volumes, Pools, Reusability

2011-08-05 17:10:32
Subject: Re: [Bacula-users] Getting back to the basics; Volumes, Pools, Reusability
From: Dan Trainor <dan.trainor AT gmail DOT com>
To: Jeff Cleverley <jeff.cleverley AT avagotech DOT com>
Date: Fri, 5 Aug 2011 14:07:31 -0700
On Fri, Aug 5, 2011 at 11:17 AM, Jeff Cleverley
<jeff.cleverley AT avagotech DOT com> wrote:
> Dan,
>
> I'll throw in some general explanations of my own.  Perhaps a
> different concept of things would help.
>
> Consider a garage to be a storage pool.  Inside the garage are 100 1
> gallon buckets.  The buckets are your volumes.  You data is water.
> Outside the garage you have a 5 gallon bucket of water (your data).
> The job puts the 5 gallons of water into the 1 gallon buckets.  Your
> files are 1 teaspoon each.  Since all the data gets mixed in the
> buckets, you cannot empty (recycle) a bucket until every teaspoon in
> them is ready to be dumped out.  The backup software will put your
> water in any bucket that has room in that garage.  It generally will
> fill each bucket in turn and not partially fill multiple buckets.
>
> You would generally have a different garage for each retention period.
>  You can move empty buckets (volumes) between garages if you need more
> in one or the other.  You may also set up a new garage when bigger
> buckets (larger tape capacity) become available and they require
> different handling.
>
>>
>> One more thing that I'm not 100% on is how volume retention and
>> recycling works.  I believe I understand that of all three retentions
>> (File, Job, Volume), Volume always has the last word.  Is this
>> correct?  Is it normal (allowed?) to add additional volumes to my
>
> A volume (tape or disk) will keep the longest retention of anything on
> that volume.  If you back up one file and tell it to keep it for a
> year, then back up 1 million files to the same volume and say to keep
> them for 1 week, the volume will not recycle until the 1 year is up.
> From that stand point the file actually has the last word.
>
>> configuration, or is it more common to add another Pool?  I think I'd
>> be wanting to add volumes to each pool, as it's been pointed out that
>> I should reserve Pool creation to their specific role (daily, weekly,
>> monthly, archive).  I know that's a loaded question as well and it's
>> highly site-specific, but I wanted to discuss it for a moment if you
>> wouldn't mind.
>
> Most people set up different pools due to the retention period of
> data.  In the example above may people would have a Full pool they
> keep for a year, and an Incremental pool they may keep for 30 days.
> Otherwise your Incremental tapes would not recycle for up to a year if
> any Full backup data was on them..  Because tape really cannot spot
> overwrite data like disk can, you will use a lot more tapes if you let
> everything write to one pool of tapes.  Most people will add/delete
> volumes from the pools to adjust for usage.
>
> Once you have a functional pool/volume/backup configuration, you would
> only add or delete pools if you have new backups that have new needs
> that cannot be accommodated by an existing pool.  You might set up a
> pool to do one time archives of data to send off-site and keep them
> for 10 years.  You don't want other data on these tapes so a new pool
> and volumes would be best for these.
>>
>> Does only a Volume get recycled?  How about pools?  If pools as well,
>> why?  I know that volumes (assumed to be tape) are recycled because
>> they're always appended (if not always, then most-always since it's
>> good practice to prevent overwriting existing data).
>
> Pools do not recycle since they don't actually contain data.  They
> only contain volumes that have the data.
>>
>> What can I do to ensure that a volume that, say, contains a years'
>> worth of data doesn't get recycled too soon and potentially
>> overwritten?  What's my stop-gap to ensure that doesn't happen?  I'd
>> much rather be bothered with some administration tasks because Bacula
>> complained about not having any other place to store backups, rather
>> than having Bacula overwrite my long lost archives.  Are retention
>> periods (on volumes) hard rules that will guarantee that this does not
>> happen?
>
> As long as your catalog/database are good, the retention rules are
> hard.  If your catalog loses track of a tape it won't know anything
> about it and could overwrite it.  I'm just learning Bacula also but
> there are generally rules on when software is allowed to overwrite and
> format new media.  If you use physical tapes, you could remove things
> from the library or just flip the write-protect tab.
>>
>> Is there any direct correlation between a File and Volume retention
>> period, and if so, between the two, which takes precedence (can I
>> think of it that way?)?  Of the two, which should I be more concerned
>> or interested with/in?
>
> As mentioned above, the volume will not expire until all files on the
> volume are expired.
>
>>
>> I hope I'm not wasting anyone's time by asking these kinds of
>> questions.  Having them answered is certainly helping me out, and I
>> really hope it helps a lot of other people get over their case of cold
>> feet, as well.
>
> I don't see it as a waste of time.
>
> Jeff
>
> --
> Jeff Cleverley
> Unix Systems Administrator
> 4380 Ziegler Road
> Fort Collins, Colorado 80525
> 970-288-4611
>

Hi Jeff -

Yep, like Phil pointed out, that was one of the best analogies ever
haha.  Good job.

Alright, so I finally got a POC working, and I like what I see.  But
it has raised mroe questions which I think are still related to these
threads.

I ran a job, and it was blocked waiting for a volume to be mounted.
Why do I need to mount a volume?  I had already set 'AutomaticMount =
yes;' for this Device.  I suppose that only applies to Devices and not
volumes contained inside of a Device?  I mounted a volume manually but
of course I don't want to have to do this in the future.  Why does the
process of mounting a volume even come up?

For the sake of simplicty, I came up with three pools:
DailyIncremental, WeeklyDifferential, MonthlyArchive.  I also created
three Device resources with the same name.  Though it works, I know I
don't have the right idea behind how I named them.  What would be a
better recommendation for naming these Devices?  Devices named sda,
sdb, sdc, or mount points on the filesystem, something like that?  I
suppose I want to keep these Devices isolated between separate pools,
but maybe I don't, because then while the aggregate space by those
devices remains the same, the available storage space per Device may
be different depending on my Pool specifications, specifically with
regards to 'Maximum Volume Bytes' and 'Maximum Volumes'.

i think that's it for now.  With these questions answered I should
have a much better understanding of what I'm doing.

Thanks!
-dant

------------------------------------------------------------------------------
BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
The must-attend event for mobile developers. Connect with experts. 
Get tools for creating Super Apps. See the latest technologies.
Sessions, hands-on labs, demos & much more. Register early & save!
http://p.sf.net/sfu/rim-blackberry-1
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users