Bacula-users

Re: [Bacula-users] Newbie: Disk-To-Disk-To-Tape

2009-04-28 15:15:58
Subject: Re: [Bacula-users] Newbie: Disk-To-Disk-To-Tape
From: Arno Lehmann <al AT its-lehmann DOT de>
To: Bacula-users AT lists.sourceforge DOT net
Date: Tue, 28 Apr 2009 21:09:10 +0200
Hi,

28.04.2009 02:31, Steven Palm wrote:
> On Apr 27, 2009, at 2:00 PM, Arno Lehmann wrote:
>> I'd suggest to just back up that pool with a very plain setup.
>> Accurate backups will be good, probably.
> 
>   My only questions is, given it's extensive use of hardlinks to  
> minimize the pool size, if there anything special needed when backing  
> it up to tape to minimize storage space on tape much like is done on  
> disk.

No. All I know about hard links is that Bacula needs some space for 
the necessary bookkeeping, but storing those on disk is nothing very 
exciting. It's more or less just like a hard link on disk - just a 
pointer to a full version of the file in question.

> I guess I need to investigate this a bit more... I guess I was  
> thinking that surely someone had actually done it before and was  
> hoping for a "tested" answer. But maybe not, because if someone has  
> gone through the process to understand Bacula enough to use it, they  
> probably don't use BackupPC any longer... :)

Possible :-)

Though, actually, I have a customer who's collecting data from a 
number of distributed servers with rsync or similar and backs up that 
pool of stuff with Bacula.

>> Should work, though I haven't used virtual fulls yet. Especially
>> regarding pool and volume management, virtual fulls can be a bit of a
>> challenge - for simplicity, I'd suggest to use a more straightforward
>> setup at least to get started with Bacula. You can refine your
>> procedures later on anyway.
> 
>   I think it is going to be complex on the setup, but hopefully simple  
> for the usage...

That's the idea, yes.

>  If it works properly, then you'd pretty much have  
> just one "full" backup for each system, because after a successful  
> incremental or differential to update things, it would roll back into  
> a current "full".  I just think it would probably require some  
> scripted stuff after every backup to handle this (or nightly), and  
> probably shuffling things from pool to pool or volume to volume to  
> keep it straight.

Just my point - I need to get a bit more experience with this (and the 
developers are considering a change to the logic in Bacula, so I 
decided it wasn't worth extensive experimenting yet).

> That's why it's puzzling to think about as I am just  
> trying to learn the basics.  You are quite right, I need to get a  
> simple system in place first. :)
> 
>>>  The notes on page 10 of the Concepts document talk about the Copy
>>> job, and give a sample config, but I think part of it is missing...
>>> It mentions a "Storage = vtl" without any storage definition.  I
>>> suppose if I were familiar with bacula, I'd know what to put here,  
>>> but
>>> I do not.
>> Ignore it for now, it will become clear once you've got a real Bacula
>> setup.
> 
>   I asked because I thought that would be required to eventually get  
> where I want to go.

Well, that's just an arbitrary storage device definition. Nothing 
extraordinary involved.

>   The real problem here is the fact that until I get this working, we  
> will have no backup mechanism in place, so I'm not really keen on the  
> idea of fiddling around for a long time to learn.  I wish I had the  
> luxury of setting up a test environment for a few weeks or months, but  
> I can't afford that.  However, I think the things I am trying to use  
> are fairly new additions to Bacula, so maybe it's not likely anyone  
> has a tested implementation to share configs from.

At least that's the case for me. Others I know about are using 
migration, copy and virtual fulls for a while already, and I don't 
think there were more complaints than was to be expected whan using 
beta software.

>   It makes me nervous to trust only a disk backup here without being  
> able to dump to tape, especially since I can't take the disk-based  
> server offsite. :)

That's the spirit... unfortunately, among many users, disk backups are 
considered the best thing today, though the advantages of being able 
to remove a tape and store it wherever you like should be quite clear.

>  Also, I am accustomed to having a set of tapes  
> daily, or weekly at the worst case, to take offsite that contain full  
> backups.  I'm having a hard time picturing how differentials and  
> incrementals all fit together into the picture of how we've done  
> things and cover us with offsite daily/weekly dumps.

I think it's really more simple than you expect...

- Do backups to disk with whatever cycles you like. For simplicity, 
use only one pool.
- Create a virtual full in a disk based pool.
- Copy that job to tape, to another pool.
- Move the virtual full volume(s) to the original backup pool.
- Start from the beginning.

As far as I know, there is some scripting required. I'm pretty sure I 
could implement it in a few days (including extensive testing), but I 
don't have a proven solution yet.

...
>   It may take months before I understand that. :)

Ok, so forget this for now.

>  I'm sure it's  
> elegant, but again, I've always only rotated the backups so that daily  
> (M-Th), Weekly (F, unique for each week) and monthly tapes were all  
> full dumps so that any one could be taken offsite and have a full set  
> of data.  That is probably why I am fighting all the stuff documented,  
> because I don't want to deal with all the incrementals/differentials  
> when it comes to the daily tape storage... For disk storage, that  
> would be great, but when I take a tape set out of the drive I want it  
> to be complete.

See above - I think the solution isn't overly complex.

But, given your concerns with roaming users, I would probably suggest 
doing an rsync-based (or BackupPPC-based) backup to disk first, and 
just create full tape backups of that data as needed. A more detailed 
investigation, looking at data set sizes, available bandwidth, backup 
windows, disk capacity etc. would be needed before I set up something 
like this, though.

Arno

> 
>   Thanks, really, I appreciate it!
> 
>   Steve
> 
> 
> ------------------------------------------------------------------------------
> Register Now & Save for Velocity, the Web Performance & Operations 
> Conference from O'Reilly Media. Velocity features a full day of 
> expert-led, hands-on workshops and two days of sessions from industry 
> leaders in dedicated Performance & Operations tracks. Use code vel09scf 
> and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 

-- 
Arno Lehmann
IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück
www.its-lehmann.de

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations 
Conference from O'Reilly Media. Velocity features a full day of 
expert-led, hands-on workshops and two days of sessions from industry 
leaders in dedicated Performance & Operations tracks. Use code vel09scf 
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users