BackupPC-users

Re: [BackupPC-users] Problems with hardlink-based backups...

2009-08-20 08:52:48
Subject: Re: [BackupPC-users] Problems with hardlink-based backups...
From: Les Mikesell <lesmikesell AT gmail DOT com>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Thu, 20 Aug 2009 07:48:42 -0500
David wrote:
> Thanks for the replies so far :-) They were very informative.
> 
> About BackupPC itself, I'm still evaluating whether or not to actually
> use it, but I'm starting to decide against it. Here are my reasons:
> 
> 1) We're not backing up a lot of machines with a huge amount of
> duplicate data between machines. Just about every server and user's
> data is different. The common stuff between servers (/usr/, etc),
> isn't that large (the vast majority of storage is for unique data).
> For user machines, we backup their user folders, not the entire C:
> drive. Pooling common files from different machines isn't a priority.
 >
> 2) A big problem, is user dbx files, 2 GB etc .. don't want to store
> multiple copies of those. Actually a reverse diff-based approach works
> a lot better imo.
> 
> My current backup system lets me define which kind of "history
> storage" system to use for backups. rdiff-backup (for most places,
> where it works), and hardlinks, for servers which cause problems with
> rdiff-backup (although that led to my current problems with du &
> locate, which I'm currently researching). I might add more "history
> storage" systems, if I find something more appropriate later (eg
> HashBackup or gibak), or write my own. I lose that kind of flexibility
> if I change to most "fully integrated, prepackaged" backup systems
> (like BackupPC and most others), as opposed to command-line tools
> which you can script and mix & match to get a backup system that works
> best for your setup.
> 
> Which is also why I asked earlier about the ability to mix and match
> parts of BackupPC separately :-)
> 
> Yeah, there are downsides to home-brewed stuff, and I prefer to use
> premade stuff most of the time. But when none of the existing stuff
> matches my needs, I won't hesitate throw together scripts that do it
> better (for my needs), by scripting command-line tools, or writing new
> tools and then calling them. That's the unixy way :-). You can read
> more here:
> 
> http://en.wikipedia.org/wiki/Unix_philosophy#Mike_Gancarz:_The_UNIX_Philosophy
> 
> 3) The incremental/full system of backuppc is bothersome. I don't want
> to copy over full servers later, after initial rsync (or if I do,
> relatively infrequently, like once a month). I actually do want most
> of the backups to be incremental (ie, how rsync does it in hardlink
> snapshot-like schemes). But, a lot can change during incrementals, and
> dealing with those multiple incremental levels seems kind of annoying,
> although I'm sure there are good reasons for them.
> 
> 4) Possibly lots of redundant config in the text files
> 
> This is very minor in the grand scheme of things, but it's a pet peeve
> of mine, with backup systems I've seen in general. Every single server
> or user backup etc has the complete backup details in the config
> files, even if they are 99% similar.
> 
> This violates the "DRY" (Don't Repeat Yourself) programming principle
> (yeah, I treat backup configuration as a programming exercise :-) ).
> 
> Like if you have 40 servers, then it looks like you need to define all
> the details for all servers, rather than just defining the parts that
> changed per-server.
> 
> In my own config, adding a new server to the backup config is as
> simple as adding one line, like this to my server backups config file:
> 
>     bkp('192.168.0.2 complete backup') # Router
> 
> This basically adds the complete specification for a backup, to a list
> of backups to be run (after all the backup configs are loaded into
> memory, and filtered according to user command-line arguments and so
> on to the main backup script, which itself is run daily from cron).
> 
> Earlier in the config file for server backups, there is a Python class
> definition, where you define the details, kind of like a template. And
> those classes can also inherit from other Python classes, to customize
> a few details. Or take advantage of other Python programming
> constructs.
> 
> Also, passwords are stored separately, in a secure text file, using a
> ~/.pgass-like format, that supports wildcards for individual fields
> (for those of you familiar with PostgreSQL). eg:
> 
>     rsync:192.168.0.2::root:rrbackups:gib5Gryn
> 
> (gib5Gryn is a password I just generated with apg, for this example)
> 
> Although, these types of config files are more oriented at people who
> prefer to edit text files directly (eg, programmers like myself :-) ),
> and understand how classes, inheritance, and other programming-related
> things work, rather than going through a web frontend. And adding a
> templating-type system can introduce more complexity by itself.
> 
> Web frontends like BackupPC's are probably a lot more usable in
> general though, especially for non-programmers :-)

So far the only thing your evaluation is actually right about is that backuppc 
isn't great at handling large files that have small changes in each run - 
although if they are compressible it may still be a win compared to other 
approaches.  It would save everyone a lot of time if you just tried it instead 
of guessing about the way it works and assuming it is wrong.   (For example, 
the 
config files only have per-host differences and inherit all other values from 
the master, and when you add a new host in the web interface you can tell it to 
copy an existing host config so you don't even have to retype that part.)  And 
everything is just perl snippets that you can hand-edit if you prefer.

-- 
   Les Mikesell
    lesmikesell AT gmail DOT com


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>