Hello,
27.02.2010 19:08, Stan Meier wrote:
> * Arno Lehmann <al AT its-lehmann DOT de>:
>> 27.02.2010 14:46, Stan Meier wrote:
>>> 1. Keeping configuration sane: With more than 120 servers, we need to
>>> find a way to keep the configuration files readable. Our servers all
>>> follow some naming scheme, for example, we got "appserver01" through
>>> "appserver08" or "webcache01" through "webcache04". We think we should
>>> split client configurations for each server group, so the file
>>> "clientdefs/appserver.conf" would define all appserver0X clients.
>>> Furthermore, most of those servers will need a default job performed
>>> (/etc, /root, /opt and so on). While it's easy to reuse a "JobDefs"
>>> stanza to actually define all those jobs, isn't there any way to
>>> "group" those servers? Do we really have to define more than 120 jobs,
>>> one for each server?
>> The way to go, in my opinion, is to create the actual configuration
>> dynamically - you can include script output into the configuration
>> *anywhere*... now use a script that creates the actual configuration
>> dynamically, starting with a template where you insert the client name.
>
> While you are right and creating a configuration based on scripts is
> quite easy (and has added benefits, for example that you dan define
> one file pool per server group), we still have to deal with 120 backup
> jobs.
>
> But, since you didn't jump on that part of my question, I presume
> there is no solution to that?
In my opinion, there's no solution because there's no problem :-)
120 jobs - plus, in the worst case, 120 copy jobs - are no problem to
Bacula. You'll need to tweak concurrency, priorities, and all the
(new) directives managing concurrency of several instances of a single
job a bit, but basically just scheduling all you jobs at the same
time, with the same priority, and letting them just should work well.
For the copy jobs, I'd recommend (as others already did) to use a
different priority. I'd also recommend to use a sqlquery as selection
scheme.
>>> 2. Backup availability: One plan would be to use a large part of the
>>> 24TB available as a FilePool (or several). Each job would then write
>>> it's data to that pool. A Copy job could copy the data to tape later
>>> on - with the advantage that restores of recent data would be quite
>>> fast since they would still be sitting on disk. Before running the
>>> backup the next day, we would simply recycle those file volumes. Is
>>> that a reasonable strategy?
>> Yes. Properly set up, that's a very reasonable approach. You'll need
>> to understand retention times and how to select jobs for migration in
>> detail.
>
> I see several things here which we will have to look at. Please
> correct me if I'm wrong or if I forgot anything:
>
> 1. Concurrency: We will need to investigate all the different places
> in Bacula where job concurrency, concurrent pool/storage usage and
> connection limits are defined and adjust them to "fit together" as
> well as optimize them to the I/O operations limit of our raid storage.
Yes. This is probably easier than it looks right now, because you will
find that having one job per client, and as many jobs per storage
daemon as possible, will serve you best. So you only have to find out
what your SD can manage reasonably.
> 2. Scheduling: Ideally, the copy job would start as soon as all the
> backup jobs have finished. But since the Schedule resource does not
> allow references to Job names, we are pretty much screwed in that
> department and will probably have to resort to a fixed schedule.
Schedule, for example, an hour after the backup job and use a
different priority.
> 3. Job selection: You already pointed that one out. Would it be as
> easy as just selecting all uncopied jobs from a given pool?
As mentioned above, I prefer to use a hand-tailored sql query.
For example, in my office, I'm doing monthly copies of the latest full
backups to tape. I select all successfully finished full backups that
were started less than four weeks ago. The regular full backups are
scheduled to happen during the first week of each month, and the copy
job is scheduled to run the second week of the month.
Works like a charm, but I would refine things a bit if I had to manage
more than my 10-20 jobs.
> Since all
> volumes in the pool are recycle before backup starts (or while it is
> running), naturally, the only uncopied jobs would be those that were
> written to disk recently.
Sounds reasonable. Depending on the available disk space for you, you
might find that keeping as many jobs on disk as possible is more
convenient, and then you couldn't rely on the above assumption anymore.
> So, for now, our backup plan would be something like:
>
> 1. Start actual backup
> 1.1 Start backup jobs in parallel
> 1.2 (Possibly erase all used volumes in a given pool)
> 1.3 Get data from clients, write to File pools
> 1.4 wait until all backup operations have finished
> 2. Copy all recent jobs to tape
> 3. Backup catalog
>
> I don't see how we can synchronize more than 120 backup jobs yet, to
> be honest. We could run backup at 9pm and the copy job at 10am to
> allow for a large margin of error, but that just doesn't feel like a
> proper solution.
Priorities... or, and that's probably even simpler, just start copy
jobs during all the daytime (reserving nights for backups). If there
are no jobs to copy, nothing happens. As soon as uncopied jobs show
up, they will be copied to tape.
However, if you want to start a copy as soon as possible after a job
is finished, I'd recommend to do this with a run script in each job.
Arno
> Stan
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
Arno Lehmann
IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück
www.its-lehmann.de
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
