Veritas-bu

Re: [Veritas-bu] Quarterly Backups and Calendar Schedule

2008-04-01 23:50:27
Subject: Re: [Veritas-bu] Quarterly Backups and Calendar Schedule
From: Dominik Pietrzykowski <dominik_pietrzykowski AT toll.com DOT au>
To: Curtis Preston <cpreston AT glasshouse DOT com>, bob944 AT attglobal DOT net, veritas-bu AT mailman.eng.auburn DOT edu
Date: Wed, 2 Apr 2008 14:33:31 +1100

> IMHO, "external, industrial strength, general purpose scheduler[s]" are
> designed for general purpose use and not designed for NBU -- and will
> never approach the level of utilization that NBU will get out of your
> resources.  I'd say save that money and spend it on data protection
> management software.

I agree with Curtis although I would spend the money on more drives. Actually, I'd buy a few SL8500s, fully populate them with T10Ks and use SUN E25Ks as media servers. That's a real man's library/server setup for a real man's backup system.


-----Original Message-----
From: Curtis Preston [mailto:cpreston AT glasshouse DOT com]
Sent: Wednesday, 2 April 2008 1:22 PM
To: bob944 AT attglobal DOT net; veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Quarterly Backups and Calendar Schedule

> > [...] Schedule-based backups (SBB) would give Randy more than
> > [...] uses Frequency Based Backups (FBB), the first thing is he
>
> Is there some reason to introduce a new, content-free term and acronym
> here?

Wow, I was brain-farting while typing late at night.  I meant CBB, not
SBB.  I just got tired of typing them over and over.

> And while I'm writing, calendar-based scheduling is, of course, the
work
> of the devil.  Real Men will use frequency-based.  Real Men with
> purchasing authority will buy an external, industrial-strength,
> general-purpose scheduler for their enterprise.

At the risk of starting another long argument with "bob944," I have to
say that I completely disagree with you on this one.  In fact, I feel
stronger about this than calendar-based backups.

IMHO, "external, industrial strength, general purpose scheduler[s]" are
designed for general purpose use and not designed for NBU -- and will
never approach the level of utilization that NBU will get out of your
resources.  I'd say save that money and spend it on data protection
management software.

Unless I'm missing something, a commercial scheduler gives you three
things:
1. Control over what runs when
2. Common reporting across NBU & non-NBU activities
3. Easier scheduling of pre- and post-backup tasks, especially
  on multiple hosts

Control

People tell me they want to be able to specify exactly when a given
backup runs.  With few exceptions, though, why do you care?  As long as
they run successfully sometime during your backup window, who cares when
they run? 

Reporting

This is the biggest argument for using a commercial scheduler, but I'd
rather have reporting that is tailored for backups, not just reporting
that can tell me a given job ran or not.  I'd also argue that having
your backups run better is more important than having the report of them
run next to the report that all your batch processing ran last night.
(And I will be making the argument that backups performed by NBU's
scheduler should run better.)

Pre-post

I'm not talking about simple pre/post processing.  I'm not talking about
backups that require multiple things to be done on multiple hosts before
you can run a backup.  This is my one exception to the rule.  For such
backups,  I'd use a commercial scheduling product.

Now let's look at NBU.

Control

For the few backups that need to run at specified times each night, fire
them off at that time in NBU.  No big deal.  As I've already stated, I
do like to specify when fulls and cumulatives run using calendar-based
backups, as I can more easily load balance my environment, making sure
my tape drives or disk drives and backup servers are fully utilized all
night long, rather than the willy-nilly way they show up when you do
frequency-based backups.

Reporting

The use of a commercial scheduler actually messes up the activity
monitor.  If NBU retries a backup several times and succeeds, it will
show up as a success in the job monitor.  If your commercial scheduler
does this for you, it will show up multiple failures followed by a
success -- yuck.

Although I think NBU's activity monitor isn't too bad, I do believe you
should get a data protection management product.  It offers way too much
value to your backup environment to not buy one.  I make this point here
only because I'd rather you see you spend your limited budget on this
and not on a scheduler.

Pre/post backups

All but the multi-host config backups I mentioned earlier, I would
schedule with NBU and I'd use bpstart_notify/bpend_notify.  They're not
perfect (by a long shot), but they get the job done and they allow me to
benefit from the other good things from the NBU scheduler.

Efficiency

The real reason to use NBU's scheduler is how well it efficiently
utilizes your backup resources.  Every resource has a finite number of
backups that can be sent to it.  Whether it's multiplexed jobs to a tape
drive or multistreamed jobs to a disk device, NBU will ensure that each
device has as many backups as it can send it and no more.  Setting aside
backups that should run at a particularly time each night (which I
generally find to be a minority of jobs), I give EVERY backup the SAME
WINDOW (e.g. 8 PM - 8 AM), and then assign priorities to each of them.
When the window opens, NBU makes as many jobs active as it needs to in
order to meet the multiplexing/multistreaming requirements you've
specified, and it puts the rest in a queued state.  The beautiful thing
about this is there could be 10,000 jobs in the queued state, and none
of them take any resources until they go active.  The second that one of
the active jobs completes, another queued job goes active (based on the
priorities you specified) and that tape/disk resource continues to be
fed the number of streams that you specified, until NBU has run out of
jobs.  (And, BTW, when we're talking about tape devices, making sure
that they constantly have the number of jobs that they need to stream
will go a long way towards increasing your backup success rate.)

Can a commercial scheduler do this?  I don't have experience with all of
them, but the ones that I have seen the answer appears to be no.  You
can specify start times, and do this one after that one, but I'm not
sure if you can say "do these 10,000 jobs, but only do 10 of them at a
time."  If a scheduler can do this, I might reconsider my position.

What I've seen in practice is that when people use a commercial
scheduler, they go to one of two extremes.  They either have lots of
gaps in coverage (where devices are not being sent the number of jobs
you specified), or they're trying to approach the way NBU queues up jobs
by firing off a bunch at a time.  The problem with the former is that
it's wasting a lot of resources.  The problem with the latter is that
manually run queued jobs DO take up resources, as a manually run backup
has processes associated with it even if it's in a queued state, where
an automatically run backup does not.

Summary

So I believe that NBU's scheduler meets the needs of 99% of most users
that I've found, and I've even managed to figure out how to use it to
meet the multi-host requirement if a customer doesn't have a commercial
scheduler.  I would therefore argue that a commercial scheduler is not
needed and I'd take the money I saved on that and buy a data protection
management product.

----
W. Curtis Preston

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu