BackupPC-users

Re: [BackupPC-users] Different Blackout Periods

2015-04-02 05:27:17
Subject: Re: [BackupPC-users] Different Blackout Periods
From: "Alexander Rehbein" <alexander.rehbein AT fmex DOT de>
To: "'General list for user discussion, questions and support'" <backuppc-users AT lists.sourceforge DOT net>
Date: Thu, 2 Apr 2015 11:25:23 +0200
Thank you, this is a really large answer. However my English is not very
well I hope I understand your answer right.

At the moment the solution is only doing incr backups and full via cron?

You told something about an delta. Are diff backups better for me? I want to
do the following:

I have an Mailserver which should do a full backup every week. It could
happen that I delete an email faulty, so I decided to backup the mails every
hour. This I want do with incr backups. If a mail get lost I can restore
this Mail from the backup archive.

I've looked in the archive. Sometimes the incr backup is only 20mb,
sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail
directory hourly instead of the full root path.

It also would be nice if some option like 'backupType' => 'full' would be
implemented. I could image something like a name you can setup for an
backup, so you can setup blackout periods different. For example:

BackupSet1 (full every week) with own blackout periods
BackupSet2 (incr full root path daily) with own blackout periods
BackupSet3 (incr /var/vmail path hourly) with own blackout periods

If no specials blackout periods defined, the default system blackout periods
would be used.

Greetings,

Alex

-----Ursprüngliche Nachricht-----
Von: Holger Parplies [mailto:wbppc AT parplies DOT de] 
Gesendet: Mittwoch, 1. April 2015 23:38
An: Les Mikesell
Cc: General list for user discussion, questions and support
Betreff: Re: [BackupPC-users] Different Blackout Periods

Hi,

Les Mikesell wrote on 2015-04-01 13:25:56 -0500 [Re: [BackupPC-users]
Different Blackout Periods]:
> On Wed, Apr 1, 2015 at 1:03 PM,  <alexander.rehbein AT fmex DOT de> wrote:
> >
> > I want to setup the following backup plan. Every week one full 
> > backup and every hour an incremental backup.
> >
> > This works but i have a problem with the balackout periods. Full 
> > backups should only done at night. is it possible to configure this?
> 
> There is no way to specify different blackouts for fulls vs.
> incrementals.

true.

> However fulls won't happen until the FullPeriod time has expired since 
> the last one, so if you force one at an appropriate time, subsequent 
> runs will happen at about the same time a week later.

That is not strictly true. If I remember correctly, a new full backup will
not happen until the FullPeriod has expired since *completion* of the last
one which might be significantly later, depending on the time the backup
takes and backup concurrency issues. Usually, you can compensate for that
with the schedule, blackouts, and an appropriately lower FullPeriod, to keep
your backups from wandering through the day.

Here, however, blackouts can't be used. What you suggest is very much
equivalent to "implement the effect of blackouts by just forgetting about
them". If blackouts are a requirement, i.e. full backups *must* only be done
at night, rather than "it would be rather convenient if they mostly run at
night until something unexpected happens, when we will be happy to manually
intervene to correct the timing", then this is not a solution.

I've got a small patch which I would have first expected would do what you
want, however the task is not as trivial as it seems. BackupPC schedules an
incremental backup or a full one as appropriate. When a full one is due, it
is rescheduled until it completes successfully, perhaps being skipped
several times due to blackouts. In the mean time, there is no point in
attempting incremental backups, because the same blackouts would apply. What
you really want to happen when a full run is skipped due to a blackout
period applying only to full backups is an incremental backup to be run
instead. However, a manually scheduled full backup should probably not be
automatically turned into an incremental. Overlapping blackout periods for
full and incremental backups should work as expected, i.e. an incremental
replacement backup should still be restricted by blackouts for incrementals,
regardless of the order in which blackouts are specified. So there are some
things to keep in mind.

An additional problem is that, programmatically, BackupPC currently handles
blackouts *before* making the decision whether a full or incremental backup
is needed. There may be a good reason for that (such as avoiding pings
during blackouts - pingCmd might attempt to wake up the remote host) or not,
I'd have to take a closer look. In any case, it would be significantly more
intrusive than changing two lines of code as my first attempt would have.

Configuration wise, I would want to add an optional hash member backupType
with supported values 'full', 'incr' or 'all', defaulting to 'all', which
would make the blackout period apply to backups of the selected type(s),
e.g.:

        $Conf{BlackoutPeriods} = [
                {
                        'weekDays' => [ 1 .. 7 ],
                        'hourBegin' => 4,
                        'hourEnd' => 2,
                        'backupType' => 'full',
                },
                {
                        'weekDays' => [ 1 .. 7 ],
                        'hourBegin' => 2,
                        'hourEnd' => 4,
                        'backupType' => 'incr',
                },
                {
                        'weekDays' => [ 3 ],
                        'hourBegin' => 1,
                        'hourEnd' => 3,
                },
        ];

(meaning no full backups from 04:00 to 02:00, no incremental backups from
02:00 to 04:00, no backups at all on wednesdays from 01:00 to 03:00, just to
give an example).

Ah, right, the web GUI would need to support setting this value, too :-).


So, while I can appreciate the need for this feature and don't see a way to
emulate it with current BackupPC capabilities, I can't offer you a solution
right now. I would suggest trying something like triggering full backups
with a cron job (and raising FullPeriod accordingly to avoid interfering
automatic
fulls) for now, though this is obviously only an approximation, too.


You do realize that with your configuration the hourly incrementals will be
growing deltas to the previous weekly full backup, right? That may not be a
problem, but with hourly backups you probably need these to be really fast.
Is retransmitting the same changes over and over again going to work for
you?

Hope this helps.

Regards,
Holger

----------------------------------------------------------------------------
--
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/