Hi,

Tino Schwarze wrote on 2009-09-03 12:11:35 +0200 [Re: [BackupPC-users] 2.x 
behavior desired on 3.1 install]:
> On Wed, Sep 02, 2009 at 12:40:45PM -0400, Jeffrey J. Kosowsky wrote:
> 
> >  > IMO the easiest approach would be:
> >  > - if BackupPC_nightly starts, it acquires a lock, then waits for backups
> >  >   to complete
> >  > - no new backups start until BackupPC_nightly finished
> >  > 
> >  > This should be rather easy since it could be implemented at the central
> >  > scheduling code - actually BackupPC_nightly wouldn't be started, just
> >  > the flag would be set that it wants to.

you are probably right that it is *possible* to implement that. It is the way
BackupPC 2.x worked, after all, and the thread subject even points this out ;-).

The point is that there are different usage scenarios for BackupPC. If you are
backing up lots of hosts over slow links, you want to get all the parallelism
you can, because your local server disk is not the limiting factor. Being able
to run backups concurrently with BackupPC_nightly is a big step, especially if
BackupPC_nightly takes a long time because you have a large pool.

On the other hand, if your local server disk *is* your limiting factor, you
probably want to *avoid* parallelism, because it effectively slows things down.
Then again, you might not care about a slowdown, as long as your infrequently
connected machines are backed up whenever they are connected, rather than
waiting for other backups (or BackupPC_nightly) to finish first. So, different
people have different requirements, and you might even have different
requirements within a single BackupPC installation.

How about this: if there were two "priorities" (lacking a better word), which
you could assign to individual backups, BackupPC could take the above
requirements into account (better than just with MaxBackups):

1.) "exclusive"
    Run this job only if no other "exclusive" jobs are currently running.
    BackupPC_nightly would implicitly be "exclusive", as would BackupPC_link
    (it's a rather short operation, in my experience, and it does fast, random
    accesses to the local disk, so it's probably better off without
    concurrency; note that BackupPC_link cannot run concurrently with
    BackupPC_nightly or other BackupPC_link invocations anyway).

2.) "whenever"
    Run this job whenever possible, regardless of other "whenever" or
    "exclusive" jobs running, limited by MaxBackups. Manually initiated
    backups might be "whenever" (then again, you could also select the
    priority, possibly deferring the manual backup to a later point in time,
    as may currently happen if MaxUserBackups is exceeded).

The current scheduling behaviour could be obtained exactly, I believe, by
setting the priority to "whenever" for all jobs (i.e. in config.pl). You could
individually set any "local" backups to "exclusive" to avoid running them
concurrently with one another or BackupPC_nightly (or BackupPC_link). If you
set *all* backups to "exclusive", you've effectively eliminated concurrency,
which might be beneficial if all your backups are fast ("local") and running
them in parallel only slows all of them down ($Conf{MaxBackups} = 1 with the
additional effect of not running BackupPC_nightly (and link) concurrently, as
requested in this thread). You could still open up concurrency for individual
backups (done over a slow link or concerning an infrequently connected
computer) by setting them to "whenever".


Alternatively (or in conjunction with that), the MaxBackups restriction should
probably count BackupPC_nightly as a backup (though it isn't). In that way,
setting MaxBackups to 1 should effectively achieve what was requested (not
running BackupPC_nightly concurrently with backups).

> By the way: The same should be configurable for the trash cleaner. It
> does about the same to disk I/O as the nightly - it make the disk head
> spin around like mad since a lot of virtually random inodes need to be
> touched. Therefore, a config like
> 
> # Configure trash cleaning strategy:
> # 1 = continuous - wake up every $Conf{TrashCleanSleepSec} and check for
> #     files to delete (default pre 3.2 behaviour B-) )
> # 2 = run only just before BackupPC_nightly - see
> #     $Conf{DelayBackupsDuringBackupPCNightly} to disable backups during
> #     that operation
> $Conf{TrashCleanMethod} = 1;
> 
> Hm... does anybody volunteer to implement that? I could take a look and
> try to figure out a patch but I've not yet digged into the BackupPC
> scheduling code...

That should be simple. I'd suggest making BackupPC_trashClean terminate after
one pass over trash/ and calling it from BackupPC_nightly instead of from the
main daemon (or incorporating the code, it's basically just a function call
inside BackupPC::Lib, so there's not much code duplication). So, the daemon
only starts BackupPC_trashClean if TrashCleanMethod is 1, and BackupPC_nightly
starts it (or emulates it by calling $bpc->RmTreeTrashEmpty("$TopDir/trash")
once) if TrashCleanMethod is 2.

Note that TrashCleanMethod = 2 almost certainly means that you are running
only one nightly job ;-).

All of that said, concurrency (more than one nightly job, trashClean ...)
might actually speed things up, if your OS resp. I/O subsystem do a good job
of reordering requests. But if it's configurable, that would mean that even
people with IDE disks could run BackupPC ;-).

Regards,
Holger

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/