Les Mikesell wrote at about 15:23:41 -0500 on Monday, August 31, 2009:
 > Jeffrey J. Kosowsky wrote:
 > >  > 
 > >  > > I see lots of advantage in keeping the database portion relatively
 > >  > > small, fast, replicable, and moveable. Then you can keep and
 > >  > > distribute the files themselves wherever you want them spread across
 > >  > > one or more separate filesystems. Then the database portion is
 > >  > > optimized for what a database does best and the file-storing
 > >  > > portion can be optimized for what a filesystem stores best. And both
 > >  > > parts are easily moveable, replicable and not dependent or limited by
 > >  > > hardlinks or other filesystem-dependent functionality.
 > >  > 
 > >  > But the parts aren't independent.  How do you propose keeping them in 
 > >  > sync or fixing them when they inevitably differ?
 > > 
 > > Perhaps analogously to the way BackupPC_Nightly now makes sure that
 > > pool is in synch.
 > 
 > It just uses filesystem operations so it is as reliable as the 
 > filesystem itself.
 > 
 > > More generally, we would need to consider two things:
 > > 1. What are the normal ways in which the two could get out of synch
 > >    and then address each of those cases
 > 
 > Start with that copy you wanted to be able to make.  If the files and 
 > metadata are separate things, there will clearly be times they are out 
 > of sync and lots of opportunity for them to stay that way.

First copying is not even my primary reason for proposing a database
approach. Second, copying a filesystem also creates synchronization
issues unless you can do some type of shadow copy which either
requires you to unmount the filesystem or use something like
ZFS. Third, how hard is it to 'halt' BackupPC for the infrequent times
that you want to backup or move the Backup tree?

 > 
 > > 2. If necessary, create a repair tool (similar to what I created for
 > >    the current system) if something breaks in non-standard ways (e.g.,
 > >    due to crashes, disk failures, etc.)
 > > 
 > > I guess I can't answer your question without knowing what use cases
 > > you are worried about.
 > 
 > All of them.  Filesystems have journal/check mechanisms.  How would you 
 > provide the equivalent?

I am certainly not a database expert but I am aware enough to know
that such integrity issues are dealt with all the time in real world
databases where you have even more complex use cases of multiple
simultaneous users across multiple servers, disks, data sets,
applications, etc.

Obviously, moving to a database version would require significant
thought and work. Even the "simple" existing BackupPC implementation
went through multiple iterations involving and eliminating various
race and locking conditions. No one is saying this would be easy or
that it is a short-term feature addition -- it may not even be
worthwhile and maybe the existing limitations could be addressed
otherwise.  I am certainly not trying to argue that the current
approach doesn't suit your needs. I am only suggesting that the
current implementation seems to be difficult to extend to address
other relatively common and standard backup requirements. We are all
just trying to suggest potential alternative long-term development
paths.

My one and only point is that if the advantages of a database approach
are compelling then I am confident that the nuts-and-bolts details are
solvable. We are not exactly stretching the cutting edge of databases
here and we shouldn't be afraid to discuss a potentially better
solution because of generic bogeymen like "atomicity." Let's all keep
an open mind, especially since we are all far from committing
development resources to any path.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/