Les Mikesell wrote at about 09:50:25 -0500 on Tuesday, June 2, 2009:
 > Jeffrey J. Kosowsky wrote:
 > > 
 > > In my mind the only major reason not to move to a database
 > > architecture is that it would require a substantial re-write of
 > > BackupPC as pointed out in my earlier note.
 > 
 > Do you actually have any experience with large scale databases?  I think 
 > most installations that come anywhere near the size and activity of a 
 > typical backuppc setup would require a highly experienced DBA to 
 > configure and would have to be spread across many disks to have adequate 
 > performance.

I am by no means a database expert, but I think you are way
overstating the complexity issues.  While the initial design would
certainly need someone with experience, I don't know why each
implementation would require a "highly experienced DBA" or why it
"would have to be spread across many disks" any more than a standard
BackupPC implementation. Modern databases are written to hide a lot of
the complexity of optimization. Plus the database is large only in the
sense of having lots of table entries but is otherwise not
particularly complex nor do you have to deal with multiple
simultaneous access queries which is usually the major bottleneck
requiring optimization and performance tuning. Similarly the queries
will in general be very simple and easily keyed relative to other
real-world databases. Remember size != difficulty or complexity.

 > When you get down to the real issues, normal operation has 
 > a bottleneck with disk head motion which a database isn't going to do 
 > any better without someone knowing how to tune it across multiple disks. 

This seems like a red herring. The disk head motion issue applies
whether the data is stored in a database or in a combination of a
filesystem + attrib files. If anything, storage in a single database
would be more efficient than having to find and individually load (and
unpack) multiple attrib files since the database storage can be
optimized to some degree automagically while even attrib files that
are logically "sequential" could be scattered all over the disk
leading to inefficient head movement. Also, the database could be
stored on one disk and the pool on another but this would be difficult
if not impossible to do on BackupPC where the pool, the links, and the
attrib files are all on the same filesystem.

 >     Also, while some database do offer remote replication, it isn't 
 > magic either and keeping it working isn't a common skill.
 > 

Again a red herring. Jut having the ability to temporarily "throttle"
BackupPC leaving the database in a consistent state would allow one to
just simply copy (e.g., rsync) the database and the pool to a backup
device. This copy would be much faster than today's BackupPC because
you wouldn't have the hard link issue. Remote replication would be
even better but not necessary to solve the common issue of copying the
pool raised by so many people on this list.

 > -- 
 >    Les Mikesell
 >     lesmikesell AT gmail DOT com
 > 
 > 
 > 
 > ------------------------------------------------------------------------------
 > OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
 > looking to deploy the next generation of Solaris that includes the latest 
 > innovations from Sun and the OpenSource community. Download a copy and 
 > enjoy capabilities such as Networking, Storage and Virtualization. 
 > Go to: http://p.sf.net/sfu/opensolaris-get
 > _______________________________________________
 > BackupPC-users mailing list
 > BackupPC-users AT lists.sourceforge DOT net
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/
 > 

------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
looking to deploy the next generation of Solaris that includes the latest 
innovations from Sun and the OpenSource community. Download a copy and 
enjoy capabilities such as Networking, Storage and Virtualization. 
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/