Interesting thread!

Jeffrey writes:

> That being said, I agree that using a database to store both the
> hardlinks along with the metadata stored in the attrib files would be
> a more elegant, extensible, and platform-independent solution though
> presumably it would require a major re-write of BackupPC.
> 
> I certainly understand why BackupPC uses hardlinks since it allows for
> an easy way to do the pooling and in a sense as you suggest uses the
> filesystem as a rudimentary database.
> 
> On the other hand as I and others have mentioned before moving to a
> database would add the following advantages:

I agree on all the points.  If I was writing BackupPC today I
would seriously consider this approach.  As Les points out, the
most important open question (other than reliability) is whether
the performance is adequate as the store expands to millions of
files (and 10^8, 10^9 or more file blocks).  Of course, BackupPC
is relatively slow, so maybe the baseline expectation is already
sufficiently low.

I recently heard about lessfs, which runs on top of FUSE to provide
a file system that does block-level de-duplication.  See:

    http://www.lessfs.com
    https://sourceforge.net/project/showfiles.php?group_id=257120
    http://tokyocabinet.sourceforge.net/index.html

The actual storage is several very large (sparse?) files on any
file system(s) of your choice.  It should provide all the benefits
you expect: no issues of local limitations on hardlink counts,
meta-data etc, and the database files can be copied or rsynced.
I'm corresponding with the author to see if some additional useful
features could be added.

Yes, taking this approach would require a very substantial rewrite.
BackupPC would become a lot simpler.  But it also creates a significant
issue of backward compatibility.  The only solution would be to provide
tools that import the old BackupPC store into a new one.  That is
possible, but would likely be very slow.

Craig

------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
looking to deploy the next generation of Solaris that includes the latest 
innovations from Sun and the OpenSource community. Download a copy and 
enjoy capabilities such as Networking, Storage and Virtualization. 
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/