Craig Barratt wrote:
>
>> On the other hand as I and others have mentioned before moving to a
>> database would add the following advantages:
> 
> I agree on all the points.  If I was writing BackupPC today I
> would seriously consider this approach.  As Les points out, the
> most important open question (other than reliability) is whether
> the performance is adequate as the store expands to millions of
> files (and 10^8, 10^9 or more file blocks).  Of course, BackupPC
> is relatively slow, so maybe the baseline expectation is already
> sufficiently low.

And, you have to tackle the issue of ensuring that the filesystem and 
database are always kept in sync - or fixing things if they aren't.

> I recently heard about lessfs, which runs on top of FUSE to provide
> a file system that does block-level de-duplication.  See:
> 
>     http://www.lessfs.com
>     https://sourceforge.net/project/showfiles.php?group_id=257120
>     http://tokyocabinet.sourceforge.net/index.html
> 
> The actual storage is several very large (sparse?) files on any
> file system(s) of your choice.  It should provide all the benefits
> you expect: no issues of local limitations on hardlink counts,
> meta-data etc, and the database files can be copied or rsynced.
> I'm corresponding with the author to see if some additional useful
> features could be added.
> 
> Yes, taking this approach would require a very substantial rewrite.
> BackupPC would become a lot simpler.  But it also creates a significant
> issue of backward compatibility.  The only solution would be to provide
> tools that import the old BackupPC store into a new one.  That is
> possible, but would likely be very slow.

I see in the roadmap with the just-released version of opensolaris that 
block de-duplication is scheduled to be in ZFS in the next version, 
perhaps early next year.  How hard would it be to simply make the links 
for pooling an option that you could disable if the filesystem handles 
it better - as you can already do with compression? I'm not sure why you 
would need any other format change - and if you had a tool to 
reconstruct the pool links you could switch back if you wanted - at some 
cost in time and CPU.

If the ZFS snapshot send/receive also works well for replication, that 
would take care of almost all of backuppc's major issues (well, except 
for working nicely on Linux - but the price is right for opensolaris 
too...).

-- 
   Les Mikesell
    lesmikesell AT gmail DOT com


------------------------------------------------------------------------------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
looking to deploy the next generation of Solaris that includes the latest 
innovations from Sun and the OpenSource community. Download a copy and 
enjoy capabilities such as Networking, Storage and Virtualization. 
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/