On Tuesday 04 May 2010 10:45:16 Craig Ringer wrote:
> On 4/05/2010 11:45 AM, Morty Abzug wrote:
> > file dedup (rather than block dedup) could mostly be handled at the
> > catalog level with another level of indirection.  I.e. instead of a
> > catalog entry containing file metadata and where the file lives on
> > media, it would contain file metadata and a foreign key to a table
> > that maps hashes to where the file lives on the media.
>
> This is really hard to do when everything is bundled into big volumes,
> and volumes are expired as units. It'd be necessary to compltely
> re-design the way storage works to do file-level deduplication, and it's
> not as easy to do (efficiently, with proper retention etc) as it first
> seems either.
>

Actually, this is an issue that I have thought a lot about, but not from the 
stand point of having the underlying filesystem do the deduping, rather from 
the stand point of having some form of deduplication in Bacula.  

One of the problems with doing sliding block or byte level deduplication is 
that you have to efficiently store the hashes and the parts of the files that 
have been deduped.  BackupPC handles deduping files by using hard links, but 
lots of hard links are a disaster for fsck and backup programs.  Thus some 
other method is needed.

I have been wondering if git technology cannot be put to use in defining a 
new "Bacula Volume" format for disks.  If anyone is familiar with or wants to 
look at how git handles storing data, I would be *very* interested in your 
views.

Best regards,

Kern

------------------------------------------------------------------------------

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users