Hello,
This (what I call deduplication) is a *really* big project as it requires not
only some rsync type technology that "seems" to be present in librsync, but
this is only part of the problem. It also requires some significant design
decisions and serious implementation in Bacula.
Friday or over the weekend, I will try to summarize some of the issues and
point out some of my current thoughts on this, so that if you are interested
I can possibly help guide you in a direction that will be most compatible
with current Bacula architecuture -- we can enhance what we have, but we
cannot totally re-write it in one crack, which imposes certain constraints.
Best regards,
Kern
On Wednesday 01 October 2008 19:57:32 Eli Shemer wrote:
> Alright. Librsync it is.
> I will try to make a separate program that will try to make it work with
> librsync. See how it goes.
>
> I don’t suppose it should be that problematic to compare the two different
> data buffers. If I have the old record in the volume, i can trigger a
> read_data session on the storage from within the fd and transfer the data
> to the fd. In the save_file() callback I can both transfer the old data
> from the storage to the buffer, and read the file acquired from find_one().
>
> The main problem may be actually implementing some kind of a merging policy
> prior to comparison since the old data must be up to the data with all
> relevant previous backups.
>
> One thing that puzzles me is how I can transfer a large chunk of data to
> the file director machine for the comparison procedure without losing the
> actual reason for this feature. It will be vastly time consuming and
> expensive but I suppose in the rsync code I will find the answer to that
> question.
>
> -----Original Message-----
> From: Kern Sibbald [mailto:kern AT sibbald DOT com]
> Sent: Wednesday, October 01, 2008 6:25 PM
> To: bacula-devel AT lists.sourceforge DOT net
> Cc: Eli Shemer; Bacula-users AT lists.sourceforge DOT net
> Subject: Re: [Bacula-devel] rsync link behavior
>
> On Wednesday 01 October 2008 17:50:24 Eli Shemer wrote:
> > Hey there,
> >
> >
> >
> > After doing several incremental backups of our remote servers, I've
> > noticed the backup sizes and transfer time are just too long.
> >
> > Making bacula almost completely improper for backing up remote servers
> > with big single files.
>
> Yes, you need high speed connections.
>
> > I wanted to know whether someone actually got to implement an incremental
> > backup to act like rsync does upon file changes ?
>
> It is a project we are currently discussing as possible in a release
> following 3.0.0 scheduled around the end of the year -- i.e. we hope to
> begin the project sometime in 2009. If there is commercial funding of this
> project (already one company interested) we could probably speed it up.
>
> > I'm looking to write a patch to the save_file callback in the bacula-fd's
> > backup procedure to utilize the gnu's diff and patches and to send those
> > over the line to the bacula-sd. Of course a corresponding implementation
> > in the storage device will also be required.
>
> Unless I am missing I am not sure such a patch would be very appropriate.
> For it to work, you would need both the original file and the new modified
> file, and I am not sure how you will accomplish that. In addition, diff and
> patches only work for source files, which is rather limiting.
>
> If you do figure out some clever way to have both files, then it would be
> *much* better to add some code that uses the librsync library calls to
> effectively implement what rsync does.
>
> Regards,
>
> Kern
>
> > Comments anyone ?
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge Build the coolest Linux based applications with Moblin SDK & win
> great prizes Grand prize is a trip for two to an Open Source event anywhere
> in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Bacula-devel mailing list
> Bacula-devel AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-devel
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
