Because this is a dump, a small change in a record near the beginning of the sort would cause the whole file to be a different structure. If you can influence the sort process for the dump and do it by date that the record was updated then you wouldnt likely have this issue.
example (csv formated)
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
3,agrtdhh,data,4/16/2009
if you dump this once you get a file with:
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
3,agrtdhh,data,4/16/2009
if you add two more data rows
4,aaewrr,junk,6/7/2009
5,atiojojj,last,6/8/2009
your export might be
4,aaewrr,junk,6/7/2009
1,aerthdfd,test,1/1/2006
2,asdfsad,stuff,3/6/2009
5,atiojojj,last,6/8/2009
3,agrtdhh,data,4/16/2009
which rsync would match up because the data has changed location within the file. if you can sort by the row id, or the date, rsync will be much happier
On Tue, Aug 25, 2009 at 6:37 AM, Les Mikesell <lesmikesell AT gmail DOT com> wrote:
Nigel Kendrick wrote:
> Hi,
>
> This may also be applicable to MySQL etc..
>
> I have an MS-SQL database that dumps out a backup that's around 700MB in
> size. BackupPC brought this over to its server via ADSL/VPN/Rsyncd
> (422mins!) and when I did another data dump about 4 hours later, around
> 16Mb of changes to the file were transferred very quickly. However,
> every single backup since then has gone back to 422 minutes. I could
> imagine that the database and dump will change over time, but it seems
> strange that the entire file needs copying every time - as if its
> structure is always 100% different from the previous.
>
> I wondered if I was missing anything obvious (being a BackupPC noob)
> like file time stamping causing a complete file transfer every day, or
> are database dumps likely to be completely different every time?
>
> I will keep some copies of the dumps and do some comparisons but in the
> meantime any tips, thoughts etc.?
incremental runs for backuppc? Assuming this is rsync or rsyncd, note that
incrementals always compare against the last full unless you have configured
multiple incremental levels, so they tend to transfer more each time. Or there
may just be enough difference that it can't find the matching parts.
If you are running rsync with ssh (probably not if this is windows), you might
save some time by adding the -C option for ssh compression. If transfer time is
important, it might be worth setting up a scheduled rsync command to copy to
something on the network local to the backuppc server so you could use the -z
option (which backuppc doesn't support), then have backuppc pick up that copy to
keep the history. Or, if rsync isn't finding matching parts anyway perhaps you
could compress the file before the transfer.
--
Les Mikesell
lesmikesell AT gmail DOT com
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
