BackupPC-users

Re: [BackupPC-users] trying to improve the speed at which backuppc rsync back up processes a large binary file in incremental backups

2012-01-10 19:39:49
Subject: Re: [BackupPC-users] trying to improve the speed at which backuppc rsync back up processes a large binary file in incremental backups
From: John Habermann <jhabermann AT cook.qld.gov DOT au>
To: backuppc-users AT lists.sourceforge DOT net
Date: Wed, 11 Jan 2012 10:39:50 +1000
Thank you for your reply Les

On Sun, 8 Jan 2012 16:00:02 -0600
Les Mikesell <lesmikesell AT gmail DOT com> wrote:

> On Sun, Jan 8, 2012 at 4:48 AM, John Habermann
> <jhabermann AT cook.qld.gov DOT au> wrote:
> >
> > You can see that the backup of the /opt share takes nearly the total
> > time of the incremental taking about 8 and half hours to complete
> > while the backup of the /opt rsync share in the full backup takes
> > about 3 and half hours. The full backup is slightly longer than
> > what it takes if I just do a rsync over ssh copy of the file from
> > the client server to the backup server.
> >
> > I have found that rsync seems to always transfer the whole file when
> > copying this file from the client server to the backup server:
> >
> > # rsync -avzh --progress -e ssh
> > administrator@isabella:ExchangeDailyBackup.bkf
> > ExchangeDailyBackup.bkf Password:
> > receiving incremental file list
> > ExchangeDailyBackup.bkf
> >      54.44G 100%   10.66MB/s    1:21:10 (xfer#1, to-check=0/1)4
> >
> > sent 3.31M bytes  received 3.27G bytes  486.33K bytes/sec
> > total size is 54.44G  speedup is 16.65.
> 
> Note that you have used the -z option with native rsync, which
> backuppc doesn't support.  You can add the -C option to ssh to get
> compression at that layer when you run rsync over ssh, though.

I will try that. I imagine it won't make much of a difference with the
ntbackup file if that is already compressed but it will be interesting
to see how it affects the transfer of other data. 

> 
> > My questions for the list are:
> > 1. Is it reasonable for rsync to transfer the whole file when
> > copying a large ntbackup file?
> 
> Yes, those files may have little or nothing in common with the
> previous copy.   If compression or encryption are used they will
> ensure that no blocks match and even if they aren't, the common blocks
> may be skewed enough that rsync can't match them up.

Ok I wonder if it might be better for me to look at having that file
backed up then by tar rather than rsync. I can't see any mentions of
using multiple transfer methods within a single client config in my
searching. Is that an option or is the only way to do this is to create
a second client.pc file that uses tar as the transfer method and just
backups up the ntbackup share?

> 
> > 2. Why does an incremental backup of this file take so much longer
> > than a full backup of it or a plain rsync of this file?
> 
> That doesn't make sense to me either.  Are you sure that is consistent
> and not related to something else that might have been using the link
> concurrently?
> 

It appears pretty consistent that the incremental backups of this
client are always longer than the full backups but there is a fair
amount of variability in the time the incremental backup takes. I am
going to set up a separate backuppc server on the same local area
network as the client and see how the full compares to the incremental
when the bandwidth of the connection is not the limiting factor.


-- 
John Habermann
IT Officer
Cook Shire Council
10 Furneaux St
Cooktown 4895
ph 40820577

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/