|
Re: [BackupPC-users] 38GB file backup is hanging backuppc (more info and more questions)
2009-02-12 07:36:41
Hi!
I had this trouble some time ago while backing up vmware virtual machines (the files were extremely large about 120GB) and rsync would behave just like you are saying it happens to you. I had other smaller vms about 10 GB and those worked perfectly with rsync.
I did some research and from what i've found the clues were going to the rsync protocol itself while transferring large files.
I changed the transference mode from rsync to tar and since then I had no trouble.
Cheers
Pedro
On Wednesday 11 February 2009 22:41:03 John Rouillard wrote:
> Hi all:
>
> Following up with more info with the hope that somebody has a clue as
> to what is killing this backup.
>
> On Fri, Feb 06, 2009 at 03:41:40PM +0000, John Rouillard wrote:
> > I am backing up a 38GB file daily (database dump). There were some
> > changes on the database server, so I started a new full dump. After
> > two days (40+ hours) it had still not completed. This is over a GB
> > network link. I restarted the full dump thinking it had gotten hung.
> > [...]
> > What is the RStmp file? That one grew pretty quickly to it's current
> > size given the start time of the backup (18:08 on feb 5). If I run
> > file (1) against that file it identifies it as
>
> >From another email, it looks like it is the uncompressed prior copy of
> the file that is currently being transferred.
>
> http://www.mail-archive.com/backuppc-users AT lists.sourceforge DOT net/msg05836.html
>
> Now because of another comment, I tried to disable the use of RStmp
> and force BackupPC to do the equivalent of the --whole-file where it
> just copies the file across without applying the rsync
> differential/incremental algorithm. I executed a full backup after
> moving the file in the prior backup aside. So I moved
>
> /backup-pc/pc/ldap01.bos1.renesys.com/326/f%2fdata%2fbak/ffedora-ds/fcurrent/fuserRoot/fid2entry.db4
>
> to
>
> /backup-pc/pc/ldap01.bos1.renesys.com/326/f%2fdata%2fbak/ffedora-ds/fcurrent/fuserRoot/fid2entry.db4.orig
>
> I claim this should make rsync diff against a non-existent file and
> thus just copy the entire file, but I still see in the lsof output for
> the BackupPC process:
>
> BackupPC_ 18683 backup 8u REG 9,2 36878598144 48203250
> /backup-pc/pc/ldap01.bos1.renesys.com/new/f%2fdata%2fbak/RStmp
>
> and there is only 1 file that is that large (36878598144 bytes) under
> that volume and it is the id2entry.db4 file.
>
> So what did I miss?
>
> Does BackupPC search more than the prior backup? I verified that run
> 327 (which is the partial copy) doesn't have any copy of:
>
> f%2fdata%2fbak/ffedora-ds/fcurrent/fuserRoot/fid2entry.db4
>
> in it's tree. So where is the RStmp file coming from?
>
> At this point it's been running about 24 hours and it has transferred
> only 10GB of the 30GB file.
>
> This is ridiculously slow. If my math is right, I should expect a
> 36GByte file at a 1Mbit/sec rate (which is about what cacti is showing
> as a steady state throughput on the ethernet port) to transfer in:
>
> '36*1024*8/(3600*24)'= ~3.413
>
> so it will take 3 and a half days at this rate. This is with both
> systems having a lot of idle time and < 1% wait state.
>
> If I run an rsync on the BackupPC server to copy the file from the
> same client, I get something closer to 10MBytes/second
> (80Mbits/sec). Which provides a full copy in a bit over an hour.
>
> Also one other thing that I noticed, BackupPC in it's
> $Conf{RsyncArgs} setting uses:
>
>
> '--block-size=2048',
>
> which is described in the rsync man page as:
>
> -B, --block-size=BLOCKSIZE
> This forces the block size used in the rsync algorithm to a
> fixed value. It is normally selected based on the size of
> each file being updated. See the technical report for
> details.
>
> is it possible to increase this or will the Perl rsync library break??
> It would be preferable to not specify it at all allowing the remote
> rsync to set the block size based on the file size.
>
> I see this in lib/BackupPC/Xfer/RsyncFileIO.pm:
>
> sub csumStart
> {
> my($fio, $f, $needMD4, $defBlkSize, $phase) = @_;
>
> $defBlkSize ||= $fio->{blockSize};
>
> which makes it looks like it can handle a different blocksize, but
> this could be totally unrelated.
>
> I could easily see a 2k block size slowing down the transfer of a
> large file if each block has to be summed.
>
> Anybody with any ideas?
>
--
----------------------------------------------------------------------------------------------------------
Pedro M. S. Oliveira
IT Consultant
Email: pmsoliveira AT gmail DOT com
URL: http://pedro.linux-geex.com
Cellular: +351 96 5867227
---------------------------------------------------------------------------------------------------------- |
------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
|
|
