Re: [BackupPC-users] backuppc slow rsync speeds

Les Mikesell <lesmikesell AT gmail DOT com> wrote on 09/17/2012 11:51:09 AM: > On Mon, Sep 17, 2012 at 10:16 AM, Mark Coetser <mark AT tux-edo.co DOT za> wrote: > > > > > > > Its the first full run but its taking forever to complete, it was running > > for nearly 3 days! > > As long is it makes it through, don't make any judgements until after > the 3nd full, and be sure you have set up checksum caching before > doing the 2nd. Incrementals should be reasonably fast if you don't > have too much file churn but you still need to run fulls to rebase the > comparison tree.
I'm writing a longer reply, but here's a quick in-thread reply:

I know exactly what you mean by waiting until after the first full. Often the second full will be faster -- but only *IF* you are bandwidth limited will you will see an improvement. In this case, neither him nor I are bandwidth limited. I don't see an improvement.

I am routinely limited to no more than 30MB to 60MB per *minute* as the maximum performance for my rsync-based backups. This is *really* pretty terrible. I also see that the system is at 100% CPU usage when doing a backup. So, my guess is that the Perl-based rsync used by BackupPC is to blame.

The other annoying part of this is that top shows 50% idle CPU. That's because I have two cores. One of them is sitting there doing *nothing*, while the other is at 100%. The icing on the cake is that there are *two* BackupPC_dump processes, each trying to consume as much CPU as they can--but they're both on the same core!

A typical top:

top - 13:07:44 up 36 min, 1 user, load average: 1.97, 1.89, 1.52
Tasks: 167 total, 3 running, 164 sleeping, 0 stopped, 0 zombie
Cpu(s): 46.1%us, 2.4%sy, 0.0%ni, 49.4%id, 2.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3924444k total, 3809232k used, 115212k free, 11008k buffers
Swap: 0k total, 0k used, 0k free, 3280072k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1731 backuppc 20 0 357m 209m 1344 R 100.0 5.5 24:14.07 BackupPC_dump
1679 backuppc 20 0 353m 205m 2208 R 92.5 5.4 21:54.89 BackupPC_dump

So, I have two CPU-bound tasks and they're both fighting over the same core.

Is there anything that can be done about this?

A quick aside about checksum caching: I very much *want* the ability to check to make sure if my backup data is corrupted *before* there is an issue, so I do not use checksum caching. So, yes, this puts much greater stress on disk I/O: both sides have to recalculate the checksums for each and every file. But the client can do it without monopolizing 100% of the CPU; the BackupPC side should be able to, too...

Tim Massey


Out of the Box Solutions, Inc. Creative IT Solutions Made Simple! http://www.OutOfTheBoxSolutions.com tmassey AT obscorp DOT com		22108 Harper Ave. St. Clair Shores, MI 48080 Office: (800)750-4OBS (4627) Cell: (586)945-8796

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/