Re: [BackupPC-users] Rsync to Windows client very slow (Was: Re: How to

rsync backups are quite slow. over slow links this is more than made up for by the decrease in network bandwidth used but on a high speed lan it is very slow. I am on gigabit from my desktop to my backuppc server but rarely exceed 6-8MB/s during a backup.

That makes sense some of the time as rsync will only be transfering checksum data but when there is a large file being transfered it is still quite slow. install iftop and watch a backup that has a new large file that is not in an exsisting backup and you will not see the transfer speed at anywhere near the LAN speed or hard disk speed because of I/O bottlenecks.

It would be nice to be able to use WindowsXP+'s index as a reference or have the rsyncd on the windows machine maintain all the checksum information but I know of no way and no plans to do this.

I/O is an obvious slow point here but rsync doesn't seem to be aware of I/O bottlenecks. it seems to scan files at random rather than trying to scan files in the on-disk-order on the client side and then cache disk writes on the server side to overcome the I/O bottleneck. Another, possibly the primary factor, in the I/O bottleneck is the way almost all modern filesystems handle small file creation, which includes all files smaller than the cluster size but also directories, softlinks, and hardlinks.

The arguement of storing files in a database has come up as a database is better able to overcome the I/O deficites of the disk storage by some intelegent caching but that would require more complex code. Alternatively using a fuse-based SQL filesystem would help overcome the I/O issue by taking advantage of MySQL's caching but fuse modules are notoriously innefficient and the overhead would negate the benefit.

One functional solution is to use a filesystem that has the caching benefits without any extra complexity. ZFS on BSD and Solaris fits the bill and performs very very well with backuppc. Unfortunately, none of the BSDs, Solaris, or openSolaris include backuppc and filling dependancies is just beyond many users as a large number of backuppc users/admins are all about Linux. In fact, ZFS can have cache disks in the storage pools and pre-write data sequentially from the RAM cache for re-organization later, making ZFS very effective at hiding the I/O bottleneck.

On Wed, May 21, 2008 at 4:19 AM, Kurt Jasper <kurt.jasper AT googlemail DOT com> wrote:

Ok,

thanks for (all) the information, now I understand the concept of
Rsync/SambaShareName, BackupFilesExlude and BackupFilesOnly.
You really helped me out, Thanks :-)

But unfortunately I've discovered another problem:

The backups of my windows xp clients take forever (nearly), while the
backup of the server (where backuppc is runnning) works fine. I've done
some testing and figured out that it is NOT an network issue, since I
get the full speed when executing rsync from the windows xp client.

I have transferred the cygwin directory of one of the clients (approx.
300 MB):

On Windows-Client:
rsync -av -h --progress /cygdrive/c/cygwin backuppc@server:~/
=> works fine!

On Server:
rsync -av -h --progress root@winxpclient:/cygdrive/c/cygwin ~/
=> takes very, verrrry long.

Browsing the web, I've found out that it may have something to do with
the Cygwin Enviroment Variables. I'm currently using the suggested
settings: CYGWIN = ntsec tty.

Question:
1) So what may be the problem?
What is the best way to back up windows xp clients? I don't understand
that running rsync from CLIENT-to-SERVER runs fine, but running from
SERVER-to-CLIENT is really slow.
2) Is is better to use Samba to backup windows clients?
Some clients are running XP (Home), so that I can not set custom file
permissions to allow access only the backuppc-user not to other clients
in the network.
3) Will rsyncd instead of rsync bring better performance?

Kind regards

- Kurt

PS: To give at least something back to this community, I thought about
writing down my experiences (from a beginners point of view), I hope
this is the right place:
http://backuppc.wiki.sourceforge.net/CommonConfigurationIssues
If not feel free to flame me :-0

Bowie Bailey wrote:

> Your RsyncShareName is '/', but your are specifying exclusions for the
> 'c' share. Try this instead:
>
> $Conf{BackupFilesExclude} = {
> '/' => ['/Dokumente und Einstellungen/*/NTUSER.DAT', ....
> };

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Rsync to Windows client very slow (Was: Re: How to BackupFilesExclude to exclude locked files on Windows XP)