Hi Adam,
thanks a lot for your long answer.
Am Monday 02 March 2009 20:44:33 schrieben Sie:
> Reinhold Schoeb wrote:
> > I set up a BackupPC server with several clients (1 MacPro, 2 MacBooks, 2
>
> Linux
>
> > PC and 2 Linux laptops). All clients are connected via Gigabit LAN with
> > full speed. My server is an AMD Sempron 2.2 GHz, 1 GB RAM, 4 x 750 MB
> > SATA harddisks connected by LVM and Debian Lenny.
> >
> > Everthing works perfect, but : Backup Speed is very low, for all machines
> > between 3 and 10 MB/s. I tried all options: with or without compression,
> > rsync, rsyncd, tar. Without any enhancement. All basic speed performances
>
> are
>
> > perfect : HD speed, LAN speed.
> >
> > That takes me to my question : The full backup size of my MacPro is about
>
> 1.6
>
> > TB. It took 2250 minutes or 1.5 days with 9 MB/s to make a full backup
> > and I had to not switch off the MacPro during that time. An incremental
> > backup took about 100 minutes - that was ok. But after 7 incremental
> > backups BackupPC is now trying to make another full backup. But since the
> > MacPro is only running during daytime, it does not come to an end.
>
> You didn't say which backup method you used when you timed those
> backups. Generally speaking, rsync would be the best/quickest method in
> my opinion, since you will transfer the least amount of data....
I tried all methods except smb and could not find big differences in speed.
And I have to say that 9 MB/sec is the best result which I get with my MacPro
as a client. My other machines do not come over 5 MB/sec.
> Basically, like any performance issue, you will need to find the
> bottleneck, and resolve it, and then find the next bottleneck and
> resolve it, etc... until you get the performance you need (not the
> performance you want which is usually too expensive)...
>
> Main things to look at:
> Memory consumption on client and server
The client has 8 GB RAM, my BackupPC server has 1 GB RAM. Both machines do not
use swap during backup.
> Disk IO on client and server
Both look perfect. Thats what hdparm says :
alphacentauri:/home/e1l52# hdparm -tT /dev/sda /dev/sdb /dev/sdc /dev/sdd
/dev/sda:
Timing cached reads: 1058 MB in 2.00 seconds = 529.87 MB/sec
Timing buffered disk reads: 204 MB in 3.01 seconds = 67.73 MB/sec
/dev/sdb:
Timing cached reads: 1110 MB in 2.00 seconds = 555.25 MB/sec
Timing buffered disk reads: 240 MB in 3.02 seconds = 79.52 MB/sec
/dev/sdc:
Timing cached reads: 916 MB in 2.00 seconds = 458.42 MB/sec
Timing buffered disk reads: 232 MB in 3.01 seconds = 77.13 MB/sec
/dev/sdd:
Timing cached reads: 862 MB in 2.00 seconds = 431.26 MB/sec
Timing buffered disk reads: 204 MB in 3.00 seconds = 67.99 MB/sec
> CPU utilisation on client and server
Thats what sar tells me about my server load :
16:25:02 CPU %user %nice %system %iowait %steal
%idle
16:35:02 all 58,21 0,00 34,07 6,66 0,00
1,06
16:45:02 all 56,63 0,00 34,98 3,49 0,00
4,90
16:55:02 all 51,65 0,00 31,25 15,12 0,00
1,98
17:05:01 all 46,54 0,00 30,62 22,17 0,00
0,67
17:15:01 all 58,46 0,00 34,59 6,95 0,00
0,00
17:25:02 all 54,70 0,00 35,91 9,39 0,00
0,00
17:35:03 all 57,65 0,00 37,39 4,97 0,00
0,00
17:45:02 all 53,36 0,00 34,44 11,94 0,00
0,27
17:55:02 all 49,53 0,00 29,45 19,11 0,00
1,91
18:05:02 all 36,25 0,00 33,04 25,87 0,00
4,84
18:15:02 all 46,57 0,00 27,28 17,31 0,00
8,84
18:25:02 all 46,35 0,00 27,13 14,64 0,00
11,87
18:35:02 all 48,38 0,00 28,15 13,67 0,00
9,80
18:45:01 all 57,14 0,00 36,30 4,30 0,00
2,25
18:55:01 all 48,61 0,00 30,83 6,25 0,00
14,31
19:05:02 all 52,04 0,00 34,70 9,44 0,00
3,83
19:15:01 all 53,92 0,00 38,91 6,59 0,00
0,58
19:25:02 all 55,69 0,00 40,78 3,54 0,00
0,00
19:35:02 all 56,09 0,00 40,33 3,46 0,00
0,12
19:45:01 all 56,15 0,00 39,67 2,66 0,00
1,52
19:55:02 all 55,60 0,00 39,78 3,88 0,00
0,74
20:05:03 all 57,59 0,00 40,46 1,94 0,00
0,01
20:15:02 all 62,72 0,00 37,28 0,00 0,00
0,00
20:25:01 all 61,21 0,00 37,78 1,01 0,00
0,00
20:35:01 all 61,12 0,00 38,30 0,58 0,00
0,00
20:45:02 all 61,01 0,00 38,14 0,85 0,00
0,00
20:55:02 all 51,68 0,00 31,36 1,88 0,00
15,08
> Some things that can reduce CPU with SSH+rsync is
> * Turning off encryption for SSH
I had a look at HPN-SSH, but since I had to replace the standard ssh on all my
machines (and especially on my Mac systems it doesn`t look so easy) I decided
to stay with standard ssh. But I played along with ssh encryption methods,
also without any big difference in speed. Thats for example what I get when I
do a simple scp from client to server :
e1l52@mebsuta:~$ scp -c blowfish test.zip alphacentauri:/tmp/
test.zip 100% 78MB 19.5MB/s 00:04
Which is the double speed I get with BackupPC. Now another speed test with a
simple ftp:
ftp> put test.zip
local: test.zip remote: test.zip
200 PORT command successful
150 Opening BINARY mode data connection for test.zip
226 Transfer complete
81773027 bytes sent in 3.90 secs (20482.8 kB/s)
ftp> get test.zip
local: test.zip remote: test.zip
200 PORT command successful
150 Opening BINARY mode data connection for test.zip (81773027 bytes)
226 Transfer complete
81773027 bytes received in 1.70 secs (47057.0 kB/s)
> * Turning off compression on SSH and in backuppc storage
I do not use SSH compression. I also tried BackupPC with or without
compression (level 3) - without any big difference in speed.
> Helping with IO:
> * On the server use RAID10 instead of RAID5/6 (you didn't tell us how
> your 4 x 750GB drives are configured)
I am using LVM for creating a 3 TB virtual device of my four 750 G harddisks.
> * Check what FS you are using on the server, and perhaps use some
> additional flags such as noatime (if appropriate) etc
Currently I am using EXT3 with noatime flag. I also tried XFS - not better in
performance.
> Also, look at how many backups are happening at the same time, consider
> reducing this to a single backup at least for performance tuning so you
> can work out whether what you have changed is better/worse.
I can say that even there is only one backup running. I do not get more that 9
MB/s transfer speed.
> Of course, also check the backuppc wiki for more information/detail on
> performance tuning...
>
> > How is the right way to deal with that ? Is it possible or makes it sense
> > to avoid additional full backups ?
>
> You also haven't told us which version of backuppc you are using...
I am using 3.1.0-4 from Debian lenny.
> In any case, it is important to do regular full backups, this can be
> more/less regular depending on your requirements. I increased the
> frequency to every 3 days to improve performance (using backuppc
> 2.1.2pl1 and doing remote backups).
>
> Hope that helps, for more assistance review the wiki, and then come back
> and provide some more information and more measurements...
Yes, of course. What is your opinion ? I think you agree that it is neccassary
to get the full backup time lower than the typical client uptime, which is in
my case about 12 hours per day. Otherwise a full backup will newer come to an
end and restarts every morning.
My guess is that my server CPU power is not good enough for BackupPC. What do
you think about that ? Is an AMD Sempron 2200 too slow for BackupPC ?
Thanks a lot for your help.
Reinhold
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
