BackupPC-users

Re: [BackupPC-users] Backup Data Transfer Speed

2013-04-07 21:50:42
Subject: Re: [BackupPC-users] Backup Data Transfer Speed
From: John Habermann <jhabermann AT cook.qld.gov DOT au>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Mon, 8 Apr 2013 11:36:46 +1000
I have one client that shows a speed of 51.24 MB/s
for its last full backup but this is definitely not a indication of the
wire speed of the link between it and the server as that is a PTP
wireless link with a max speed of 150 mb/s.
The client is linux file server with a i5 processor 4 GB of ram and a 4
disk raid 10 array using 1.5 TB RE4 drives. The backup server is a
Xeon X3320 2.5 Ghz processor with 4 GB ram and a 6 disk Raid 10 array
largely consisting of RE4 2 TB disks with a the last remaining green 2
TB disk just failing on me on Friday actually so have to go and swap
that out with a RE4 drive today. 

I think the key thing with why the last full got that speed was that
out of a 1.3 TB back up there where only 588 new files (1.6 GB in
total) so most of the performance will come down to the speed with
which the client and the server can do the calculations to determine
the changes on the client and in this is limited more by cpu, memory
and disk speed. I have another file server in the same room as the
backuppc server so has a GB link to the backup server and this
has hit 66 MB/s for a full backup but this is on a 4 disk Raid5 array
with SCSII 15k rpm disks and the file server has a XEON cpu. The full
backup speed again though varies from 18 MB/s to 66 MB/s on the amount
of new files in the backup so again with rsync based backups the
network speed is not a factor.

On Sun, 07 Apr 2013 18:33:39 -0400
"Phil K." <phillip.kennedy AT yankeeairmuseum DOT org> wrote:

> Another consideration is file system. Once you get beyond basic
> hardware, there are a number of tweaks to look at on the software
> side.
> 
> Your setup isn't terrible, but there's a lot of room for improvement.
> As mentioned upstream, RAM would help. It wont be a cure all, but
> it's a start. WD Cav Greens are insanely slow. Black or Red are a big
> step in the right direction, but ideally I'd get that pool into a
> RAID 1 (again as mentioned by others.)
> 
> Last thing to consider; Just because the switch can move data at a
> gigabit per second doesn't mean that it's going to be getting a gig
> of data per second from the host, nor is the server going to be able
> to write a gig of data per second. BackupPC isn't always moving data,
> especially with Rsync. There's a great deal of time spent listing
> directory and partition contents, hashing to compare existing data to
> potentially new. The best speeds I've personally seen are in the
> 25-30 Mbps range.
> 
> Bottom line, I'd bet you're getting 4 or 6 mbps. There's some room
> for you to improve, but you're not going to see backups in the 50-60
> Mbps range.
> 
> Gary Roach <gary719_list1 AT verizon DOT net> wrote:
> 
> >Hi,
> >
> >I am modifying my whole backup system for greater volume and speed.
> >The
> >
> >new system is as follows:
> >
> >     Software - Backuppc using rsync (without SSH)
> >     Server -    D865PERL Motherboard
> >                     Pentium 4 - 2.4 GHz Processor
> >                     IDE System hard drive
> >                    SATA, 1 TB WD green backup data storage disk.
> > (S/B 
> >1.5 Gb/S)
> >                     1GB system memory
> >                     PCI bus
> >                     Intel PWLA8391GT PRO/1000 GT PCI Network Adapter
> >                     Cat 6 cable
> >
> >     Clients (2)
> >Intel DP55KG   motherboard
> >                 Intel i5-750 Processor 4 core, 2.66 GHz
> >                 Hard Drive - Western Digital WD5000AADS -500GB,
> > 32MB 
> >Cache, SATAII (3 Gb/s)
> >                 PCIX bus
> >                 2 GB system memory
> >                 Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
> >                 Cat 6 Cable
> >
> >I have read all sorts of complaints about this type setup and need
> >some
> >
> >advice. It would appear that I should get around 1 Gbit/S transfer
> >rate
> >
> >for continuous data transfer. If this is not true, why not and what
> >can
> >
> >I realistically expect.
> >
> >Gary R
> >
> >
> >
> >
> >
> >------------------------------------------------------------------------
> >
> >------------------------------------------------------------------------------
> >Minimize network downtime and maximize team effectiveness.
> >Reduce network management and security costs.Learn how to hire 
> >the most talented Cisco Certified professionals. Visit the 
> >Employer Resources Portal
> >http://www.cisco.com/web/learning/employer_resources/index.html
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >BackupPC-users mailing list
> >BackupPC-users AT lists.sourceforge DOT net
> >List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> >Wiki:    http://backuppc.wiki.sourceforge.net
> >Project: http://backuppc.sourceforge.net/
> 


-- 
John Habermann
IT Officer
Cook Shire Council
10 Furneaux St
Cooktown 4895
ph 40820577

------------------------------------------------------------------------------
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire 
the most talented Cisco Certified professionals. Visit the 
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/