|
Re: [BackupPC-users] Split share into several shares
2008-06-09 20:06:59
WLAN is going to slow you down because of latency. rsync does quite a bit of 2 way communication. in fact it does it for every single file it inspects as it sends the checksum back and forth. wireless is a killer here, it often has latencies that are longer than a remote DSL site!
The other thing is that you should measure your backup performance by filecount with backuppc as filecount has as much or more effect of backup time as file size, especially when dealing with small files. the double I/O expense(I/O on client AND server) and bandwidth use of the rsync checksum, as well as the time of the file list generation and transfer can be much much higher that the cost of simply transfering a small file.
you can do a bit to spead these things up with some added rsync flags. for instance, if you set your --block-size for the checksum to something less than but close to your MTU, that will help make sure you dont send checksums inefficiently. for instance, if MTU is 1500(thats bytes) and you are sending a file that is 7500 bytes, it is much much better to have block size=1500 so there are only 3 chunks sent rather than have the blocksize=1000, which would require 8 chunks, or blocksize=2000, which would require 6 chunks.
if you use -u you will skip any files that are the same size and have the same timestamp(or the backuppc file is newer, but i wouldnt expect that). that can be an issue if you are backing up a system that is not diligent with timestamps. this may help a little as it wont even checksum those files, saving I/O.
you could also do the option -W if your network connection is faster than your I/O, as it will skip the incrementing algorythm and just transfer the whole file. This will save a ton of I/O but will use up proportionaly more bandwidth. I have this running on a few hosts on one of my bpc servers because the clients are on gigabit, so i have no network bandwidth concerns. a windows XP client with 15GB of data on takes about 5 minutes to transfer the data, and a minute or so for the file list which is about 3 times faster than using the incremeting algorythm but is immensly bandwidth intensive. you still have to deal with the checksum locally for the hardlinking algorythm.
Alternatively, backup using smb on the local network. no bandwidth savings at all and you also loose access to some files and file permissions.
On Sat, Jun 7, 2008 at 1:24 AM, Hendrik Friedel < hen_mail AT web DOT de> wrote:
Hello,
interesting. Thanks for your reply.
My backups take longer. Much longer. My last incremental
took nearly 280 minutes for about 600 MB. This is over a 54 Mbit Wlan
connection. But even over this connection 600MB are transferable in a fraction
of 280 minutes... I really don't know what's the problem.
Have you checked, how much time you save by splitting the
backup?
Greetings,
Hendrik
I actually do this on my tru64 unix and my sco unix machines.
This works quite well. It also uses up way less RAM with rsync, which is
good because some of the tru64 machines don't have a lot of extra RAM.
I have one specific tru64 alpha 4cpu machine that I have 4 hosts setup
for and I trigger all 4 hosts for backup at the same time with cron as they
are 1Ghz cpus and I can peg all 4 and complete backupps much quicker as my
scsi disk array is much much faster that the cpus can handle. I have one
backup the / with exclusions for each directory I will backup with one of the
other backups, then the other machines are backing up all the other
directories. I just did some guess work on how to split it up
efficiently by comparing file count and then filesize. This works very
well and I can get an incremental backup done in about 20 minutes and a full
in under an hour. its about 12GB worth of data, but an alpha es45 is not
exactly the fastest machine around these days. I also dont have many
files that change, maybe 100, but a few of them are 1GB+ each day.
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php _______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
|
|
