Tim,
First off, thank you very much for your insightful and helpful approach, despite the limited information I provided. And thank you as well for being the “unpaid” mechanic (good analogy). I will try to present a detail account. Please read
my comments to your questions below:
Environment:
·
Server (using gigabit wired Ethernet on a managed switch)
o
running under VMWare ESX5.5
o
VM allocated w/ 4gb ram, all procs/cores, and max cpu/mem bursts
o
intel i5-3330 3GHz
o
24gb ram
o
Backup location 5 disk hardware raid 5 under Centos 6.5
§
Backup avail space 6 TB
·
Windows Workstation Client (using gigabit wired Ethernet on a managed switch)
o
Intel i7-4770K cpu 3.5GHz
o
32 gb ram
o
2 internal logical drives
§
C: HW raid 0 comprising of TWO 120GB SSD drives (238gb capacity)
·
89.5GB used (162,337 files)
§
D: HW raid 5 consisting of 4 x 2TB (5589gb capacity)
·
1.5TB used (213,594 files)
I have not completed a full backup of this workstation yet. I have done many incremental removing exclusions in an attempt to incrementally backup the entire computer (drive c &d) in steps.
I will disable compression and re-attempt, although I will not be able to report the information for xx days until its complete.
> Now, these are established backups. I've found that initial backups can be 2-3 times as long. And at 3TB, you're nearly triple of the bigger one. So, that could mean that the initial backup can take 6000
> minutes or more, or four days. And personally, I've found that compression can double or triple the time again. Worst-case of all of that could be two weeks! So it is possible that you're seeing correct
> operation for that first backup, and that future backups should be much better.
>
So I guess compression is out. I will disable. Thank you for the info.
> Also, those are fulls. Incrementals take 4 hours and 1 hour, respectively.
>
> Is rsync “really†that slow?
>
> No: rsync is only going to be marginally slower than a non-rsync copy, even on the first time, assuming you're not limited by something else (CPU or RAM) that would not be a limit for a normal copy.
>
> That could be related to the number of files: that's an area where rsync can get tripped up. As you can see, I've got >1 million files, so the definition of "too many" is pretty big. But if you had, say, 10M files,
> maybe that's an issue to consider.
I don’t have anywhere near a million files, so I guess my workstation should be quicker. I suspect vmware is causing *some* penalty, but it should not be that great. Regardless, it’s what I have so that’s got to work fine.
> Yes. First, you've given us *NOTHING* to go on other than "it's slow". It's like telling a mechanic "my car doesn't run right." Of course, you're probably expecting to pay the mechanic, so he's incented to ask
> lots of questions to figure things out. I think it's pretty telling that I can't see a single reply to your request. While your request includes a lot of words, there was almost *NOTHING* of substance in your
> request except a *VERY* brief description of your hardware. We have almost no description of what your backups look like (size and number of files, for example), what your backup jobs look like (compression
> being a very big one), or what your backup server is doing (CPU, memory or disk usage). And we're not getting paid, so we're not really incented to ask you a lot of questions. But I'll give you a few:
>
Yes, I’m sorry. Your points are well made. I’m trying to correct my mistake in this email.
> First, is your system limited in some way? Are you hitting 100% CPU, or swapping (not enough RAM), or are your disk overloaded, or something else? Learn to use top and vmstat (or dstat).
CPU normally sits around 50%-70% depending if a 2nd job kicks in. I’ve used “iostat -k 5” and %iowait from 0% to as much as 40% but usualy hovers around 10%-20%. I was unaware of dstat, but I’ll start running it when I perform
the full backup
While doing some googling, I read somewhere about adjusting the TCP buffers for rsync. I’ve applied the following line in my rsync.conf file on the windows machine:
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=65536
Have you experimented with this or something similar in order to improve performance? I don’t have any hard numbers with/without (yet), but thought I would mention this.
> Is your infrastructure fundamentally limited in some way? Have you tried doing a straight-up copy from a target to your backup system to make sure that the underlying infrastructure is capable of delivering
> what you expect it to? If you can only get 1-2MB/s copies using SMB, tar, NFS, FTP, etc. , then that's all you'll get with BackupPC, too. But if you can get 70MB/s copies between the same two systems some
> other way, then we can expect better of BackupPC. (But all that does is re-ask the question of what is limiting you.)
No, my baseline was based on my prior experience with bacula, and a trial of Retrospect (which I quickly discounted). Both software I was able to fully back up this workstation within 4-8 hrs—not days. This is what started to get me worried.
> From my e-mail, you know what is possible and reasonable to get. If you're far away from those results, then you need to figure out what is different about your system and causing the slowdown.
> The second thing to try is to simplify things. For me, the first thing I do is disable compression. In today's multi-core universe, compression is rapidly becoming a bottleneck again. The compression algorithms
> in common use today do *not* use multiple cores. On a system with more than a couple of disks I can easily max out one core with compression.
Yes, I learned this while experimenting between gzip and pigz. For now, I’ll disable compression.
> Another option would be to expand your testing from a very small section to something larger: say, 100GB. That is big enough to be somewhat representative of the whole, but should be able to complete
> quickly enough even with compression, encryption, etc. to get some baseline numbers to work with, including both the *first* and *additional* full backups. That way, you might find that the initial backup will
> take a week, but each additional backup after that will only take 12 hours and you're OK with that. Or, you might find that things are still broken, but now it won't cost you a week of your life every time you
> want to test.
GREAT idea. Before I embark on trying to do a full backup, I’ll isolate ONE folder with some large files (around 100gb) and focus on doing some tests on that with/without compression. I will report my findings.
> If it helps, the server is beefy, quad-core i5 proc w/ 4gb ram and 6TB raid5.
Tim, again, thank you for taking the time and interest to both help and point out the vagueness of the email. I will get back with you.
Thanks,
Marco
