|
Re: [BackupPC-users] Help before giving up on BackupPC
2014-05-15 11:34:32
Marco Nicolayevsky <marco AT specialtyvalvegroup DOT com>
wrote on 05/14/2014 08:47:24 PM:
> Hello all,
>
> I am using a pretty vanilla installation of BackupPC
and love the
> simplicity and fact that it just works as it’s supposed to.
>
> My problem arises when trying to back up windows
clients over rsyncd.
>
> I have 3 win boxes, all running Win7 and the
rsncd server.
> Functionally, they perform fine. I’m able to backup, restore, etc.
> My initial tests were with small folders and not the entire drive.
>
> HOWEVER… each win box has somewhere between
20GB and 3TB of data,
> and despite being on wired gigabit Ethernet, the 3TB machine ran for
> over 8 days and still wasn’t done before I pulled the plug and
> called it quits. Backing up 3TB over gig-e should be able to be
> accomplished in under 1 day, so I’m at a loss of what to do next.
Here are some hard numbers for one of my systems.
All targets Windows-based, using rsyncd, no SSH, no encryption, no
compression. Servers are using 7200RPM SATA drives.
Full size: 1372GB, 1114587 files. Time
to complete: 1175.3 minutes.
(Intel Xeon E5 CPU, 16GB RAM, 6-drive Software RAID-6)
Full size: 627GB, 452759 files. Time to
complete: 682.4 minutes.
(Intel Atom D610 (really slow) CPU, 4GB RAM, 5-drive
Software RAID-6)
Now, these are established backups. I've found
that initial backups can be 2-3 times as long. And at 3TB, you're
nearly triple of the bigger one. So, that could mean that the initial
backup can take 6000 minutes or more, or four days. And personally,
I've found that compression can double or triple the time again. Worst-case
of all of that could be two weeks! So it is possible that you're
seeing correct operation for that first backup, and that future backups
should be much better.
Also, those are fulls. Incrementals take 4 hours
and 1 hour, respectively.
> Is rsync “really” that slow?
No: rsync is only going to be marginally slower
than a non-rsync copy, even on the first time, assuming you're not limited
by something else (CPU or RAM) that would not be a limit for a normal copy.
That could be related to the number of files: that's
an area where rsync can get tripped up. As you can see, I've got
>1 million files, so the definition of "too many" is pretty
big. But if you had, say, 10M files, maybe that's an issue to consider.
> Am I doing something wrong?
*You*? Who knows? Is something wrong?
Possibly.
> Is it
> limited to just the windows client?
No.
> When I was evaluating Bacula, I
> was able to do the same backup in 1/10th the time (or at least it
> felt that way since I don’t have hard numbers).
You've got hard numbers above to consider as a comparison
and see if they will fit your needs.
> Before giving up on BackupPC and considering
an alternative, can
> someone give me some advice?
Yes. First, you've given us *NOTHING* to go
on other than "it's slow". It's like telling a mechanic
"my car doesn't run right." Of course, you're probably
expecting to pay the mechanic, so he's incented to ask lots of questions
to figure things out. I think it's pretty telling that I can't see
a single reply to your request. While your request includes a lot
of words, there was almost *NOTHING* of substance in your request except
a *VERY* brief description of your hardware. We have almost no description
of what your backups look like (size and number of files, for example),
what your backup jobs look like (compression being a very big one), or
what your backup server is doing (CPU, memory or disk usage). And
we're not getting paid, so we're not really incented to ask you a lot of
questions. But I'll give you a few:
First, is your system limited in some way? Are
you hitting 100% CPU, or swapping (not enough RAM), or are your disk overloaded,
or something else? Learn to use top and vmstat (or dstat).
Is your infrastructure fundamentally limited in some
way? Have you tried doing a straight-up copy from a target to your
backup system to make sure that the underlying infrastructure is capable
of delivering what you expect it to? If you can only get 1-2MB/s
copies using SMB, tar, NFS, FTP, etc. , then that's all you'll get with
BackupPC, too. But if you can get 70MB/s copies between the same
two systems some other way, then we can expect better of BackupPC. (But
all that does is re-ask the question of what is limiting you.)
From my e-mail, you know what is possible and reasonable
to get. If you're far away from those results, then you need to figure
out what is different about your system and causing the slowdown.
The second thing to try is to simplify things. For
me, the first thing I do is disable compression. In today's multi-core
universe, compression is rapidly becoming a bottleneck again. The
compression algorithms in common use today do *not* use multiple cores.
On a system with more than a couple of disks I can easily max out
one core with compression.
Or, try to use SMB instead of rsyncd. I would
not suggest that as a final stop (I really like rsyncd for my systems),
but it will help you to see if there is a problem with something unique
to rsync, rather than fundamentally wrong with your system.
Another option would be to expand your testing from
a very small section to something larger: say, 100GB. That
is big enough to be somewhat representative of the whole, but should be
able to complete quickly enough even with compression, encryption, etc.
to get some baseline numbers to work with, including both the *first* and
*additional* full backups. That way, you might find that the initial
backup will take a week, but each additional backup after that will only
take 12 hours and you're OK with that. Or, you might find that things
are still broken, but now it won't cost you a week of your life every time
you want to test.
> If it helps, the server is beefy, quad-core i5
proc w/ 4gb ram and 6TB raid5.
Eh. That's pretty baseline for anything you
would call a server today, and an i5 processor screams client-grade, not
server grade. It's not especially beefy, but BackupPC doesn't really
require beefy. See my stats on a really crummy embedded-class CPU
and software RAID-6! :)
Tim Massey
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs _______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
|
|
