On Thu, Mar 20, 2014 at 12:21 PM, Timothy J Massey <tmassey AT obscorp DOT com> 
wrote:
>
> >
> > You've used BackupPC before, Les, right? ;-)
> > BackupPC prefers pool reads over writes when possible, and it typically
> > accesses large amounts of data almost randomly. Caching metadata will help,
> > caching data likely won't.
>
>
> I've given up explaining this to Mr. Mikesell.  I've posted info to the list 
> several times showing the performance on BackupPC servers that had 512MB 
> (Megabytes) of RAM.  Zero swapping, and something like 300MB used for 
> caching.  I have then upgraded them to 4GB of RAM (8 times as much RAM!) and 
> saw a whopping 15 or so minutes savings on a backup that took 10 to 12 hours. 
>  (In fact, the only real reason I use 4GB minimum across the board now is 
> that I ran into a problem where a fsck wouldn't complete without more RAM.)

4GB is still a tiny amount of RAM these days.  Try 20+.  You basically
want to cache all of your pool directory/inode data plus the pc backup
trees being traversed by rsync.  And without threshing it out with the
rsync copy of the remote trees that are being processed.


> And I haven't seen that RAID-5 has been that much of an issue.  As you 
> mention, BackupPC is *READ* heavy so the RMW penalty doesn't hurt much.  My 
> experience with both RAID-5 *AND* RAID-6 (even software RAID-6!) is just fine.

Sure, they work.  But they force all of the disk heads to seek every
time unless you have a large number of disks, and every short write
(and most of them are with all of the directory/inode operations
happening), is going to cost an extra disk revolution time for the
read/modify/write step.  And that 'read heavy' assessment is somewhat
optimistic if you have any large files that are modified in place.
There you get the worst case of alternating small reads/writes in
different places as the changes delivered by rsync are merged into a
copy of the old file.

> Of course, I have a minimum of 4 drives in a RAID array (6 minimum for 
> RAID-6), so I'm usually bottlenecked somewhere else anyway, such as a single 
> 1Gb link.  It's not hard to mange writing 70MB/s of data!

Seriously?  You have rsync pegging a 1Gb link for long periods of time
anytime but the first run?  Do you have something creating a lot of
huge new files?  My times are mostly constrained by the read time on
the targets with a relatively small amount of data transfer.

-- 
   Les Mikesell
      lesmikesell AT gmail DOT com

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/