I thought I would give an update to this old thread. To those unfamiliar with
this thread, basically, I was looking for a way to do backuppc data
synchronization, and we brainstormed a way using the ZFS file system (as it has
an rsync-like ability (sending only changes), but at the filesystem/block level
(like dd), rather than the file level of rsync, so the problems that plague
trying to rsync a BackupPC pool shouldn't affect ZFS send/receive). Then that
topic expanded into using ZFS with deduplication (which is block based, to
compliment the dedup of BackupPC's pools, which is file based) to further cut
down on disk usage. It is on that subject that I wanted to report my findings.
A spare Supermicro server was recently acquired that had an Opteron quad core,
16GB of RAM, and a RAID-5 array with 4 250GB drives, giving it a 750GB array
(25% less than my production system, but good for an experimental setup). So I
decided to try and use it as an experimental ZFS/BackupPC box. I started by
loading CentOS 6.4 on it, installed ZFSOnLinux (I know, experimental, but
that's exactly what this is), and installed BackupPC. I created a single disk
ZFS pool from a partition of the array in the same location relative to / as it
is on my production BackupPC box & enabled both dedup & compression. I then
copied over /etc/BackupPC from production to my test box, and modified the
server config to not do any compression ($Conf{CompressLevel}=0)--not doing
this completely negated dedup's abilities. Once I created the basic BackupPC
pool structure on the ZFS pool, I started the BackupPC service, and let her go
for several days (but monitored her), accumulating the same backups that my
production box does.
It should be noted that this box needed ALL 16GB of RAM for the dedup feature,
but it never crashed, or even used swap significantly, and performance remained
reasonable. Over the course of 7 days, this box has been extremely successful
in its dedup & compression features. At 25% smaller total disk space, I'm
currently at 2.17x Dedup rate (really good!), and it is storing nearly as many
backups as production is, with more space free!
I have not yet had a second box to play with to do ZFS transfers, but when I
do, I will report on that too.
Thanks,
--Mark
-----Original Message-----
From: Tyler J. Wagner [mailto:tyler AT tolaris DOT com]
Sent: Thursday, March 07, 2013 10:14 AM
To: General list for user discussion, questions and support
Cc: Mark Campbell
Subject: Re: [BackupPC-users] BackupPC Pool synchronization?
On 2013-03-07 14:34, Mark Campbell wrote:
> My thinking at this point is that I'll leave the pooling be--it may
> require some extra CPU cycles & RAM from time to time, but my
> understanding of the zfs dedup & compress features are that they
> should be transparent to BackupPC, so while pooling in BackupPC won't
> avail much, it probably wouldn't hurt anything either.
Except that it's the pooling (hardlinking) that makes pool synchronization suck
so badly. Although perhaps ZFS mirror might make that better, I'd much rather
disable pooling entirely (disable the linking process), and then just use rsync
to sync the backuppc/pc tree between primary and secondary hosts.
Regards,
Tyler
--
"... I've never seen the Icarus story as a lesson about the limitations of
humans. I see it as a lesson about the limitations of wax as an adhesive."
-- Randall Munroe, "XKCD What IF?: Interplanetary Cessna"
------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
