i have tested the send and receive functionality of zfs on openbsd. the problem is that it sends the entire fileset, block by block. this is not going to work for remote replication within a reasonable timeframe.
rsync is about the only option. i know it does not scale well with file count but it still does the best job of syncing up filesystems remotely.
the best solution here is to get with the rsync team and see what can be done about the memory usage. maybe an option to write the file list to a temp file or maybe compress the file list in memory.
I do remote rsyncs for a large fileset with millions of files. It does work, reliably even, though i have 4GB of ram available on both computers.
I find that raid5 is quite evil for backuppc and/or rsync. though this is already known on this list, maybe the degree of evil is not known. raid5 will more than double the write time of an rsync transfer. one might think that this would be hidden behing the low bandwidth of a WAN connection but it is not because these are not parrallel transactions, they are serial so those I/O delays cause a waiting period before the next chunk is downloaded.
so, dont use raid5. do use raid1+0.
I have also tried some cluster filesystems recently and lets just put all of those on the 'dont use' list.
I have done massive amounts of testing and trial and error and have found that this is the best setup for my needs, and purhaps many people's needs
a raid set with raid1+0 running reiserfs on linux -or- zfs on bsd or solaris with hardware raid. zfs is dramatically faster because it is much much better at cacheing I/O than any filesystem I have tested. also, I find that exporting a zfs snapshot via iscsi and mounting that on the remote machine and doing the rsync *local* from the iscsi mount to the local backuppc share is MUCH(maybe 20-30%) faster that just regular rsync as iscsi is much better at handling I/O than rsync. use many spindles in raid0! pair the drives in raid1 and get as many spindles as possible. raid0 will do a good job of spreading I/O(raid5 does NOT). if using zfs, do NOT use compression, and do not do a raidz or raidz2 as you will have a parity penalty though it is much smaller than a regular raid5. if using bsd or solaris, use a hardware raid card(a good one) and run zfs on top of that, again dont use zfs' raid functionalty or you will get a parity penalty.
On Wed, Jul 23, 2008 at 12:25 PM, Les Mikesell <
lesmikesell AT gmail DOT com> wrote:
Holger Parplies wrote:
>
> Chances are, you
> should be aware of what will happen. But still: extensive, potentially
> destructive operations happening automatically without a chance of user
> interaction is something that belongs in the world of Windoze. Linux is
> becoming more like that fast enough already, so I won't let this chance pass
> to argument against it :-).
But this is the backuppc list where you might not be running Linux... I
still think zfs with it's incremental send/receive operation might be
the way to go to duplicate changes in the archive filesystem, but
haven't been able to do any testing myself yet.
> Of course there's also the chance of you simply plugging the drive in at an
> inappropriate moment, when you wouldn't want the sync to happen. You might
> notice it when logging in to call the script to start the sync, but you won't
> log in just to check, if you went to the trouble of automating the sync in the
> first place, will you?
>
> Just because you *can* automate something, that doesn't necessarily mean it's
> a good idea to do it. Then again, under some circumstances it may well be.
Another alternative that seems like it should work is to plug the
external drive into some other machine on a fast lan, export the
partition via iscsi, then add the iscsi device to the raid set. Really
the same scenario, but you wouldn't have to walk over to the server room
or keep the server at your desk.
