I'm not qualified to disagree Cesar, but my understanding is that the issue:
A - Has nothing to do with the size in TB of the filesystem, but the
number of hardlinks - therefore the number of source files, the
frequency of backups and the number of clients.
b Wasn't/isn't related to memory leaks, but the sheer size of the
array that needs to be built in RAM to track all the files being
processed.
There have been extensive discussions on this quite recently,
including about recent versions of rsync improvement of memory leaks
not fixing the core issue, which remains a non-trivial one once the
number of links reaches a critical point relative to the RAM resources
available. Your situation hasn't hit that point yet, and perhaps never
will.
I'm simply pointing out a potential gotcha to watch out for, so the OP
can plan and design with that in mind, and giving alternative
workarounds for if/when the issue does arise, or to be used to prevent
that possibility.
On Fri, Mar 11, 2011 at 4:08 PM, Cesar Kawar <kawarmc AT gmail DOT com> wrote:
>> On Fri, Mar 11, 2011 at 10:56 AM, Rob Poe <rob AT poeweb DOT com> wrote:
>>> I'm using RSYNC to do backups of 2 BPC servers. It works swimmingly, you
>>> plug the USB drive into the BPC server, it auto-mounts, emails that it's
>>> starting, does an RSYNC dump (with delete), flushes the buffers, dismounts
>>> and emails.
>>
>> Sounds great Rob, would you be willing to post the script?
>>
>> Rsync'ing is all fine and good until your hardlinked filesystem (I
>> don't know the proper term for it, as opposed to the pool") gets "too
>> big". It's a RAM issue, and an unavoidable consequence of rsync's
>> architecture - I'm not faulting rsync mind you the kind of filesystem
>> that BPC (and rdiff/rsnapshot etc) build over time is a pretty extreme
>> outlier case.
>>
> That is not a problem anymore with latests versions of rsync. I've been using
> this technique for a year now with a cpool of almost 1Tb with no problems.
>
> Don't expect it to run on a celeron machine as it requieres big processors.
> Rsyncing 1Tb of compressed hardlinked data to a new filesystem is a very cpu
> intensive task. But it does not leak memory as before. You can relay on rsync
> to mantain a usb disk for off-site bakups.
>
>
>> Therefore an alternative solution when that time comes is adding RAM,
>> and yet another is periodically switching to a new target filesystem
>> and deleting the old one after the new one's had a chance to build up
>> its history. In fact this last is the easiest way of all to migrate
>> over for those that didn't design their disk infrastructure to handle
>> future growth (e.g. expandable filesystems built on LVM.)
>>
>>
>>> On 3/10/2011 8:35 PM, hansbkk AT gmail DOT com wrote:
>>>> On Fri, Mar 11, 2011 at 3:46 AM, Michael Conner<mdc1952 AT gmail DOT com>
>>>> wrote:
>>>>> That is good to know. Actually things are a little better than I thought,
>>>>> the spare machine is Dell Dimension 2400 with a Pentium 4, max 2 gb
>>>>> memory. So I guess I could slap a new bigger drive into it and use it. My
>>>>> basic plan is to get backups going to one machine and then dupe those to
>>>>> an NAS elsewhere in the building. While we have a small staff, our
>>>>> building is 62,000 sq ft with three floors, so I can get them physically
>>>>> separated even if not really off site. For the web server, we have a two
>>>>> drive raid set up with two spare drive bays. Besides backing up with BPC,
>>>>> I would also dupe the drive on a schedule and take off site.
>>>>
>>>> To expand on Jeffrey's comment below - the idea of "duping" your
>>>> backups is fraught with issues when the BPC filesystem gets past a
>>>> certain size.
>>>>
>>>> To handle the creation of a redundant backup, I would advise one of
>>>> the following:
>>>>
>>>> A - Periodically use BPC to run a full backup set to a different
>>>> target filesystem - this is simplest and quite likely the fastest, and
>>>> only becomes an issue if you have a limited time window - in which
>>>> case LVM snapshotting can help as Jeffrey mentioned.
>>>>
>>>> B - use a block-level cloning process (like DD or its derivatives, or
>>>> Ghost-like COTS programs if that's more comfortable for you, to do
>>>> partition copying to a removable drive. Some use temporary RAID1
>>>> mirrors, but I don't recommend it.
>>>>
>>>> C - a script included with BPC called BackupPC_tarPCCopy, designed to
>>>> do exactly this process.
>>>>
>>>> Where you run into problems is trying to copy the hardlinked BPC
>>>> filesystem over at the **file level** - even rsync will choke when
>>>> you've got millions and millions of hardlinks to the same inodes to
>>>> keep track of.
>>>>
>>>> BTW even if you don't do snapshots, you should use LVM from the
>>>> beginning as the basis for your new BPC target filesystem, gives you
>>>> future flexibility to avoid having to do the above any more than
>>>> necessary.
>>>>
>>>> Hope this helps. . .
>>>>
>>>> On Fri, Mar 11, 2011 at 5:04 AM, Jeffrey J. Kosowsky
>>>> <backuppc AT kosowsky DOT org> wrote:
>>>>> Keep in mind the point that Les made regarding backing up BackupPC
>>>>> archives. Due to the hard link structure, the fastest way to back up
>>>>> any reasonably large backup is at the partition level. This also makes
>>>>> it hard to enlarge your archive space should you outgrow your
>>>>> disk. One good solution is to use lvm since you can
>>>>> enlarge/expand/move partitions across multiple disks. You can also use
>>>>> lvm to create partition snapshots that can then be replicated as backups.
------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
