Les Mikesell wrote at about 12:16:49 -0500 on Monday, March 11, 2013:
> On Mon, Mar 11, 2013 at 11:57 AM, <backuppc AT kosowsky DOT org> wrote:
> >
> > I still suggest trying a native rsync copy of the server to determine
> > max memory usage. Then we will know if it's a BackupPC or an rsync
> > (protocol <= 28) problem. If it's an rsync problem, buy more memory or
> > split into smaller shares. If it's a backupppc problem, then fix the
> > problem.
>
> I'm not sure how it helps to know if native rsync works. Perl's data
> structures are bigger to permit its dynamic typing concepts and
> reference counting so it will naturally need more memory.
It should still be proportional...
Plus, the scale-up depends in large part how File::RsyncP is
implemented... I believe some of it is implemented in C (hence, the
need for compilation) and it is possible/likely that some of the data
is stored in more efficient C-like data structures.
Indeed, if File::RsyncP blindly used Perl memory storage, we *all*
would have encountered memory issues on much smaller file sets. For
example, on a 64 bit platform (not necessary 64 bit Perl), storing
an integer takes 24 bytes, storing a string takes 48 bytes plus
multiples of 8 bytes to store the contents. Hashes, take 150-200 bytes
just for each hash key, not including the memory required to store
the hash value.
As they say, "to estimate memory usage of Perl, assume a reasonable
algorithm for memory allocation, multiply that estimate by 10, and
while you still may miss the mark, at least you won’t be quite so
astonished."
Given that I can back up shares containing 500,000+ files on a
plugcomputer with only 512MB of memory with no swap and still have
memory to spare for 2-3 simultaneous backups, I think it's
likely that File::RsyncP is not just blindly using inefficient
perl storage...
------------------------------------------------------------------------------
Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
endpoint security space. For insight on selecting the right partner to
tackle endpoint security challenges, access the full report.
http://p.sf.net/sfu/symantec-dev2dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
