Hi,
backuppc AT kosowsky DOT org wrote on 2013-03-11 09:39:52 -0400 [Re:
[BackupPC-users] BackupPC_dump memory usage]:
> Les Mikesell wrote at about 08:08:15 -0500 on Monday, March 11, 2013:
> > On Mon, Mar 11, 2013 at 4:59 AM, ashka <shellgratuit AT gmail DOT com>
> wrote:
> > >
> > > I do have millions of files, I can't really get them out the backup
> though.
> > > Those files are really important, and I excluded them temporarily just to
> > > see how it was working out.
> >
> > It would be sort-of interesting [...]
... to get some answers to the questions we are asking. It's not out of
curiosity. I don't really care what your server looks like. I just don't like
wasting my time with guesses when you couldn't be bothered to give more than a
vague description of what you are seeing.
> > to see how much difference it would
> > make to run a 32-bit perl on the server.
With a 32-bit executable, you've got a maximum of 3 GB total available user
address space, I believe. Even if all data structures were only half as large
(and that should be true only for integers and pointers, not for strings),
you'd still need 5 GB, so you should run out of memory earlier. By all means,
try it out. You'd be using a different build of File::RsyncP, so that might
make a difference. Shouldn't really, but might.
> > But the solution is
> > probably [...]
Right. Probably. This has been discussed often enough that without specific
details we can just as well say "search the archives".
> It would also be interesting to see how much memory a "naked" native
> rsync of the same server takes up...
Yes, that's simple enough to try. Does it complete, how much max. memory, how
long?
> [...] other than the needs of File::RsyncP, the
> BackupPC_dump process itself shouldn't be consuming memory that scales
> with number of files.
That is also my understanding. So far, we haven't really established yet that
it is in fact number of files and not something completely different going
wrong (some sort of loop, maybe, for whatever reason). The original post is a
good example of ignoring the basics. We don't know anything about the setup
except that it's BackupPC 3.2.1. We don't even have an explicit confirmation
that rsync is being used as XferMethod. For all we know, he might have ported
File::RsyncP to <some obscure architecture> and introduced bugs in the
process.
> The only potential memory scaling of BackupPC
> itself would be with the size of an individual file that needs to be
> decompressed/compressed/compared but even so it should be mostly
> buffered...
Completely, I believe. There's the 1 MB buffer for determining the pool file
name and possibly several smaller buffers for chain comparisons. RStmp is
on-disk for a reason, though I can't say I completely understand how rsync
transfers work in BackupPC. Files larger than 1 MB should be fairly common,
though ;-).
> So, I think it behooves you to figure out where these 10GB are being
> consumed...
Yep. Presuming they are consumed and you're not just reading the wrong column
of your htop output.
Regards,
Holger
------------------------------------------------------------------------------
Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
endpoint security space. For insight on selecting the right partner to
tackle endpoint security challenges, access the full report.
http://p.sf.net/sfu/symantec-dev2dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
