BackupPC-users

Re: [BackupPC-users] FTP, large files, and memory use

2011-04-25 02:34:41
Subject: Re: [BackupPC-users] FTP, large files, and memory use
From: "Michael Stowe" <mstowe AT chicago.us.mensa DOT org>
To: "John Sellens" <jsellens AT syonex DOT com>
Date: Mon, 25 Apr 2011 01:32:02 -0500
> On Fri, Apr 15, 2011 at 07:27:29PM -0500, Michael Stowe wrote:
> | > I don't think Windows is relevant here, I think the problem is because
> | > it's a large file and backup is via FTP.
> |
> | Find out!  See if you can ftp the file from a command line client.
>
> The strace I included shows the file being read from the the remote
> side and more memory being allocated on the server side.  Eventually
> BackupPC_dump hits its limit and can't allocate memory, gives up, and
> writes "Out of memory!" to the LOG file.

Yes, but this doesn't imply that the file can be FTP'ed at all.  I still
recommend you find this out.

> I'll have to look into the FTP libraries and see if I can find an
> alternate approach that perhaps FTPs the file into a local tmp file.
> The current approach for FTP in BackupPC clearly doesn't work for
> larger files.

I'm not sure I'd recommend it in any case.

> | > (I'm using FTP because my Windows rsync of choice - DeltaCopy - seemed
> | > to be confused by Windows 7's multitude of links/shortcuts, and we
> | > don't want to use a share on portable machines, out of a belief that
> | > we can control FTP better than we can control shares.)
> |
> | An intriguing belief, as is the concept that rsync is confused by links.
> | Mine isn't, but I don't use DeltaCopy.
>
> I didn't say rsync was confused by links, I said DeltaCopy ran into
> problems on Windows 7.  Though I did use the wrong terminology - it's the
> Windows "junctions" that it has trouble with.  My guess might be that
> DeltaCopy (or the underlying windows built rsync) needs to treat
> a junction the same as a symbolic link.  Some others ran into the
> same problem and mentioned it here:
>     http://forums.aboutmyx.com/forums/posts/list/176.page

It sure seems like your problem is DeltaCopy, not rsync.  For the record,
my cygwin copy of rsync treats Windows links like ... links.  (Windows 7
uses links, not junctions, for things like "documents.")

Seems to be that it would be easier to simply locate a non-broken version
of rsync than to try to get ftp to do something it's not particularly good
at.


> Cheers!
>
> John
>


------------------------------------------------------------------------------
Fulfilling the Lean Software Promise
Lean software platforms are now widely adopted and the benefits have been 
demonstrated beyond question. Learn why your peers are replacing JEE 
containers with lightweight application servers - and what you can gain 
from the move. http://p.sf.net/sfu/vmware-sfemails
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>