Why don't you ask the developers to write a script that creates one or a 
few tar files out of this massive number of files?
The execution of that script could be triggered via http request (with 
authentification). On the backuppc side you could call this script via 
pre backup command before every backup... Then just transfer the files 
in whatever way you like.

This way they don't need to give you access to their system but you 
could do fast and easy backupps.



Am 16.12.11 16:57, schrieb Les Mikesell:
> On Fri, Dec 16, 2011 at 9:00 AM, Jean Spirat<jean.spirat AT squirk DOT org>  
> wrote:
>>> Excuse my "off topic"-ness, but with that many small files I kind of expect 
>>> a
>>> filesystem to reach certain limits. Why is that webapp written to use many
>>> little files? Why not with a database where all that stuff is in blobs?
>>> That whould be easier to maintain and easier to back up.
>>>
>>> Have fun,
>>>
>> fortunately it is not in my power to discuss choice of the developper my
>> only job is to try to figure a way to make backup of the thing work :)
>>
>> i am sure most know the pain, i allready strive to exclude caching
>> directory because devs seems to invent new way to name them every two
>> days (temp temporary temporaire<enter your custom name>  cache mycache
>> appli-cache img-cache  variation with plurals...).
>>
>>
>> the backuppc server has 16Gb of ram so on the server side it should be
>> okay memory wise. on my monitoring i never go under 5gb free (but i only
>> have  the data every 5 minutes).
>>
> Aside from excluding unneeded parts, one other approach that can
> sometimes help is to split the target into separate runs for different
> subdirectories.   To do that, you can make different 'host' entries,
> then use the ClientNameAlias setting to point them back to the same
> real target.  If the files in question are split into some reasonable
> upper-level subdirectories, you may be able to get sets that complete
> in a reasonable amount of time.
>

------------------------------------------------------------------------------
Learn Windows Azure Live!  Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for 
developers. It will provide a great way to learn Windows Azure and what it 
provides. You can attend the event by watching it streamed LIVE online.  
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/