On Apr 18, 2008, at 11:32 AM, Tony Schreiner wrote:
> And yes, I struggle with what needs be backed up. The users
> (bioinformatics research) can generate a couple of 100 GB of data
> every day, some of it very large files, some of it hectathousands  of
> small files, some of which needs to be saved, some of which does not.
> There is no easy way for me to predict yes or no.

How much memory does the server have?  Just a guess, but especially if  
you are using rsync, you could be running into problems with the size  
of the "list" that rsync has to maintain in memory, and rsync could  
spend all of its time paging.  "top" for e.g. should give you some  
idea.  I would recommend using 64-bit OS and max out the memory if  
possible, that should help unless something else is going on.  Also  
the BackupPC server(s) should be dedicated for that purpose and not  
running other jobs (my prior experience with bioinformatics is they  
like to squeeze CPU cycles from any box they can get their hands on).

Second, take a look at aggregation and backplane speed on the network  
switches, maybe links are getting saturated somewhere, or you need to  
do some trunking, make sure your link speeds are really Gb/s from end  
to end, try some large file transfers to see how much throughput you  
really get from point to point.  Take a look at the network interface  
stats e.g. from ifconfig to see if any of the numbers look awry,  
double check duplex settings and for MTU consistency, any routers or  
firewalls in the path.  I am sure you already know all that, but once  
in awhile I know myself I forget to check the things that are quick  
and easy to check and would be easy to fix, very little time wasted if  
that turns out not to be relevant.

If possible, you may want to have a parallel "storage network"  
connected to extra NIC card in each server to get maximum bandwidth  
for backups--if you make that secure and separate from the regular  
LAN, then you don't need to e.g. tunnel through SSH, that will save  
you some overhead if you are encrypting right now.  I take it for  
granted that you are using some type of hardware RAID or a SAN to get  
better I/O throughput on the server, you would pretty much have to be  
doing that for your capacity.

Jonathan

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/