> nsrmmd on the storage node is the obvious candidate, as it's  
> consuming ~40% of a processor and making ~5K system calls per  
> second.   Is it possible to raise the overall blocking factor 
> to make  
> more efficient use of the CPU?

You should be able to get more performance out of the drive - 50MB/sec
suggests your drive is not running at the fastest spindle speed given
that you are sending uncompressed data to it - maybe the drive is
running at 30MB/sec with a moderate amount of compression (figure out
the compression by seeing how much data fits on the tape, 400GB is 1:1
compression)

Networker is probably already choosing the best blocking factor for LTO3
- I wouldn't go changing that in a hurry.

I Think there's a bottleneck either at the stage of pulling the data off
the filesystem or because of the network.

Try increasing window size on the tcp connections - 30 miles of latency
and wan gear can have an impact on ACK's getting back, unfortunately I
know of no way to influence how nsrmmd's choose their block sizes, so
fiddling with tcp is all that can be done.

As for the filesystems...   I imagine this has to consume some cpu to
decompress the zfs data, maybe you'd get some increase in performance
from faster cpu clocks. Also you don't say if the 4 streams come from
the one zfs pool - have you tried doing just 1 stream, it may be that
the 4 streams are competing for the same disks?

NOTICE
This e-mail and any attachments are confidential and may contain copyright 
material of Macquarie Group Limited or third parties. If you are not the 
intended recipient of this email you should not read, print, re-transmit, store 
or act in reliance on this e-mail or any attachments, and should destroy all 
copies of them. Macquarie Group Limited does not guarantee the integrity of any 
emails or any attached files. The views or opinions expressed are the author's 
own and may not reflect the views or opinions of Macquarie Group Limited.

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER