At 11:39 AM -0500 11/2/00, Richard, Alain wrote:
What do you mean about "untuned gigabit interface" what kind of tuning you 
talking ????

>Rob Worman <rob AT colltech DOT com>
>Turn on 6 of those drives and feed them data from your untuned gigabit 
>interface, and you'll probably be unhappy with what you get...

>Rob

Alain.Richard AT menv.gouv.qc DOT ca
Systems Admin


When implementing GBE under Solaris 2.6 several months ago, we only saw about 
300 Mbit performance between gigabit hosts (even without a switch, i.e. direct 
connected). 

We were given the following recommendations from SUN:

=====
1. use TTCP or netperf, etc to test performance.

http://www.caida.org/tools/taxonomy/performance.xml  (tool overview)
ftp://ftp.fore.com/pub/utils/  (we got nttcp here)

    do not use FTP, or spray <ouch>.
    Higher throughput seen with larger message/buffer sizes.

2. running multiple concurrent TTCP streams may improve results.

3. install GEM GbE v2.0 patch (ge-driver performance patch).
   106745-xx, 106764-xx, or 106765-xx.  Get latest for OS.

4. install GEM PCI cards in 64-bit/66mhz slots!
5. use 2, 3, or 4 cpus (faster clockspeeds better, >300 mhz each).
6. test on Sol-2.6 or better.  (2.6 HW5/98 MU)
7. Check switch firmware level & GbE module with switch vendor(s).
8. do NOT tune ndd param tcp_cwnd_max.  Sol-2.6 default is 256k.
9. Might set ndd params  tcp_xmit_hiwat & tcp_revc_hiwat = 64k.

    For example, for testing purposes on high-bandwidth cards.
    Be aware that this will affect all network interfaces.
       ndd -set /dev/tcp tcp_xmit_hiwat = 65536
       ndd -set /dev/tcp tcp_recv_hiwat = 65536
       ndd -set /dev/udp udp_xmit_hiwat = 65536
       ndd -set /dev/udp udp_xmit_hiwat = 65536

 10. Might want to ensure you are not routing or ip_forwarding
     packets in Solaris kernel.  It could hurt performance
     measurements as kernel forwards between multiple adapters.
     If you don't disable this, Solaris default is enabled!

     touch /etc/notrouter
     ndd -set /dev/ip ip_forwarding 0

 11. Is the STP spanning-tree protocol disabled on switch port(s)?

     You most likely want STP disabled on Sun host-connected ports.
     If STP detects a state-change, it may freeze traffic and affect
     performance and test results...  and it may affect other switches.

     STP is primary used for connecting redundant switch configurations
     and has little use on end-nodes.  The only Sun NIC adapter that
     can utilize STP is Gigabit v1.1 with redundant NIC configuration
    Several switch vendors have updates/fixes for old GbE modules.

    Some folks run into basic solaris problems.
    Fastest tests with say 4 x 336mhz  OR  4 x 300 mhz cpus.
    Faster clockspeeds often result in faster network results.

    Gigabit cannot increase cpu clockspeeds,
    Gigabit under load may require multiprocessor configs,
    Gigabit does not make ftp ascii/binary conversion go faster.
    Gigabit cannot make disk drive RPMs rotate faster.
    Gigabit cannot make small packet sizes become larger.
    Gigabit cannot make a single socket or stream fatter.
    Gigabit cannot make poor tests run faster.
===========

In our case (running on a 4 CPU E450 with 2GB RAM), it was the tweaking of the 
tcp_xmit_hiwat and tcp_recv_hiwat values that seemed to make a difference.  We 
were then seeing more like 700Mbit performance.

Hope this Helps,
rob