On Fri, 24 Jun 2005, Robert Maiello wrote:
RM> Yes, it seems the Cassini interface can do 900+ Mbits/sec but that Sun
RM> recommends 3to4 UltraSparc III's per NIC. We can't use Jumbo frames either.
OK, do you know of any recommendations from Sun for the bge
(Broadcom) interfaces? It would be interesting to know how they compare in
regard to CPU useage for the same type of network I/O.
RM> I heard Solaris 9 and, of course Solaris 10 make some strides in Network
RM> performance. Still, I'd love to hear if anyone is maxing out 2 or more
RM> and what Sun hardware they use to do it?? Do the UltraSparc IV's help
RM> in any way here?? ie...2 ce worker threads per CPU ?? Of course PCI
RM> bus bandwidth becomes an issue as well...
We're running solaris 9, with 4 UltraSPARC IIIi CPUs with 1MB cache, at
1281MHz, on a Sun V440. PCI bandwidth shouldn't be a problem, since
they're on two different buses. There is just no way we can max out both
NICs on this hardware, at least not without using jumbo frames, and I
doubt that will add more than 10-20% extra performance. We rather need to
double it. :)
Anyway, yesterday I decided to see if I can't tune kernel and driver
parameters in order to increase performance. I used a few documents I
found on google, and compared them (they tend to conflict somewhat), and
then I came up with a few settings that seem better than the previous
defaults. I'm pretty sure they're not optimal, but I have seen a 10%
performance increase on our NetWorker server. The performance gained is
10% higher backup throughput. This is clearly visible on the MRTG graph,
that is drawn by using load data from the port-channel interface on the
switch (i.e. both ce0 and ce1 aggregated, since we're running Sun Trunking
1.3).
I was thinking of applying "set ce:ce_taskq_disable = 1", but some
documents suggested that this might be a bad idea. Does anyone know why or
if it should be enabled/disabled and under what circumstances?
This is what I applied (no warranty that this will work for you. Who
knows, it might even mess up your system instead? :) ):
---
bash-2.05# pwd
/etc/rc2.d
bash-2.05# more S94local
#!/sbin/sh
# Local commands run before networker
/usr/sbin/ndd -set /dev/tcp tcp_max_buf 4194304
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 196608
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 65536
/usr/sbin/ndd -set /dev/tcp tcp_maxpsz_multiplier 10
/usr/sbin/ndd -set /dev/ce instance 0
/usr/sbin/ndd -set /dev/ce rx_intr_pkts 48
/usr/sbin/ndd -set /dev/ce rx_intr_time 30
/usr/sbin/ndd -set /dev/ce instance 1
/usr/sbin/ndd -set /dev/ce rx_intr_pkts 48
/usr/sbin/ndd -set /dev/ce rx_intr_time 30
--
In /etc/system:
set maxusers=2048
set maxphys=2097152
set sq_max_size=1600
set rlim_fd_max=8192
set rlim_fd_cur=8192
set pt_cnt=1024
set rlim_fd_max=8192
set rlim_fd_cur=8192
set tcp:tcp_conn_hash_size=32768
set ce:ce_bcopy_thresh=512
set ce:ce_dvma_thresh=512
set ce:ce_taskq_disable=1
set ce:ce_ring_size=512
set ce:ce_comp_ring_size=2048
set ce:ce_tx_ring_size=4096
[NOTE: Although the number of open files doesn't have anything to do with
performance, I tend to change it on my systems anyway]
---
bash-2.05# more /kernel/drv/ce.conf
interrupts=1;
---
//Oscar
--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listserv.temple DOT edu or visit the list's Web site at
http://listserv.temple.edu/archives/networker.html where you can
also view and post messages to the list. Questions regarding this list
should be sent to stan AT temple DOT edu
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
|
