>From AIX support:

If you have many hosts on a single subnet who will communicate
with you simultaneously (>100), the arp tables are going to need
to be tuned larger than the shipped defaults.

The arp tables on AIX are fixed at the time the kernel loads
the netinet kernel extension, so they are not tunable on
the fly. You will have to change the settings in /etc/rc.net
and do so at the very TOP of the file (just below the #/!/bin/ksh)

#!/bin/ksh
/usr/sbin/no -o arptab_bsiz=35
/usr/sbin/no -o arptab_nb=125
...rest of rc.net....
The above was found to be adequate to maintain a population
of 200 continuously connected machines on the local subnet.

You may install APAR IY05739, but what that APAR really does,
is kick an existing (non-expired) entry out of the arp table
to put a new one in. -Although this will give the outward
appearance of having "fixed" the problem, (and it is good to
have this APAR in place, just in case you have an occasional
rise in the number of concurrent connections).
*BUT*
If you live on a large, flat network: then simply installing
this APAR can hide the real problem:
(Your arp table is undersized for the number of concurrent
local-subnet connections to your machine.)
Use of the APAR is going to allow AIX to remove an entry from the
arp table which has not yet expired via the normal timeout mechanism,
and replace it with another hosts' entry, so you
can send the pending IP packet to the "new" machine.

If the APAR is put in place, and no additional consideration is
given to the real problem, -then on a large "flat" network where
hundreds of machines communicate using no routers (all using arp):
a situation can arise where you are "thrashing" the arp table
and consuming the network bandwidth with arp requests.

I have seen it so bad on some networks, that you acutally
see an arp request go out for every IP packet that needed to
be sent. (Effectively tripling the number of packets on the wire at
any given point in time, and reducing the available bandwidth).
(For every outbound IP packet that you wind up removing one arp entry,
to replace it with another, when it really wasn't time
to kick the 1st host out just yet because you still have traffic
pending for him, then you have caused another arp request to
go out on the wire, and another reply to come backi BEFORE the
current IP packet can be sent).

In the process, when arp gets handed an outbound packet,
if he already has the entry in his table - no problem he stamps
the appropriate MAC address on it, and sends it on it's way.
But if it's somebody you just recently removed, (he does not
have the entry in the arp table) then arp must queue up that packet
and must send out an arp requesti packet, and wait for the arp reply
packet, so it will have this new entry to send the current packet with.
As one can see, it's bad on the AIX machine too - dealing with three
packets has 3 times the overhead that the original one does.

The APAR is a good thing to have if someone sticks a lot of new
machines onto your local subnet while you are away, and you
don't want your pager to go off when they do it, but if you
think you really need this APAR, 1st examine the size of the
local subnet, then resize the arp tables first,
then put the APAR on.

--Good Luck
--Robert Manning,
IBM AIX
Austin, Tx.