We had a similar, *HELLISH* connectivity problem at my site... it was  a
massive pain in the butt
to pinpoint, much less diagnose.

The symptoms were many concurrent SQLbacktrack connexions that would
initiate, and die, within one second, and then re-initiate, and re-die.
This would go on for hours until the backup window exceeded.

Turns out it was an Mbuf overflow on our ADSM server's ATM card.  Aparrently
there wasn't enough system memory allocated to the card to buffer the
incoming packets quickly enough.    This was a pain to fix,
as the AIX R50 we use has architectural limitations upon how much memory can
be allocated to this purpose (a setting called THEWALL).

It was a nightmare... eventually we band-aided the problem by reducing the
tcpwindow size (server side) to 1024.  This slows down the rate of data
transfer, allowing the atm card to breath a bit easier and do it's job.

Things work hunky dory now (At least by ADSM standards).

Hope that helps,
Dan.