On Jan 21, 2009, at 4:35 PM, Fred Johanson wrote:
We've got two machines which are identical in every way, same
hardware, same AIX, same TSM level, same options set, same storage
pools, domains, everything except one has the DB and Log on local
disk. On this box things run very slowly: expiration may take a
week, filespace deletion creeps, and we see this as the normal
behavior
Fred -
In my experience, it's seldom the case that two machines of the same
model are identical, even if ordered at the same time. Vendors are
fond of outfitting computers with "equivalent" components, from
multiple suppliers, during manufacturing. It's common to see same
type, but different OEMs for memory, cards, and disks. The cartons
containing the computers may well have come from different
manufacturing periods or different assembly plants. There can easily
be variations in configuration methods for differing components; and
sometimes there are factory errors in placing jumpers on drives and
cards. Where the OS perceives hardware elements as different,
differing device driver software may be called upon, and that software
may have differing characteristics as it evolves.
Your staff will have to perform all the usual configuration and
performance reviews involved in chasing a problem like this. Whereas
this is AIX, the first thing I would look at is the Error Log for any
irregularities, and 'lscfg -v' for detailed comparison of the two
boxes. You are fortunate to have the advantage of a comparable system
with good performance as a basic for pursuit. Given that this is
local disk, also comparatively check attributes, as via 'lsattr -El
hdisk4', and particularly check the queue_depth value. For disks
which IBM's OS recognizes as being programmed for, a Queue Depth of 3
or more will be used and you will get good performance; for disks that
the OS does not recognize, it will minimize Queue Depth to 1, and
performance will be poor. (See discussions of Queue Depth on the Web,
to perceive impact.)
Your computer implementation people may not have benchmarked or
otherwise performance-checked the systems when they came in, where one
would perform disk and memory stress testing as part of acceptance
verification before committing the boxes to production. If not,
consider making small logical volumes on the disks to now perform such
disk performance tests. And definitely assure that all installed
memory and processors are online: I've seen such elements rather
quietly fail and go offline, resulting in impaired performance whose
cause it not apparent.
One thing that I do when performance is hurting on AIX systems is an
iptrace/ipreport run, to see what network traffic is hitting the box.
A DoS attack, or unintentional equivalent, is identifiable only by
such an examination. Inordinate network activity can greatly elevate
the interrupt rate on the system (particularly with GigE), which will
congest the system bus. Look for and put a stop to any network access
which should not be happening.
Beyond all that, you need to perform stepwise examination of the
system elements upon which TSM, as an application, runs, including
things like I/O path contention from other system activity.
Thereafter, examination of TSM elements would be warranted.
Richard Sims http://people.bu.edu/rbs/
|
