Re: [Networker] AX150 performance problems.
2007-03-12 07:36:50
We have similar staging performance issues with a Clariion from an
inherrited config which hit brick walls when its load was increased from
the initial light setup.
In our case the issue comes down to a number of factors:
1. LUN type is RAID-3 - more on this later
2. The disks are 320G 5400 PATA disks - no idea why other cost
3. Multiple AFTD devices on the same LUNs.
The issue is that RAID-3 is a recommended setup for streaming sequential
transfers (like staging) *PROVIDED* you are reading a single stream.
Because we have multiple AFTD's per LUN, we were doing multiple reads
from the LUNs (i.e effectively scattered read) which killed the RAID-3
performance. This is because RAID-3 synchronises all the disk heads and
needs to read from all disks to satisfy a read. Because each read has
to wait for all 5 disks in the RAID to reseek, it was killing the
performance as the disks rattled around for each new read, and being
PATA didnt help.
Changing to 2 AFTD's instead of 4 picked performance up substantially.
We are trying to change to a single AFTD per LUN with some better disks
- work in progress there though.
Co-incidently I am looking at sequential I/O issues on a Sun Fire 6900
which comes down to UFS - benchmarked UFS at 20Mb/s, ZFS at 210Mb/s on
the same device (DMX-3 LUN). Yes - 10 fold improvement and it wasnt
filesystem caching.
For UFS you want to tune your filesystems to overcome some UFS caused
bottlenecks. Set maxphys and md:md_maxphys to 1Mb (0x100000), and use
tunefs to change maxcontig to 128. You should see much better
sequential I/O performance from that.
See how that helps out.
Cheers,
Adrian
Yaron Zabary wrote:
Hello all,
Recently, we purchased a Dell/EMC AX150SC storage for the disk backup
option. We have some performance problems with it which are described in
the letter below (which I sent to my Dell/EMC support). Does anyone have
a similar setup and can tell if they see similar problems or were able
to get decent performance from this machine ?
-------- Original Message --------
Subject: AX150 performance problems.
Date: Mon, 05 Mar 2007 12:24:05 +0200
From: Yaron Zabary <yaron AT aristo.tau.ac DOT il>
To: michaelz AT bynet.co DOT il
Hello,
I have performance problems with the AX150SC we have. These are the
details:
. The AX150 is connected to a single Sun 280R server via an Emulex HBA.
. The Server is a Sun 280R with two 750Mhz CPUs and 2Gb RAM running
Solaris 9 (SunOS legato.tau.ac.il 5.9 Generic_118558-39 sun4u sparc
SUNW,Sun-Fire-280R). I applied all patches required by EMC for Solaris
9. The server also has four LTO2 drives connected to another dual-LVD HBA.
. The HBA is Emulex 9802 running the latest firmware (1.91a5) with
driver 6.10g. The HBA shares the 33Mhz PCI bus with a GigE card. The LVD
HBA has its own 66Mhz PCI bus.
. The AX150SC has 12 500Gb disks. Its Flare was upgraded to
02.20.150.5.022. All disks are in a single pool with a single hot spare.
Currently, there are two 2Tb LUNs defined (due to Solaris 2Tb
limitations). Only the first LUN is in use. Event log is empty. All
components are green on the view/component screen.
. /etc/system has the following lines (as required by EMC):
set sd:sd_max_throttle=20
set sd:sd_io_time=0x3c
set ssd:ssd_max_throttle=20
set ssd:ssd_io_time=0x3c
. The backup software we use is EMC Networker version 7.2.2 build
422. Networker uses the first LUN for disk bacckup (using advfile
device). The backups are then staged (copied) to the LTO2 drives.
. The problem starts when the server tries to stage savesets from the
disk while running backups. During such operations the read speed can be
as low as 500Kb/s. Write speed also drops from ~25Mb/s to ~10Mb/s. While
this happens 'sar -d' reports a huge number for average wait time for
I/O requests (over 1.5 seconds per request). Also notice the huge queue
(over 300 requests) and the large service time (~100msec). The numbers
below are 10 minutes averages:
sd60 99 306.6 190 11077 1508.4 101.0
sd60,a 99 306.6 190 11077 1508.4 101.0
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 350.0 187 10387 1769.8 106.1
sd60,a 100 350.0 187 10387 1769.8 106.1
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 348.3 194 10767 1690.5 102.0
sd60,a 100 348.3 194 10767 1690.5 102.0
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 311.7 199 13024 1470.7 99.0
sd60,a 100 311.7 199 13024 1470.7 99.0
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 364.3 209 10714 1644.8 94.9
sd60,a 100 364.3 209 10714 1644.8 94.9
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 375.6 193 10093 1840.4 103.0
sd60,a 100 375.6 193 10093 1840.4 103.0
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 346.1 197 11119 1653.3 100.4
sd60,a 100 346.1 197 11119 1653.3 100.4
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 341.2 197 11666 1628.1 100.5
sd60,a 100 341.2 197 11666 1628.1 100.5
sd60,h 0 0.0 0 0 0.0 0.0
sd60 100 339.6 196 11948 1629.5 100.8
sd60,a 100 339.6 196 11948 1629.5 100.8
sd60,h 0 0.0 0 0 0.0 0.0
Please advice.
--
-- Yaron.
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|