Don't know exactly what your AX150 has in it, but I can see from the
references to FLARE and Navisphere that it's a clariion rebadge...
Our Clariions have two Service processors - to get the best
performance we figured out that each AFTD should be a filesystem
striped between 2 LUNs, with each LUN managed by a different SP, as
these are the bottlenecks for writing to RAID3 or RAID5.
On the solaris 9 box I use a 64mb wide stripe with SVM between two 5
disk RAID3 LUNs for each filesystem - same disks as you have. When doing
a full backup run I have 6 filesystems each with one AFTD on them, the
target parallelism of the AFTD's is 1 to spread the load - the result is
around 100MB/sec, which we live with but certainly doesn't bear much
resemblance to the numbers in the brochure. We avoid reading at the same
time by managing staging manually, but the striping helps when we
absolutely have to read at the same time.
The LUNs we use are RAID-3 - that's what EMC said to do in some old
whitepaper, so that's what we all do. IMHO The RAID3 recommendation
comes straight out of university theory, and takes absolutely no account
of how ufs turns sequential traffic into random traffic. Our tests with
ZFS has been very encouraging in that they indicate a much more
'sequential' type of writing.
We are setting up a new system, using 500GB disks - we will try
combinations of RAID3 RAID5 RAID10 and maybe even raw +zfs RAIDZ,
although not real sure if that's a good idea yet.. With zfs managing the
pool of LUNs we can forget about trading off number of filesystems
(driven by how many tape drives you have to stage to) against LUNs, but
there seems to be a problem in that we can't shape the pool to ensure
load is spread evenly between SP's, that's why I'm considering going to
RAW or RAID1/10 because these minimise the load on the SP's.
Happy to share any thoughts and opinions on this - other than people
telling me about virtual tape and how good it is.
-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Adrian Saul
Sent: Monday, 12 March 2007 10:19 PM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: Re: [Networker] AX150 performance problems.
We have similar staging performance issues with a Clariion from an
inherrited config which hit brick walls when its load was increased from
the initial light setup.
In our case the issue comes down to a number of factors:
1. LUN type is RAID-3 - more on this later
2. The disks are 320G 5400 PATA disks - no idea why other cost
3. Multiple AFTD devices on the same LUNs.
The issue is that RAID-3 is a recommended setup for streaming sequential
transfers (like staging) *PROVIDED* you are reading a single stream.
Because we have multiple AFTD's per LUN, we were doing multiple reads
from the LUNs (i.e effectively scattered read) which killed the RAID-3
performance. This is because RAID-3 synchronises all the disk heads and
needs to read from all disks to satisfy a read. Because each read has
to wait for all 5 disks in the RAID to reseek, it was killing the
performance as the disks rattled around for each new read, and being
PATA didnt help.
Changing to 2 AFTD's instead of 4 picked performance up substantially.
We are trying to change to a single AFTD per LUN with some better disks
- work in progress there though.
Co-incidently I am looking at sequential I/O issues on a Sun Fire 6900
which comes down to UFS - benchmarked UFS at 20Mb/s, ZFS at 210Mb/s on
the same device (DMX-3 LUN). Yes - 10 fold improvement and it wasnt
filesystem caching.
For UFS you want to tune your filesystems to overcome some UFS caused
bottlenecks. Set maxphys and md:md_maxphys to 1Mb (0x100000), and use
tunefs to change maxcontig to 128. You should see much better
sequential I/O performance from that.
See how that helps out.
Cheers,
Adrian
Yaron Zabary wrote:
> Hello all,
>
> Recently, we purchased a Dell/EMC AX150SC storage for the disk
backup
> option. We have some performance problems with it which are described
in
> the letter below (which I sent to my Dell/EMC support). Does anyone
have
> a similar setup and can tell if they see similar problems or were able
> to get decent performance from this machine ?
>
>
>
> -------- Original Message --------
> Subject: AX150 performance problems.
> Date: Mon, 05 Mar 2007 12:24:05 +0200
> From: Yaron Zabary <yaron AT aristo.tau.ac DOT il>
> To: michaelz AT bynet.co DOT il
>
> Hello,
>
> I have performance problems with the AX150SC we have. These are the
> details:
>
> . The AX150 is connected to a single Sun 280R server via an Emulex
HBA.
>
> . The Server is a Sun 280R with two 750Mhz CPUs and 2Gb RAM running
> Solaris 9 (SunOS legato.tau.ac.il 5.9 Generic_118558-39 sun4u sparc
> SUNW,Sun-Fire-280R). I applied all patches required by EMC for Solaris
> 9. The server also has four LTO2 drives connected to another dual-LVD
HBA.
>
> . The HBA is Emulex 9802 running the latest firmware (1.91a5) with
> driver 6.10g. The HBA shares the 33Mhz PCI bus with a GigE card. The
LVD
> HBA has its own 66Mhz PCI bus.
>
> . The AX150SC has 12 500Gb disks. Its Flare was upgraded to
> 02.20.150.5.022. All disks are in a single pool with a single hot
spare.
> Currently, there are two 2Tb LUNs defined (due to Solaris 2Tb
> limitations). Only the first LUN is in use. Event log is empty. All
> components are green on the view/component screen.
>
> . /etc/system has the following lines (as required by EMC):
>
> set sd:sd_max_throttle=20
> set sd:sd_io_time=0x3c
> set ssd:ssd_max_throttle=20
> set ssd:ssd_io_time=0x3c
>
> . The backup software we use is EMC Networker version 7.2.2 build
> 422. Networker uses the first LUN for disk bacckup (using advfile
> device). The backups are then staged (copied) to the LTO2 drives.
>
> . The problem starts when the server tries to stage savesets from
the
> disk while running backups. During such operations the read speed can
be
> as low as 500Kb/s. Write speed also drops from ~25Mb/s to ~10Mb/s.
While
> this happens 'sar -d' reports a huge number for average wait time for
> I/O requests (over 1.5 seconds per request). Also notice the huge
queue
> (over 300 requests) and the large service time (~100msec). The numbers
> below are 10 minutes averages:
>
> sd60 99 306.6 190 11077 1508.4 101.0
> sd60,a 99 306.6 190 11077 1508.4 101.0
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 350.0 187 10387 1769.8 106.1
> sd60,a 100 350.0 187 10387 1769.8 106.1
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 348.3 194 10767 1690.5 102.0
> sd60,a 100 348.3 194 10767 1690.5 102.0
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 311.7 199 13024 1470.7 99.0
> sd60,a 100 311.7 199 13024 1470.7 99.0
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 364.3 209 10714 1644.8 94.9
> sd60,a 100 364.3 209 10714 1644.8 94.9
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 375.6 193 10093 1840.4 103.0
> sd60,a 100 375.6 193 10093 1840.4 103.0
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 346.1 197 11119 1653.3 100.4
> sd60,a 100 346.1 197 11119 1653.3 100.4
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 341.2 197 11666 1628.1 100.5
> sd60,a 100 341.2 197 11666 1628.1 100.5
> sd60,h 0 0.0 0 0 0.0 0.0
> sd60 100 339.6 196 11948 1629.5 100.8
> sd60,a 100 339.6 196 11948 1629.5 100.8
> sd60,h 0 0.0 0 0 0.0 0.0
>
>
> Please advice.
>
> --
>
> -- Yaron.
>
>
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
NOTICE
This e-mail and any attachments are confidential and may contain copyright
material of Macquarie Bank or third parties. If you are not the intended
recipient of this email you should not read, print, re-transmit, store or act
in reliance on this e-mail or any attachments, and should destroy all copies of
them. Macquarie Bank does not guarantee the integrity of any emails or any
attached files. The views or opinions expressed are the author's own and may
not reflect the views or opinions of Macquarie Bank.
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
