Just so it's in the archives... If anyone is planning on using some sort
of software RAID5 solution for TSM disk storagepool volumes, a few
things
to note. We're managing to get up to about 40 MiB/s to/from fibre
attached
disk using Veritas Volume Manager RAID5. Here's how.

TSM does disk storagepool I/Os in blocks of 256 KiB. Optimally, a write
I/O
should represent a full-stripe write. If it is not a full stripe, the
RAID software must do a read-modify-write cycle which can reduce
bandwidth
by an order of magnitude, or more. This doesn't apply to hardware RAID
that
has either battery backed caches or NVRAM, since they can delay and
combine
many small I/Os into a few larger I/Os. I'd imagine it would apply to
hardware RAID solutions without cache, or with write-through cache.

To force full stripe writes, use raw devices for volumes. This also
bypasses
the OS file-cache which can be a big win (especially with older
versions of
Solaris). Next is to make the size of a stripe 256 KiB. This requires
that
the number of disks in the RAID5 set be one more than a power of two.
i.e.
3, 5, 9, 17. There are other combinations that work, I'll leave that to
your
imagination. So, hopefully a write will hit all disks and not involve
any
unnecessary reads. In our case, we have a A5x00 series array with 14
disks,
which are broken up into 5 disk RAID5 and 9 disk RAID5, and I've seen
the
disk pool peak at about 40 MiB/s writes, especially on the 9 disk set.

Cheers,
--
Paul Ripke
Unix/OpenVMS/TSM/DBA
101 reasons why you can't find your Sysadmin:
68: It's 9AM. He/She is not working that late.
-- Koos van den Hout