> I am using a PCIe megaraid SAS controller, and the drives are 7200 RPM SATA.
>
> I have just tried reconfiguring the drives as a hardware raid 5 array, and I
> still only get about 66 MB/sec throughput.  I have tried both ext3, and xfs,
> with various format and mount options.
>
If this is a true hardware raid array and not a biosraid (or fakeraid)
some controllers default to turn the wirte cache off especially if you
do not have the battery unit attached. This causes the same effect as
a software raid with too small of a stripe cache.

>
> Even using md raid 5, with exactly the same xfs mount options as you, and
> the same stripe sizes only gives me about 60 MB/sec
>
> I am at a loss as to what you are doing differently to me....
>
> As soon as I switch to RAID 0, I get over 300 MB/sec throughput.

have you got any hdparm numbers for the individual disks?

hdparm -tT /dev/sdb

Mine get 105MB/s for the 7200.11 drives and 85 MB/s for the 7200.10

Here are some more results of stripe_cache_size. Perhaps increasing
the 1024 to 2048 or 4096 may help.

This time with a slower single core 2.0Ghz Athlon 64 3200, with the
same ASUS M2N motherboard and 2GB of memory instead of 8 (I repeated
the test on the other machine with 16 GB file and got 186MB/s).

# free -m
             total       used       free     shared    buffers     cached
Mem:          2010       1992         17          0         53       1647
-/+ buffers/cache:        291       1718
Swap:         4424          0       4423

datastore0 ~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 79
model name      : AMD Athlon(tm) 64 Processor 3200+
stepping        : 2
cpu MHz         : 2000.000
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
fxsr_opt rdtscp lm 3dnowext 3dnow up pni cx16 lahf_lm svm cr8_legacy
bogomips        : 4021.83
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc


Here I am using software raid 6 with 6 X 320 GB Segate 7200.10 drives.

datastore0 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [multipath]
md0 : active raid1 sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      256896 blocks [6/6] [UUUUUU]

md2 : active raid6 sdf4[5] sde4[4] sdd4[3] sdc4[2] sdb4[1] sda4[0]
      1199283200 blocks level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid6 sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
      46909440 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

unused devices: <none>

The first test starts with the system default stripe cache:

datastore0 ~ # dd if=/dev/zero of=/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 141.612 s, 60.7 MB/s

datastore0 ~ # echo 1024 > /sys/block/md1/md/stripe_cache_siz
datastore0 ~ # dd if=/dev/zero of=/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 112.452 s, 76.4 MB/s

datastore0 ~ # echo 2048 > /sys/block/md1/md/stripe_cache_size
datastore0 ~ # dd if=/dev/zero of=/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 65.093 s, 132 MB/s

For reproducibility I run again with 1024

datastore0 ~ # echo 1024 > /sys/block/md1/md/stripe_cache_size
datastore0 ~ # dd if=/dev/zero of=/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 109.951 s, 78.1 MB/s

Now 4096

datastore0 ~ # echo 4096 > /sys/block/md1/md/stripe_cache_size
datastore0 ~ # dd if=/dev/zero of=/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 59.2806 s, 145 MB/s


John

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users