ADSM-L

Slow/serial database reads?

2006-10-16 23:42:35
Subject: Slow/serial database reads?
From: Jason Lee <english AT ANIM.DREAMWORKS DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Mon, 16 Oct 2006 20:40:40 -0700
HI there,

I've been banging my head against the wall for a little while on
this.....

I'm running TSM 5.3.4 on RedHat EL4 U4 for x86_64 (HP AMD blade server).

My databases are on an HP MSA1500cs on a shelf of 14 72GB 15K drives.
This is fibre attached.

Now, I've been watching my I/O patterns via iostat -d -x, and it
looks for all the world like it's doing serial access to the
database. I'm maxing out at about 250 (4KB) reads per second, which
is lame. I have 60 dbvs residing on 16 LUNs. They all used to live on
a single LUN but I broke that up last night to see if the thread
locking was LUN based, rather than volume (as I had originally
thought). But apparently not. My DB is ~500MB, though there is a lot
of cruft in there, since expirations take so long.

Is this a normal access pattern?

Here's an iostat, just for fun (may I recommend a mono-space font :-)  )

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/
s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.20  42.00 25.40  6.20  321.60  385.60   160.80
192.80    22.38     0.02    0.52   0.18   0.56
sda          0.00   1.20  8.80  6.00   70.40   57.60    35.20
28.80     8.65     0.11    7.69   6.49   9.60
sdb          0.00   0.20 56.40  2.60  451.20   22.40   225.60
11.20     8.03     0.40    6.84   6.63  39.14
sdc          0.00   0.20 25.60  2.60  208.00   22.40   104.00
11.20     8.17     0.14    5.13   5.01  14.14
sdd          0.00   0.20 10.40  4.20   83.20   35.20    41.60
17.60     8.11     0.10    6.60   4.95   7.22
sde          0.00   0.20  3.80  1.20   30.40   11.20    15.20
5.60     8.32     0.03    5.36   5.24   2.62
sdf          0.00   0.20  6.00  1.80   49.60   16.00    24.80
8.00     8.41     0.07    9.26   8.74   6.82
sdg          0.00   0.40 23.00  1.00  196.80   11.20    98.40
5.60     8.67     0.17    6.97   6.88  16.52
sdh          0.00   0.20 16.80  2.80  134.40   24.00    67.20
12.00     8.08     0.12    6.26   5.99  11.74
sdi          0.00   0.20 160.40 1.80 1283.20   16.00   641.60
8.00     8.01     0.87    5.34   5.26  85.32
sdj          0.00   0.20 10.60  1.20   84.80   11.20    42.40
5.60     8.14     0.05    4.63   4.58   5.40
sdk          0.00   0.20 10.40  1.20   84.80   11.20    42.40
5.60     8.28     0.08    6.76   6.67   7.74
sdl          0.00   0.20  6.40  1.00   51.20    9.60    25.60
4.80     8.22     0.13   17.78   9.95   7.36
sdm          0.00   0.40  0.80  1.00    6.40   11.20     3.20
5.60     9.78     0.00    2.44   2.22   0.40
sdn          0.00   0.40  0.60  1.60    4.80   16.00     2.40
8.00     9.45     0.00    2.18   1.73   0.38
sdo          0.00   0.40 13.60 15.00  108.80  123.20    54.40
61.60     8.11     0.18    6.43   3.38   9.66
sdp          0.00   0.60  7.80  8.60   62.40   73.60    31.20
36.80     8.29     0.10    6.23   3.71   6.08
sdq          0.00   0.20  0.00  2.20    0.00   19.20     0.00
9.60     8.73     0.00    1.18   0.27   0.06
sdr          9.00 3309.20 82.80 73.20 39580.80 27059.20 19790.40
13529.60   427.18    15.27   97.89   6.08  94.82
sds          9.20   0.20 83.40  0.40 41272.00    4.80 20636.00
2.40   492.56     0.94   11.24  11.13  93.24
sdt          0.00  12.60  0.00  3.00    0.00  124.80     0.00
62.40    41.60     0.00    1.53   0.33   0.10

The last three LUNs are my disk pools and recovery log (all on
different RAIDs). The sum of average queue length is alarmingly close
to one, or put another way, alarmingly far from a big number. TCQ is
set to 128 or so. The RAID supports 255 in the queue. The fiber card
is a Qlogic 2312 and the driver is the standard RedHat 8.0.something
(as required by IBMtape).

Anyone have any thoughts, or an example iostat from a linux system
where there *are* concurrent accesses going on? BTW the system is
running 20ish client sessions and an expiration. Buffer pool is 2GB.


Thanks for any pointers.


Jason


--
Jason Lee
DreamWorks Animation

<Prev in Thread] Current Thread [Next in Thread>