Some things to think about and look at - in no order:
- What kind of response time are your I/O's getting at the AIX level (ioscan)?
- Check the average sampled read/write times for your luns in the vmax. You
should be seeing 1ms writes and <10ms reads (fc tier and flash tier) and slow
+15ms reads for a sata tier. (going off the top of my head, but that sounds
like what I see on ours)
- I assume you have dual paths. Are you using PowerPath or MPIO. Pull up nmon
and see if your I/O's are evenly spread between your hba paths.
- Look at the queue depth for the hba and hdisks. We find the hba default
often to be too low for large I/O processing.
- Are your luns on FASTVP in the vmax? If you have a sata tier/pool,
expiration could be reading lots of little used blocks that have fallen to the
lowest tier. You would see very slow reads in this case for the expiration
reads.
- Are the front end ports on the vmax overloaded? We've seen major queuing
problems in the vmax FA's if iops exceed 20k on a port. QUite simply - host
hba's can over power a vmax fa port. (or multiple host hba's sharing a FA
port) I don't think this is it - expiration does a small continious data
stream, but if the fa port is overloaded you could have a problem.
- Watch the tsm cache hit ratio. I've seen our say it's 99% when I've looked
at it during the day. One night I was in late and saw that during backup
processing it was way down in the low 90%. Check it over time.
The single thing above for a quick check on if this is a disk performance issue
is the vmax sampled average read/write times for your luns.
Rick
-----"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote: -----
To: ADSM-L AT VM.MARIST DOT EDU
From: "Loon, EJ van - SPLXO"
Sent by: "ADSM: Dist Stor Manager"
Date: 02/16/2012 10:14AM
Subject: Re: Expiration performance TSM 5.5 (request)
Hi Daniel!
Been there, done that...
a) We completely redesigned our database layout. Each database file is located
on one single hdisk, one single vg and one single filesystem. We are using 10
database volumes and tried everything. LUN's are striped across multiple
arrays, the backend is using raid 1. Performance of the disks outside of TSM is
fine: 120 Mb/sec read as long as we do not use mirroring, write even better
(because of cache) around 180 Mb/sec. Even when we try to imitate TSM (by doing
4k reads using dd) read performance is fine.
b) Tried several setups for the log too, still no improvement.
c) I think you mean the way the vg is mirrored? All vg's are using parallel.
d) Tried that too, the only effect is that log utilization during the day is
much lower (of course).
e) Cache hit ration is 99.9% so the bufferpool should be large enough.
I have done extensive I/O analysis along with TSM support, there is no queuing,
0.0 most of the time...
Thanks for your reply!
Kind regards,
Eric van Loon
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Daniel Sparrman
Sent: donderdag 16 februari 2012 15:12
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Ang: Expiration performance TSM 5.5 (request)
Hi
To begin with, the guidelines for database setup is something similiar to:
a) 8-12 primary database volumes (since you're on 5.x you can still use TSM
mirroring). Each volume should be in it's own filesystem, preferably on their
own spindles (harddrives). If possible, make sure that your storage group
assigns you 8-12 volumes from different arrays within the Vmax if possible, or
at least as many arrays as possible. If they assign you 8 volumes from the same
array, performance will be horrible.
b) Log should reside on it's own volume(s). Since the log is sequential,
raid-10 is the optimal setup.
c) Using DB mirroring in parallell mode will increase performance
d) Using LOG mirroring in normal mode will aswell increase performance
e) Max sure you have enough bufferpool
From your description, it sounds like a) is the place to start, I wouldnt be
surprised if your db volumes are located (some of them or all of them) within
the same array.
What operating system are you using? If you're on AIX, try checking I/O
statistics during expiration to see if your queues are getting full (as in 100%
utilization of the disks using iostat). If so, try increasing the queues, and
go to your storage guys and have them look at the underlying Vmax to determine
if there are any configuration issues there.
Performance issues with expiration and database backups usually relates to the
fact that the read-performance of the underlying array is limited.
Best Regards
Daniel Sparrman
Daniel Sparrman
Exist i Stockholm AB
Växel: 08-754 98 00
Fax: 08-754 97 30
daniel.sparrman AT exist DOT se
http://www.existgruppen.se
Posthusgatan 1 761 30 NORRTÄLJE
-----"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> skrev: -----
Till: ADSM-L AT VM.MARIST DOT EDU
Från: "Loon, EJ van - SPLXO"
Sänt av: "ADSM: Dist Stor Manager"
Datum: 02/16/2012 15:02
Ärende: Expiration performance TSM 5.5 (request)
Hi TSM-ers!
I'm struggling with the performance of our expiration process. I can't
get it any faster than 100 object/second max. We tried everything, like
using more or less database volumes, multiple volumes per filesystem,
mirroring, unmirroring, but nothing seems to have any positive effect.
We are using SAN attached enterprise class storage (EMC Vmax) with the
fastest disks available.
I have seen other users with similar (or larger) databases with much
higher figures, like more than 1000 objects/sec, so there must be
something I can do to achieve this. In 2007 at the Oxford TSM Symposium
(http://tsm-symposium.oucs.ox.ac.uk/2007/papers/Dave%20Canan%20-%20Disk%
20Tuning%20and%20TSM.pdf page 25) IBM also stated that 1000 object/sec
is possible.
I would really like to know from other TSM 5.5 users how their
expiration is performing. Could you please let me know by sending me the
output from the following two SQL queries, along with the platform you
are using:
select activity, cast((end_time) as date) as "Date",
(examined/cast((end_time-start_time) seconds as decimal(18,13))*3600)
"Objects Examined/Hr" from summary where activity='EXPIRATION' and
days(end_time)-days(start_time)=0
select capacity_mb as "Capacity MB", pct_utilized as "Percentage in
use", cast(capacity_mb*pct_utilized/100 as integer) as "Used MB" from db
Thank you VERY much for your help in advance!!!!
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines
</pre>********************************************************<br>For
information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain confidential and
privileged material intended for the addressee only. If you are not the
addressee, you are notified that no part of the e-mail or any attachment may be
disclosed, copied or distributed, and that any other action related to this
e-mail or attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately by return
e-mail, and delete this message.<br><br>Koninklijke Luchtvaart Maatschappij NV
(KLM), its subsidiaries and/or its employees shall not be liable for the
incorrect or incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.<br>Koninklijke Luchtvaart Maatschappij
N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The
Netherlands, with registered number ÿ33014286
<br>********************************************************<pre>
********************************************************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain confidential and
privileged material intended for the addressee only. If you are not the
addressee, you are notified that no part of the e-mail or any attachment may be
disclosed, copied or distributed, and that any other action related to this
e-mail or attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately by return
e-mail, and delete this message.
Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its
employees shall not be liable for the incorrect or incomplete transmission of
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
Airlines) is registered in Amstelveen, The Netherlands, with registered number
33014286
********************************************************
-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.
|
