Very good - thanks Zlatko! I have only one thing to add: Do not stripe
the Database; it makes things slower. I tested it out, and it was MUCH
slower. With the random I/O pattern of the Database, and the multiple
threads inside the server program, it just made the arm contention
issues worse.
It is OK to stripe the log and disk storage pools; they are much more
sequential.
Roger Deschner University of Illinois at Chicago rogerd AT uic DOT edu
============ "The World's Least Intuitive Operating System" ============
=============== -- from the cover of "Unix for Dummies" ================
On Mon, 17 Feb 2003, Zlatko Krastev/ACIT wrote:
>Arnaud,
>
>--> 6h1 machine, 2Gb memory ...
>--> ... I increased bufpoolsize from 151552 to 524288 (where my performance
>problems began)!
>
>512k pages, 4 kB each equals 2 GB. So you left no space for AIX, file
>buffers, TSM code, etc. Consequences: excessive paging and performance
>degradation.
>
>--> Vmtune settings : -p10 -P40
>
>This means file buffers will occupy 10-40% of the real memory. Count it
>50% with AIX kernel, TCP/IP buffers and TSM code. The result - TSM data
>structures (mainly DB bufferpool and log befferpool) should not exceed 1
>GB. LOWER the buffpoolsize to 262144 !!!
>
>--> (Copy 1) Status (Copy 2)
>--> /tsmdb/db01.dsm Sync'd /tsmdb_m/db01.d
>+
>--> LV NAME TYPE LPs PPs PVs LV STATE MOUNT
>--> lvtsmdb jfs 128 256 8 open/syncd /tsmdb
>--> lvtsmdb_m jfs 128 128 4 open/syncd /tsmdb_m
>+
>--> /tsmdb/db01.dsm ... /tsmdb/db08.dsm
>
>Rodney already pointed you have excessive mirroring. Actually it is not
>4-way but 2+1 way (secondary copies are not AIX-mirrored). However the
>results are same - both AIX and TSM mirroring are applied sequentially
>introducing the sum of all consistency delays (you had not shown the
>"mirrorread db", "mirrorread log", "mirrorwrite db" and "mirrorwrite log"
>options from dsmserv.opt, so I am assuming the defaults are used).
>
>The discussion raw LVs vs. filesystem dbvols was always short - the
>performance benefit on AIX is small (on Solaris is much higher). However
>it is worth to try using raw LVs and further reduce file buffering using
>"vmtune -p 5 -P 10" (as Rodney already suggested). This may also allow to
>raise bufpoolsize to 65-75% of the RAM.
>
>The discussion how many dbvols per HDD was sparkling several times on this
>list. I personally am in the group of believers that single dbvol per HDD
>is better. The argument is simple - TSM attempts to "parallelize" the load
>over many dbvols results disk heads thrashing. The example in your case:
>- 8 dbvols within single filesystem;
>- the filesystem is on 4x2 disks (32 PPs each);
>- let TSM has to write 16 pages;
>1. dbvol1 is occupying PPs 1 through 4 on each disk, dbvol2 is on PPs 5-8,
>etc.
>2. TSM server attempt to "parallelize" will write page 1 on dbvol1, 2 on
>dbvol2, ..., 8 on dbvol8, 9 again on dbvol1, etc.
>3. the result will be write of pages 1&9 on PP 1 (for dbvol1), pages 2&10
>on PP 5 (for dbvol2), ..., and pages 8&16 on PP 29 (for dbvol8). ALL THOSE
>on HDD1 !!! What happened to the parallelism :-((( And why the disk should
>move the heads back'n'forth.
>
>The medicine (all can be done under load and without restarting AIX and/or
>TSM; however some performance impact should be expected):
>1. eliminate AIX mirroring (use rmlvcopy command)
>2. rearrange the tsmvg and free 4 disks (migratepv and reducevg)
>3. create new VG from those 4 disks using smaller PP size (more PPs per
>PV)
>4. create *separate* jfs logs on each disk (mklv -y <lognameN> -t jfslog
><vgname> 1 <pvN>)
>5. initialize each log (logform /dev/<lognameN>)
>6. create the filesystem LVs (mklv -y <dbvolN> <vgname> XYZ <pvN>)
>7. create each filesystem with *own* log (crfs -v jfs -d /dev/<dbvolN> -m
>/tsm/dbN -A yes -a logname=/dev/<lognameN>). Note the "-a logname" option
>of AIX crfs/chfs commands.
>8. define *single* big volume on each filesystem
>9. if volumes created in step 8 can be created same size as existing ones,
>define new volumes as third copy and delete the first copy. If not - dbvol
>delete/migrate ought to be used.
>10. delete the rest of tsmvg and mirror using only one method (use
>mklvcopy for AIX mirroring; use extendvg, repeat steps 4-8 and "def dbc"
>for TSM mirroring).
>11. repeat steps 1-10 for TSM log
>12. get rid of tsmvg_m and tsmvg_log_m. Use the disks for diskpool or add
>them as third copy within *the same* mirroring scheme (both AIX and TSM
>use LVM which allows three copies).
>
>--> ... and won't be back before Monday
>
>Happy Monday! You've got something to play with for the whole day :-)
>
>Zlatko Krastev
>IT Consultant
>
>P.S. I am charging my customers for such advices but hopefully I can get a
>beer (or Swiss chocolate) for this one :-)
>
>
>
>
>
>
>
>PAC Brion Arnaud <Arnaud.Brion AT PANALPINA DOT COM>
>Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>14.02.2003 16:08
>Please respond to "ADSM: Dist Stor Manager"
>
>
> To: ADSM-L AT VM.MARIST DOT EDU
> cc:
> Subject: Re: OS390 TSM Performance questions.
>
>
>Hi Rodney,
>
>The big picture :
>System : aix 4.3.3.0 on a 6h1 machine, 2Gb memory, 2 cpu
>Vmtune settings : -p10 -P40 -R 256 -F376 -W256 -s1
>TSM version : 4.2.3.1
>Bufpoolsize : 524288
>Iostat:
>tty: tin tout avg-cpu: % user % sys % idle %
>iowait
> 0.0 25.9 55.0 13.0 10.9
>21.1
>
>Vmstat :
>kthr memory page faults cpu
>----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 3 3 288460 112615 0 0 0 1220 2288 0 2034 3080 369 55 13 11 21
>
>>q dbvol
>
>Volume Name Copy Volume Name Copy Volume Name
>Copy
>(Copy 1) Status (Copy 2) Status (Copy 3)
>Status
>---------------- ------ ---------------- ------ ----------------
>------
>/tsmdb/db01.dsm Sync'd /tsmdb_m/db01.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db02.dsm Sync'd /tsmdb_m/db02.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db03.dsm Sync'd /tsmdb_m/db03.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db04.dsm Sync'd /tsmdb_m/db04.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db05.dsm Sync'd /tsmdb_m/db05.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db06.dsm Sync'd /tsmdb_m/db06.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db07.dsm Sync'd /tsmdb_m/db07.d- Sync'd
>Undef-
> sm
>ined
>/tsmdb/db08.dsm Sync'd /tsmdb_m/db08.d- Sync'd
>Undef-
> sm
>ined
>>q logvol
>
>Volume Name Copy Volume Name Copy Volume Name
>Copy
>(Copy 1) Status (Copy 2) Status (Copy 3)
>Status
>---------------- ------ ---------------- ------ ----------------
>------
>/tsmlog2/log02.- Sync'd /tsmlog_m/log02- Sync'd
>Undef-
> dsm .dsm
>ined
>/tsmlog2/log01.- Sync'd /tsmlog_m/log01- Sync'd
>Undef-
> dsm .dsm
>ined
>/tsmlog2/log03.- Sync'd /tsmlog_m/log03- Sync'd
>Undef-
> dsm .dsm
>ined
>/tsmlog2/log04.- Sync'd /tsmlog_m/log04- Sync'd
>Undef-
> dsm .dsm
>ined
>/tsmlog2/log05.- Sync'd Undef-
>Undef-
> dsm ined
>ined
>/tsmlog2/log06.- Sync'd Undef-
>Undef-
> dsm ined
>ined
>
>Some info about disk layout :
>
>tsmvg:
>LV NAME TYPE LPs PPs PVs LV STATE MOUNT
>POINT
>loglv00 jfslog 1 2 2 open/syncd N/A
>lvtsmdb jfs 128 256 8 open/syncd /tsmdb
>lvtsmdb1 jfs 128 256 8 open/syncd /tsmdb1
>tsmvg_log_m:
>LV NAME TYPE LPs PPs PVs LV STATE MOUNT
>POINT
>lvtsmlog_m jfs 96 96 3 open/syncd /tsmlog_m
>loglv05 jfslog 1 1 1 open/syncd N/A
>tsmvg_m:
>LV NAME TYPE LPs PPs PVs LV STATE MOUNT
>POINT
>lvtsmdb_m jfs 128 128 4 open/syncd /tsmdb_m
>loglv01 jfslog 1 2 2 open/syncd N/A
>tsmvg_log:
>LV NAME TYPE LPs PPs PVs LV STATE MOUNT
>POINT
>lvtsmlog2 jfs 96 192 6 open/syncd /tsmlog2
>loglv04 jfslog 1 2 2 open/syncd N/A
>
>All volumes for TSM db and logs are striped.
>What I'm experiencing is very high cpu usage (mini 50 %, max 99), and
>paging as soon as backups or expire inventory are started. Also Cache
>Hit Pct is low (98.69) although I increased bufpoolsize from 151552 to
>524288 (where my performance problems began)! Expire inventory needs
>more or less 20 hours to explore approx 7 million objects, and to delete
>8-9 % of them ...
>If you have a clue on wat is happening here, I'm taker ;-)
>(Take your time, I'll leave for W.E. in a couple of hours, and won't be
>back before Monday).
>Thanks in advance.
>
>Arnaud
>
>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>| Arnaud Brion, Panalpina Management Ltd., IT Group |
>| Viaduktstrasse 42, P.O. Box, 4002 Basel - Switzerland |
>| Phone: +41 61 226 19 78 / Fax: +41 61 226 17 01 |
>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
>
>
>-----Original Message-----
>From: Rodney clark [mailto:Rodney.Clark AT INGBANK DOT COM]
>Sent: Friday, 14 February, 2003 13:33
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: OS390 TSM Performance questions.
>
>
>Post us some details iostat vmstat and how much memory disks e.t.c. The
>big quick win on AIX is vmtune -p5 -P10 But I guess you a.ready know
>that.
>
>
>-----Original Message-----
>From: PAC Brion Arnaud [mailto:Arnaud.Brion AT PANALPINA DOT COM]
>Sent: Friday 14 February 2003 09:44
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: OS390 TSM Performance questions.
>
>
>Hi all,
>
>I followed your discussion with much interest, as I'm suffering from
>huge performance problem problem too. Unfortunately I'm not under OS390,
>but using AIX 4.3.3 : could someone tell me if there is some some trick
>like this one, that should be considered, when using this OS ? Another
>thing that annoys me : using "show memu SHORT" on my server (TSM
>4.2.3.1) returns : ANR2000E Unknown command - SHOW MEMU
>Could it be that this command is only available for OS390 TSM version ?
>Thanks in advance.
>
>Arnaud =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>| Arnaud Brion, Panalpina Management Ltd., IT Group |
>| Viaduktstrasse 42, P.O. Box, 4002 Basel - Switzerland |
>| Phone: +41 61 226 19 78 / Fax: +41 61 226 17 01 |
>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
|
