ADSM-L

Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned

2007-07-27 21:29:20
Subject: Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned
From: Craig Ross <craig.outlook AT GMAIL DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Sat, 28 Jul 2007 13:25:46 +1200
Thanks for all input guys,

Firstly sorry for lack of detail.

TSM is installed on Solaris 10, No I did not do any benchmarking, as we were
not replacing any existing setup just adding more, I have 6 LTO drives
already installed and about 17TB of SAN storage which is Primary Random, I
have since added 4 new LTO 3 drives (with different Device classes) and they
run better than the LTO 1 drives, when server is not logpinning. And the new
15 TB of SATA, now approx 1 TB of the DISK is Primary Random storage and the
remainder is SEQ file and its all FS not RAW, I have had heavy discussion
over RAW vs FS and I have not been able to find definitive answer

Clients I don;t think are causing the issue any TSM processes can pin the
log from migrations DB backups and clients sessions. Once the server starts
getting busy.

Currently (sorry not in front of installation) but I guess of the 15TB I
have about 60 sequential volumes across 4 Stgpools, and I have still more to
define.

I have not had clients utilize this new storage yet, all I have done is
start to migrate data into these STGPools to release some of the legacy
STGpools


 The Transport to DB and Recovery Log is SAN, however yesterday I created
local copy and this did not improve things.

The STGpools are on WMS SAN and AMS500 SAN. Both SATA disk. All across
Fibre!!


 The SAN engineer when installing the Disk's saw expected performance out of
the DISK's.

I also don't see it being maxsessions because I can Pin the log with 3 or 4
sessions and 3 processes!


 I think its safe to say its configuration somewhere, because now I think
about it its not taking much load to pin the log. Load in which TSM normally
copes ok!!

Next step may be to remove the New DISK's now will I need to just unmount
the FS or will i Need to migrate data off new storage and delete volumes and
New STGpools?

Thanks


On 7/28/07, Lawrence Clark <Larry_Clark AT thruway.state.ny DOT us> wrote:
>
> Assuming the SATA are on AIX, were the logical volumes set up to hold
> the volumes
> defined as JFS2?
>
> >>> ian-it.smith AT DB DOT COM 07/27/2007 2:30:54 PM >>>
> Do the client backup sessions pin the log? What is the throughput on
> the
> actual client session and are these backups direct to disk? If the
> sessions are cancelled does the system come back to life?
>
> 15 TB of SATA sounds like a lot of storage. how has this been
> added/configured- What raw throughput do you get on these disks outside
> of
> TSM itself?
>
> You say the LTO3 drives are new. Do you have existing LTO3 drives?
> Have
> you configured them correctly with new device class etc if you are
> mixing
> LTO generations in the library?
>
> I have seen this type of pinning/dramatic slow down before. I saw
> itself
> manifest by the server hitting the maxsessions limit as all the
> sessions
> were running so slowly to the disk pool.
>
> Lots of questions i know, but as you have made multiple changes at the
> same time- its going to be difficult to nail down without additional
> info.
>
> Ian Smith
> -------------------------------------------------------
> Core Engineering - Storage
>
>
>
>
>
> Robert Clark <Robert_Clark AT MAC DOT COM>
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> 27/07/2007 18:01
> Please respond to
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
>
> To
> ADSM-L AT VM.MARIST DOT EDU
> cc
>
> Subject
> Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned
>
>
>
>
>
>
> Is the SATA setup as disk storage pools? Is it filesystem or raw
> logical volumes?
>
> What is the OS? vmstat or top/topas may give some ideas.
>
> What is the network transport? Fast ethernet?
>
> [RC]
>
> On Jul 27, 2007, at 2:49 AM, Craig Ross wrote:
>
> > 10 days ago I Recently added 15TB of SATA storage and a new Fabric
> > with 4
> > new LTO drives to our 3584 library,
> > The DB is approx 90GB TSM
> >
> > Few days ago I noticed processing had ground to halt, after digging
> > around I
> > have found as soon as server gets busy maybe 4 processes 8 or so
> > sessions
> > the recovery log begins "sh logpinned" to pin and the Database gets
> > locks.
> > Shown by running "sh locks"
> > And as result the server suffers!
> > Now today I have stopped using the new Tech LTO 3 and SATA and
> > things are
> > coping better but still worse than previous as soon as load is
> > increased Log
> > pins and processing slows drastically.
> >
> > Are there any steps I can take which will help my scenario.
> > Would a DB UNLOAD RELOAD help that much?
> >
> > Reference: Recovery log has heaps of room DB has heaps of room 90Gb
> > DB with
> > 100GB of room.
> >
> > Any advice is much appreciated.
>
>
>
> ---
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
> Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> additional EU corporate and regulatory disclosures.
>
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain information that is confidential, privileged, and/or otherwise
> exempt from disclosure under applicable law.  If this electronic message is
> from an attorney or someone in the Legal Department, it may also contain
> confidential attorney-client communications which may be privileged and
> protected from disclosure.  If you are not the intended recipient, be
> advised that you have received this message in error and that any use,
> dissemination, forwarding, printing, or copying is strictly
> prohibited.  Please notify the New York State Thruway Authority immediately
> by either responding to this e-mail or calling (518) 436-2700, and destroy
> all copies of this message and any attachments.
>