ADSM-L

Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned

2007-07-31 10:10:02
Subject: Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned
From: Richard Rhodes <rrhodes AT FIRSTENERGYCORP DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 31 Jul 2007 09:15:13 -0400
To create a db with raw volumes . . . .

create a vg:      mkvg  . . . .
create log vols:  mklv for each log vol
create dbvols:  mklv for each db vol

create db . . .  Here is the script/cmd I used to create a tsm db on the
raw vols

rsfebkup7p.fenetwork.com:/tsmdata/tsm3/config==>cat z_tsm3_s1_format_db.ksh
#!/bin/ksh
#
# Set the language....
#
export LANG=en_US

#
# Max out size of data area
#
ulimit -d unlimited

#
# Allow the server to pack shared memory segments
#
export EXTSHM=ON

# setup to run tsm3
cd /tsmdata/tsm3/config
export PATH=${PATH}:/usr/tivoli/tsm/server/bin
export DSMSERV_DIR=/usr/tivoli/tsm/server/bin
export DSMSERV_CONFIG=/tsmdata/tsm3/config/dsmserv.opt
export DSMSERV_ACCOUNTING_DIR=/tsmdata/tsm3/config

dsmserv_tsm3 format 3 /dev/rtsm3log01 \
                      /dev/rtsm3log02 \
                      /dev/rtsm3log03 \
                    9 /dev/rtsm3db01 \
                      /dev/rtsm3db02 \
                      /dev/rtsm3db03 \
                      /dev/rtsm3db04 \
                      /dev/rtsm3db05 \
                      /dev/rtsm3db06 \
                      /dev/rtsm3db07 \
                      /dev/rtsm3db08 \
                      /dev/rtsm3db09




             Mark Scott
             <MScott@BUNNINGS.
             COM.AU>                                                    To
             Sent by: "ADSM:           ADSM-L AT VM.MARIST DOT EDU
             Dist Stor                                                  cc
             Manager"
             <[email protected]                                     Subject
             .EDU>                     Re: TSM performance very poor,
                                       Recovery log is being pinned

             07/31/2007 08:16
             AM


             Please respond to
             "ADSM: Dist Stor
                 Manager"
             <[email protected]
                   .EDU>






Evening I have been watching these comments with interest as we are
currently in the process of building a new TSM server.
Discussing with colleagues we are baffled by how you create the TSM log
or DB on a raw presented Lun without creating at least a JFS for a mount
point first. We are running AIX 5.3.

Look forward to your response

Regards

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Craig Ross
Sent: Tuesday, 31 July 2007 7:09 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] TSM performance very poor, Recovery log is being
pinned

Thanks guys.

All this advise is much appreciated.
For the record my TSM servers seems to have returned to more normal
routines.
However the log is still being pinned and log fills up to about 12% then
flushes, not ideal but acceptable.
I am starting to wonder if my install has always had the log pinning
issue I
just did not know!!
I will keep close eye on it.   But I feel ultimately I will install 2nd
TSM
server and migrate some Node's to new server.
I did find a bad configured batch of new SAN storage which is promoting
the
slowdown/pinning. I have stopped using this particular disk and
performance
has returned to norm.  However this was not obvious before because
server
was busy, I could not pin point easy as once stopping the trouble spot
TSM
was not jumping into life straight away it had too process the back log
of
requests And of course I could not afford to leave server on go slow for
long periods.

As result I am currently in review of all new Storage installed to
ensure
its running optimally.

I will propose to management to install new fast disk for log and DB as
the
pinning still issue.

However this is where the debate continues, I have experimented with DB
volumes log volumes and storage volumes with FS and raw volumes and have
not
seen any performance difference (on the same disks) ie have deleted
volumes
created RAW and FS and seen no difference.
I have read material on both sides of this story and no seems more
convincing than the other except I remember reading somewhere the only
eyebrow raiser was with RAW on solaris you can have issues, I cannot
remember exact issue but potential is there. So after testing both
formats
finding no difference. I use FS for everything, if this is an
indisputable
mistake please let me know.
Also we have 10 10 Gb volumes for DB should I create more smaller ones?

While I am here though I have another cloudy area, since upgrading
server
from 5.1 to 5.3 installing IBM tape device driver and adding 4 LTO3
drives I
swear my 6 old LTO 1 drives are running slower than previous is there
some
gotchas when installing IBM Tape to get drives running well!

6 LTO 1 drives are SCSI attached and 4 new LTO 3 are fibre.

Cheers

A happier TSM administrator the last 2 days  :>




On 7/31/07, Roger Deschner <rogerd AT uic DOT edu> wrote:
>
> .
> I think you are right about the Log - it need not be spread across
> multiple volumes. It's only got one writer.
>
> Your RAID type can affect the performance of the Disk Storage Pools
and
> the Database dramatically. In particular, RAID5 is very poorly suited
> for this, because it is 50% writes. RAID5 is also not ideal for the
> Database, though it can be tolerated for the Log. RAID10 is much
better.
>
> You should be using fast disks, not SATA, for the primary Disk Storage
> Pools. I've got 10,000rpm IBM SSA disks for these.
>
> I use RAID10 for the Disk Storage Pools. I use JBOD disks with TSM
> mirroring for the Log and Database. This is slightly slower than OS
> mirroring or RAID-array mirroring, but it is somewhat safer. Each
> physical volume for Storage Pools and Database is broken into many
> Logical Volumes.
>
> You should be saving your fastest disks for the Database. I've got
> 15,000prm disks for the Database. When I moved the Database from
> 10,000rpm disks to 15,000rpm disks, everything in TSM got noticeably
> faster. For instance, DB backups now take 1/3 less time. RAID boxes
just
> get in the way for the Database; it really runs best on JBOD disks
with
> TSM doing the mirroring.
>
> Here's a controversial paper written by a guy at Oracle. He says you
> should "Stripe And Mirror Everything" (S.A.M.E.) I've read and reread
> this several times, and while I definitely do not agree with
everything
> said, it does raise some very interesting points that definitely apply
> to TSM. For one thing he strongly advocates RAID10, as do I.
>
http://www.oracle.com/technology/deploy/availability/pdf/oow2000_same.pd
f
>
> Most of my Log pinning problems have been caused by clients. If a
client
> suffers a networking problem (typically a half-duplex vs. full-duplex
> conflict) and if that client tries to back up a large file such as a
> movie, that can pin the log on our system until it fills completely.
> Minimum throughput controls in TSM can help here, though it can still
> happen. I wrote a daemon that watches the Log fullness and if it gets
to
> about 70% it cancels the session that has the Log pinned. I still have
> problems, because the cancel command can take hours to work if the
> client is backing up a large file slowly. If the Log gets to 95% it
does
> a TSM shutdown command, which is vastly easier to recover from than a
> 100% full log. At least with a full TSM shutdown, our novice
sysadmin's
> first impulse which is to try to restart it, is generally a good thing
> to do. It usually restarts with an empty Log in these cases, so they
can
> claim, "I fixed it!" without knowing the underlying complexities.
>
> Roger Deschner      University of Illinois at Chicago
rogerd AT uic DOT edu
> ===== "Standards are great. That's why there are so many of them."
=====
>
>
>
>
> On Mon, 30 Jul 2007, Andrew Carlson wrote:
>
> >always heard the DB should, because it opens multiple threads with
> multiple
> >volumes, but since the log is sequentially written to for the most
part,
> I
> >can't figure out why that should be in multiple volumes.  Thanks.
> >
> >On 7/30/07, Charles A Hart <charles_hart AT uhc DOT com> wrote:
> >>
> >> Your DB and Log shold be RAW as well, and in small vols.  (ie 12GB
log
> >> should be in 2-3GB VOls, DB, vols, depengin on size of db should be
> 5-10GB
> >> vols.  Also try to make sure the raw logical vols are evenly spread
> >> accross as many LUNs as possible.
> >>
> >> Charles Hart
> >>
> >>
> >>
> >>
> >>
> >> "Stapleton, Mark" <mark.stapleton AT BERBEE DOT COM>
> >> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >> 07/29/2007 07:03 AM
> >> Please respond to
> >> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >>
> >>
> >> To
> >> ADSM-L AT VM.MARIST DOT EDU
> >> cc
> >>
> >> Subject
> >> Re: [ADSM-L] TSM performance very poor, Recovery log is being
pinned
> >>
> >>
> >>
> >>
> >>
> >>
> >> From: ADSM: Dist Stor Manager on behalf of Craig Ross
> >> >TSM is installed on Solaris 10
> >>
> >> This is something that popped right out for me. Do you have your
> storage
> >> pools located on raw logical volumes or mounted filesystems? If the
> >> latter, that might be your problem. Solaris has traditionally had
> >> incredibly poor throughput performance on mounted filesystems.
> >>
> >> You might give thought to rebuilding those storage pools on raw
logical
> >> volumes. Of course, that will require that you completely flush all
> data
> >> from your disk storage pools to tape storage pools first, so as not
to
> >> lose client data.
> >>
> >> --
> >> Mark Stapleton (mark.stapleton AT berbee DOT com)
> >> Berbee Information Networks (a CDW company)
> >>
> >>
> >>
> >> This e-mail, including attachments, may include confidential and/or
> >> proprietary information, and may be used only by the person or
entity
> to
> >> which it is addressed. If the reader of this e-mail is not the
intended
> >> recipient or his or her authorized agent, the reader is hereby
notified
> >> that any dissemination, distribution or copying of this e-mail is
> >> prohibited. If you have received this e-mail in error, please
notify
> the
> >> sender by replying to this message and delete this e-mail
immediately.
> >>
> >
> >
> >
> >--
> >Andy Carlson
>
>
>-----------------------------------------------------------------------
----
> >Gamecube:$150,PSO:$50,Broadband Adapter: $35, Hunters License:
> $8.95/month,
> >The feeling of seeing the red box with the item you want in
it:Priceless.
> >
>

************************************************************************
Bunnings Legal Disclaimer:

1)     This email is confidential and may contain legally privileged
information.  If you are not the intended recipient, you must not
disclose or use the information contained in it.  If you have received
this email in error, please notify us immediately by return email and
delete the document.

2)     All emails sent to and sent from Bunnings Group Limited.
are scanned for content.  Any material deemed to contain inappropriate
subject matter will be reported to the email administrator of all
parties concerned.
************************************************************************



-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.