ADSM-L

Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned

2007-07-31 13:08:37
Subject: Re: [ADSM-L] TSM performance very poor, Recovery log is being pinned
From: Wanda Prather <wprather AT JASI DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 31 Jul 2007 11:12:17 -0500
In the TSM admin guide for AIX, look up "raw volumes".
It has examples.


> Evening I have been watching these comments with interest as we are
> currently in the process of building a new TSM server.
> Discussing with colleagues we are baffled by how you create the TSM log
> or DB on a raw presented Lun without creating at least a JFS for a mount
> point first. We are running AIX 5.3.
>
> Look forward to your response
>
> Regards
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Craig Ross
> Sent: Tuesday, 31 July 2007 7:09 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] TSM performance very poor, Recovery log is being
> pinned
>
> Thanks guys.
>
> All this advise is much appreciated.
> For the record my TSM servers seems to have returned to more normal
> routines.
> However the log is still being pinned and log fills up to about 12% then
> flushes, not ideal but acceptable.
> I am starting to wonder if my install has always had the log pinning
> issue I
> just did not know!!
> I will keep close eye on it.   But I feel ultimately I will install 2nd
> TSM
> server and migrate some Node's to new server.
> I did find a bad configured batch of new SAN storage which is promoting
> the
> slowdown/pinning. I have stopped using this particular disk and
> performance
> has returned to norm.  However this was not obvious before because
> server
> was busy, I could not pin point easy as once stopping the trouble spot
> TSM
> was not jumping into life straight away it had too process the back log
> of
> requests And of course I could not afford to leave server on go slow for
> long periods.
>
> As result I am currently in review of all new Storage installed to
> ensure
> its running optimally.
>
> I will propose to management to install new fast disk for log and DB as
> the
> pinning still issue.
>
> However this is where the debate continues, I have experimented with DB
> volumes log volumes and storage volumes with FS and raw volumes and have
> not
> seen any performance difference (on the same disks) ie have deleted
> volumes
> created RAW and FS and seen no difference.
> I have read material on both sides of this story and no seems more
> convincing than the other except I remember reading somewhere the only
> eyebrow raiser was with RAW on solaris you can have issues, I cannot
> remember exact issue but potential is there. So after testing both
> formats
> finding no difference. I use FS for everything, if this is an
> indisputable
> mistake please let me know.
> Also we have 10 10 Gb volumes for DB should I create more smaller ones?
>
> While I am here though I have another cloudy area, since upgrading
> server
> from 5.1 to 5.3 installing IBM tape device driver and adding 4 LTO3
> drives I
> swear my 6 old LTO 1 drives are running slower than previous is there
> some
> gotchas when installing IBM Tape to get drives running well!
>
> 6 LTO 1 drives are SCSI attached and 4 new LTO 3 are fibre.
>
> Cheers
>
> A happier TSM administrator the last 2 days  :>
>
>
>
>
> On 7/31/07, Roger Deschner <rogerd AT uic DOT edu> wrote:
>>
>> .
>> I think you are right about the Log - it need not be spread across
>> multiple volumes. It's only got one writer.
>>
>> Your RAID type can affect the performance of the Disk Storage Pools
> and
>> the Database dramatically. In particular, RAID5 is very poorly suited
>> for this, because it is 50% writes. RAID5 is also not ideal for the
>> Database, though it can be tolerated for the Log. RAID10 is much
> better.
>>
>> You should be using fast disks, not SATA, for the primary Disk Storage
>> Pools. I've got 10,000rpm IBM SSA disks for these.
>>
>> I use RAID10 for the Disk Storage Pools. I use JBOD disks with TSM
>> mirroring for the Log and Database. This is slightly slower than OS
>> mirroring or RAID-array mirroring, but it is somewhat safer. Each
>> physical volume for Storage Pools and Database is broken into many
>> Logical Volumes.
>>
>> You should be saving your fastest disks for the Database. I've got
>> 15,000prm disks for the Database. When I moved the Database from
>> 10,000rpm disks to 15,000rpm disks, everything in TSM got noticeably
>> faster. For instance, DB backups now take 1/3 less time. RAID boxes
> just
>> get in the way for the Database; it really runs best on JBOD disks
> with
>> TSM doing the mirroring.
>>
>> Here's a controversial paper written by a guy at Oracle. He says you
>> should "Stripe And Mirror Everything" (S.A.M.E.) I've read and reread
>> this several times, and while I definitely do not agree with
> everything
>> said, it does raise some very interesting points that definitely apply
>> to TSM. For one thing he strongly advocates RAID10, as do I.
>>
> http://www.oracle.com/technology/deploy/availability/pdf/oow2000_same.pd
> f
>>
>> Most of my Log pinning problems have been caused by clients. If a
> client
>> suffers a networking problem (typically a half-duplex vs. full-duplex
>> conflict) and if that client tries to back up a large file such as a
>> movie, that can pin the log on our system until it fills completely.
>> Minimum throughput controls in TSM can help here, though it can still
>> happen. I wrote a daemon that watches the Log fullness and if it gets
> to
>> about 70% it cancels the session that has the Log pinned. I still have
>> problems, because the cancel command can take hours to work if the
>> client is backing up a large file slowly. If the Log gets to 95% it
> does
>> a TSM shutdown command, which is vastly easier to recover from than a
>> 100% full log. At least with a full TSM shutdown, our novice
> sysadmin's
>> first impulse which is to try to restart it, is generally a good thing
>> to do. It usually restarts with an empty Log in these cases, so they
> can
>> claim, "I fixed it!" without knowing the underlying complexities.
>>
>> Roger Deschner      University of Illinois at Chicago
> rogerd AT uic DOT edu
>> ===== "Standards are great. That's why there are so many of them."
> =====
>>
>>
>>
>>
>> On Mon, 30 Jul 2007, Andrew Carlson wrote:
>>
>> >always heard the DB should, because it opens multiple threads with
>> multiple
>> >volumes, but since the log is sequentially written to for the most
> part,
>> I
>> >can't figure out why that should be in multiple volumes.  Thanks.
>> >
>> >On 7/30/07, Charles A Hart <charles_hart AT uhc DOT com> wrote:
>> >>
>> >> Your DB and Log shold be RAW as well, and in small vols.  (ie 12GB
> log
>> >> should be in 2-3GB VOls, DB, vols, depengin on size of db should be
>> 5-10GB
>> >> vols.  Also try to make sure the raw logical vols are evenly spread
>> >> accross as many LUNs as possible.
>> >>
>> >> Charles Hart
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> "Stapleton, Mark" <mark.stapleton AT BERBEE DOT COM>
>> >> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>> >> 07/29/2007 07:03 AM
>> >> Please respond to
>> >> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>> >>
>> >>
>> >> To
>> >> ADSM-L AT VM.MARIST DOT EDU
>> >> cc
>> >>
>> >> Subject
>> >> Re: [ADSM-L] TSM performance very poor, Recovery log is being
> pinned
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> From: ADSM: Dist Stor Manager on behalf of Craig Ross
>> >> >TSM is installed on Solaris 10
>> >>
>> >> This is something that popped right out for me. Do you have your
>> storage
>> >> pools located on raw logical volumes or mounted filesystems? If the
>> >> latter, that might be your problem. Solaris has traditionally had
>> >> incredibly poor throughput performance on mounted filesystems.
>> >>
>> >> You might give thought to rebuilding those storage pools on raw
> logical
>> >> volumes. Of course, that will require that you completely flush all
>> data
>> >> from your disk storage pools to tape storage pools first, so as not
> to
>> >> lose client data.
>> >>
>> >> --
>> >> Mark Stapleton (mark.stapleton AT berbee DOT com)
>> >> Berbee Information Networks (a CDW company)
>> >>
>> >>
>> >>
>> >> This e-mail, including attachments, may include confidential and/or
>> >> proprietary information, and may be used only by the person or
> entity
>> to
>> >> which it is addressed. If the reader of this e-mail is not the
> intended
>> >> recipient or his or her authorized agent, the reader is hereby
> notified
>> >> that any dissemination, distribution or copying of this e-mail is
>> >> prohibited. If you have received this e-mail in error, please
> notify
>> the
>> >> sender by replying to this message and delete this e-mail
> immediately.
>> >>
>> >
>> >
>> >
>> >--
>> >Andy Carlson
>>
>>
>>-----------------------------------------------------------------------
> ----
>> >Gamecube:$150,PSO:$50,Broadband Adapter: $35, Hunters License:
>> $8.95/month,
>> >The feeling of seeing the red box with the item you want in
> it:Priceless.
>> >
>>
>
> ************************************************************************
> Bunnings Legal Disclaimer:
>
> 1)     This email is confidential and may contain legally privileged
> information.  If you are not the intended recipient, you must not
> disclose or use the information contained in it.  If you have received
> this email in error, please notify us immediately by return email and
> delete the document.
>
> 2)     All emails sent to and sent from Bunnings Group Limited.
> are scanned for content.  Any material deemed to contain inappropriate
> subject matter will be reported to the email administrator of all
> parties concerned.
> ************************************************************************
>