Hi. I had a server like that - log filled up extremely quickly, multiple log
pins etc.
I ended up actually doing a full DB DUMP, DB RELOAD, as it turned out to be
something corrupted within my database + logs. It has been generally running
fine ever since.
I do agree, this is something related to workload for sure, but also do look at
the health of your database and logs - this may be pointing to possible issues
within them.
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Loon, EJ van - SPLXO
Sent: Monday, 16 May 2011 11:49 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] TSM Recovery log is pinning since upgrade to 5.5.5.0 code
Hi Richard!
> So is 5.5.5.0 also having this issue with the log pinning?
Yes. As soon as the log hist 80% TSM starts delaying transactions. In
this level a bug was fixed which caused the log to fill too soon after
it reached 80%. I'm running 5.5.5.2 and it's still filling to 80%
multiple times a night.
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Cheung, Richard
Sent: zondag 15 mei 2011 04:02
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code
Hi
The worry about this is I had a PMR opened for this issue on my servers,
which are currently at 5.5.4. I was advised it's a known issue, refer to
APAR IC64438, and the recommended fix is to go to 5.5.5.0
So is 5.5.5.0 also having this issue with the log pinning?
I also believed this was happening in early versions of TSM v6.. But the
later builds are okay - can anyone confirm?
Richard Cheung | Server Engineer | Information Systems | Santos Limited
Santos Centre | 60 Flinders Street Adelaide SA 5000 Australia
Ph +61 8 8116 7125 | 7 Fax +61 8 8116 5073 | richard.cheung AT santos DOT com
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Dave Canan
Sent: Saturday, 14 May 2011 1:22 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] TSM Recovery log is pinning since upgrade to
5.5.5.0 code
Andy Raibeck and I have been monitoring this thread for the past few
weeks and have been having numerous discussions about what we need to do
to address these problems. In addition, we have noted the many PMRs
coming into the IBM support queue, and many of these PMRs are now on my
queue. Andy will be assisting me along with other support people as
needed.
As part of this analysis we will be doing, we have a new questionnaire
(titled "Items to Gather for a PMR with Log Pinning Condition for TSM
V5") that customers will be filling out that ask several questions
regarding your environment. We understand that this involves extra work
to gather this information, but there can be many different areas that
can cause log pinning conditions, so this information is needed. In
addition, there will be a script provided (named serverperf5a.pl) that
will help us gather additional data. Both of these will be provided to
you by support. When these PMRs are now opened, please make sure that
level 1 support adds the keyword LOGPIN55 to the PMR. This will allow
Andy and I to quickly find all the PMRs being worked for this issue.
Eric, your PMR is now one that I have on my queue (or I will shortly
today).
We will be contacting you to work the PMR.
Dave Canan
IBM ATS TSM Performance
ddcananATUSDOTIBMDOTCOM
916-723-2410
On Thu, May 12, 2011 at 7:53 AM, Loon, EJ van - SPLXO
<Eric-van.Loon AT klm DOT com
> wrote:
> Hi Rick!
> You are running in normal mode. In this mode the recovery log only
> contains uncommited transactions. Don't you agree that in this case,
> the log should NEVER reach 90%? It should not even reach something
like 40%!
> I don't want to have to implement extra monitoring to keep TSM from
> crashing. It should just work. Like it did when we were running 5.3...
> Kind regards,
> Eric van Loon
> KLM Royal Dutch Airlines
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Richard Rhodes
> Sent: donderdag 12 mei 2011 15:04
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code
>
> I have a cron job that checks log utilization every 15m. If the log
> goes over 90% the scripts displays a bunch of stuff (q sess, q proc, q
> log, show logpin, etc), runs a show logpin cancel, displays the
> stuff again,
> and finally emails me. (We run in normal mode.) It's been very
> effective
> in preventing a log full condition.
>
> rick
>
>
>
>
>
>
> From: "Bos, Karel" <Karel.Bos AT ATOSORIGIN DOT COM>
> To: ADSM-L AT VM.MARIST DOT EDU
> Date: 05/12/2011 06:11 AM
> Subject: Re: TSM Recovery log is pinning since upgrade to
5.5.5.0
> code
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
>
>
> Hi,
>
> Dunno if u already looked at these options: THROUGHPUTDATATHRESHOLD
> and THROUGHPUTTIMETHRESHOLD. After a couple of crashes due to log
> filling up, a couple of years back, I standard added them to all my
> clients (with low values to only kill the slowest/hung ones) and
> untill now never had any issues with the V5.5 and below log space
> under normal backup loads.
>
> Log filling only happend when some sys admin made some changes to the
> biggest file servers we had but even this was handled like it should
> be done by ITSM.
>
> Regards,
>
> Karel
>
>
>
> ________________________________________
> Van: ADSM: Dist Stor Manager [ADSM-L AT VM.MARIST DOT EDU] namens Loon, EJ
> van
> -
> SPLXO [Eric-van.Loon AT KLM DOT COM]
> Verzonden: donderdag 12 mei 2011 11:11
> Aan: ADSM-L AT VM.MARIST DOT EDU
> Onderwerp: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0
> code
>
> Hi Paul!
> We are already running 5.5.5.2 and the log is still filling up, even
> after switching from rollforward to normal mode.
> Management currently is questioning whether TSM is the right product
> for the future. Although I'm a big fan of TSM for 15 years, I'm really
> in doubt too...
> Kind regards,
> Eric van Loon
> KLM Royal Dutch Airlines
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Paul Fielding
> Sent: woensdag 11 mei 2011 20:05
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code
>
> Hi folks, if this has already been commented on, my apologies, I
> hvaen't been following closely but just noticed this thread.
>
> We were experiencing log pinned issues after upgrading to 5.5.0.0 code
> at one of my client sites. What we were finding was that,
> occasionally, I'd get up in the morning and look at the server to see
> it completely bogged down with a huge number of backups still
> attempting to run, but nothing was moving - the log was at 84% (over
> it's trigger), and it had been firing off db backups for the last 4h
> to no avail, the log wasn't getting low enough.
>
> A key symptom was that in the actlog we were seeing messages to the
> effect of "Recovery log is at 84%, transactions will be delayed by
> 3ms" (or something like that).
>
> In opening a ticket, it was found there was an issue in the 5.5.0.0
> code where, when the recovery log gets too full, it would start
> delaying transactions in order to prevent the log from filling too
quickly.
> However,
> the side effect was that the pinning transaction would also get
> delayed, causing a bit of a never-ending loop. Transactions keep
> getting delayed, pinned transaction never gets to finish, and
> everything would just grind to a halt. I would have to halt the
> server and restart in order to get things back to normal.
>
> IBM recognized this as an issue and recommended going to any 5.5.5.0
> level of code, where the problem was supposed to be fixed. I
> installed 5.5.5.2,
> and the problem has indeed gone away. It was supposed to be fixed at
> 5.5.5.0, though perhaps they didn't quite get it done at that code
> level as they hoped? I'd try installing 5.5.5.2 and see what
> happens....
>
> regards,
>
> Paul
>
>
> On Wed, May 11, 2011 at 9:03 AM, Loon, EJ van - SPLXO
> <Eric-van.Loon AT klm DOT com
> > wrote:
>
> > Hi Robert!
> > Thanks you very much for your reply! Several others on this list
> > reported this behavior and (as far as I know) three other users
> > opened
> a
> > PMR too. I hope they have more luck, because I'm stuck. Level 2
> > keeps
> on
> > saying that the log keeps on growing because of slow running client
> > sessions. Indeed I see slow running client sessions, but they are
> slowed
> > down by the fact that TSM is delaying all transactions because the
> > log is used for more that 80% during a large part of the backup
window!
> Now
> > they refuse to help me, unless I buy a Passport Advantage
> contract!!!!!!
> > Kind regards,
> > Eric van Loon
> > KLM Royal Dutch Airlines
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> > Behalf
> Of
> > Robert Clark
> > Sent: woensdag 11 mei 2011 16:05
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0
> > code
> >
> > I believe we've been seeing this problem as well.
> >
> > One night in the busiest backup period, I issued a "q actlog
> > begint=now-00:30", and got no results back but an error.
> >
> > I started dsmadmc -console on that server, and could see that the
> > console output was most of an hour behind. (And the console output
> > was
> scrolling
> > so fast that it could barely be read.)
> >
> > In that case, I think we determined that SQL LiteSpeed was set to
> > some rediculously small transaction size, and this was causing way
> > too many actlog entries.
> >
> > I think I also noted that the session number incremented by
> > something like 100,000 in an hour.
> >
> > Asking the users of SQL LiteSpeed to make some changes was enough to
> > rememdy this problem, although we continue to fight with the logs
> > getting full.
> >
> > Thanks,
> > [RC]
> >
> >
> >
> > From: "Loon, EJ van - SPLXO" <Eric-van.Loon AT KLM DOT COM>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 05/11/2011 02:05 AM
> > Subject: Re: [ADSM-L] TSM Recovery log is pinning since
upgrade
> > to
> > 5.5.5.0 code
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> >
> >
> > Hi TSM-ers!
> > Here is a small follow up on my PMR about the recovery log
> utilization.
> > I'm at TSM level 2, still trying to convince them that there is
> > something broken in the TSM server code. To convince them, I have
> > changed the logmode to normal on one of my servers. I created a
> > graph (through TSMManager) which shows the recovery log utilization
> > during last night's client backup window and is doesn't differ much
> > from the night before, with logmode rollforward. When running in
> > normal mode,
> TSM
> > should only use the recovery log for uncommitted transactions, so
> > utilization should be very low. My log is 12 Gb and the
> > backuptrigger value (75%) was still hit twice!
> > This clearly shows that there is something wrong with TSM, let's
> > hope
> I
> > can convince Level 2 too, so my case gets forwarded to the lab.
> > I'll keep you guys posted!
> > Kind regards,
> > Eric van Loon
> > KLM Royal Dutch Airlines
> > ********************************************************
> > For information, services and offers, please visit our web site:
> > http://www.klm.com. This e-mail and any attachment may contain
> > confidential and privileged material intended for the addressee
only.
> If
> > you are not the addressee, you are notified that no part of the
> > e-mail or any attachment may be disclosed, copied or distributed,
> > and that any other action related to this e-mail or attachment is
> > strictly prohibited,
> and
> > may be unlawful. If you have received this e-mail by error, please
> > notify the sender immediately by return e-mail, and delete this
> > message.
> >
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
> > and/or its employees shall not be liable for the incorrect or
> > incomplete transmission of this e-mail or any attachments, nor
> > responsible for any delay in receipt.
> > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> Dutch
> > Airlines) is registered in Amstelveen, The Netherlands, with
> registered
> > number 33014286
> > ********************************************************
> >
> >
> >
> > U.S. BANCORP made the following annotations
> > --------------------------------------------------------------------
> > - Electronic Privacy Notice. This e-mail, and any attachments,
> > contains information that is, or may be, covered by electronic
> > communications privacy laws, and is also confidential and
> > proprietary in nature. If
> you
> > are not the intended recipient, please be advised that you are
> > legally prohibited from retaining, using, copying, distributing, or
> > otherwise disclosing this information in any manner. Instead, please
> > reply to
> the
> > sender that you have received this communication in error, and then
> > immediately delete it. Thank you in advance for your cooperation.
> >
> >
> >
> > --------------------------------------------------------------------
> > -
> > ********************************************************
> > For information, services and offers, please visit our web site:
> > http://www.klm.com. This e-mail and any attachment may contain
> > confidential and privileged material intended for the addressee
only.
> If you
> > are not the addressee, you are notified that no part of the e-mail
> > or
> any
> > attachment may be disclosed, copied or distributed, and that any
> > other action related to this e-mail or attachment is strictly
> > prohibited,
> and may
> > be unlawful. If you have received this e-mail by error, please
> > notify
> the
> > sender immediately by return e-mail, and delete this message.
> >
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
> > and/or
> its
> > employees shall not be liable for the incorrect or incomplete
> transmission
> > of this e-mail or any attachments, nor responsible for any delay in
> receipt.
> > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> Dutch
> > Airlines) is registered in Amstelveen, The Netherlands, with
> registered
> > number 33014286
> > ********************************************************
> >
> >
> ********************************************************
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain
> confidential and privileged material intended for the addressee only.
> If you are not the addressee, you are notified that no part of the
> e-mail or any attachment may be disclosed, copied or distributed, and
> that any other action related to this e-mail or attachment is strictly
> prohibited, and may be unlawful. If you have received this e-mail by
> error, please notify the sender immediately by return e-mail, and
> delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
> its employees shall not be liable for the incorrect or incomplete
> transmission of this e-mail or any attachments, nor responsible for
> any delay in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with
> registered number 33014286
> ********************************************************
>
>
>
> -----------------------------------------
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and that
> any review, dissemination, distribution, or copying of this message is
> strictly prohibited. If you have received this communication in error,
> please notify us immediately, and delete the original message.
> ********************************************************
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain
> confidential and privileged material intended for the addressee only.
> If you are not the addressee, you are notified that no part of the
> e-mail or any attachment may be disclosed, copied or distributed, and
> that any other action related to this e-mail or attachment is strictly
> prohibited, and may be unlawful. If you have received this e-mail by
> error, please notify the sender immediately by return e-mail, and
delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
> its employees shall not be liable for the incorrect or incomplete
> transmission of this e-mail or any attachments, nor responsible for
any delay in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with
> registered number 33014286
> ********************************************************
>
>
<html>
<body>
<font face="arial" color=#808080 size="-2"><img
alt="Santos Logo" src="http://www.santos.com/library/logo.gif";>
<br>Santos Ltd A.B.N. 80 007 550 923<br>
Disclaimer: The information contained in this email is intended only for
the use of the person(s) to whom it is addressed and may be confidential
or contain privileged information.
If you are not the intended recipient you are hereby notified that any
perusal, use, distribution, copying or disclosure is strictly
prohibited.
If you have received this email in error please immediately advise us by
return email and delete the email without making a copy.</font>
<font face="arial" color=#008000 size="-2">Please consider the
environment before printing this email</font>
</body>
</html>
</pre>********************************************************<br>For
information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain confidential and
privileged material intended for the addressee only. If you are not the
addressee, you are notified that no part of the e-mail or any attachment may be
disclosed, copied or distributed, and that any other action related to this
e-mail or attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately by return
e-mail, and delete this message.<br><br>Koninklijke Luchtvaart Maatschappij NV
(KLM), its subsidiaries and/or its employees shall not be liable for the
incorrect or incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.<br>Koninklijke Luchtvaart Maatschappij
N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The
Netherlands, with registered number 33014286
<br>********************************************************<pre>
|
![http://www.santos.com/library/logo.gif"](http://www.santos.com/library/logo.gif"){kind=link}