ADSM-L

Re: [ADSM-L] TSM Recovery log is pinning since upgrade to 5.5.5.0 code

2011-05-12 06:05:33
Subject: Re: [ADSM-L] TSM Recovery log is pinning since upgrade to 5.5.5.0 code
From: "Loon, EJ van - SPLXO" <Eric-van.Loon AT KLM DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 12 May 2011 11:56:18 +0200
Hi Grigori!
The log is filling up very rapidly. The fact that TSM starts delaying
transactions when reaching 80% prevents the log from filling up to 100%
most of the time. If I would change LOGWARNFULLPERCENT to an even higher
percentage, the 100% will be reached even faster. During the weekend,
when a lot of database clients are making full backups, TSM desperately
tries to lower the log by triggering incremental after incremental, but
it happened several times that the log filled up to 100% and I was paged
to fix it from home...
Yesterday we did some extensive LAN performance testing, to see if the
LAN adapter of the host is the bottleneck: we measured 128 Mbyte/sec, so
no bottleneck here!
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Grigori Solonovitch
Sent: donderdag 12 mei 2011 11:39
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code

Have you tried to resolve problem by "SETOPT LOGWARNFULLPERCENT 95" to
avoid "Recovery log is at 84%, transactions will be delayed by 3ms" for
some time.
Maybe some slow sessions will be completed till log utilization reaches
95%?

Grigori G. Solonovitch


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Loon, EJ van - SPLXO
Sent: Thursday, May 12, 2011 12:11 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] TSM Recovery log is pinning since upgrade to
5.5.5.0 code

Hi Paul!
We are already running 5.5.5.2 and the log is still filling up, even
after switching from rollforward to normal mode.
Management currently is questioning whether TSM is the right product for
the future. Although I'm a big fan of TSM for 15 years, I'm really in
doubt too...
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Paul Fielding
Sent: woensdag 11 mei 2011 20:05
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code

Hi folks, if this has already been commented on, my apologies, I hvaen't
been following closely but just noticed this thread.

We were experiencing log pinned issues after upgrading to 5.5.0.0 code
at
one of my client sites.  What we were finding was that, occasionally,
I'd
get up in the morning and look at the server to see it completely bogged
down with a huge number of backups still attempting to run, but nothing
was
moving - the log was at 84% (over it's trigger), and it had been firing
off
db backups for the last 4h to no avail, the log wasn't getting low
enough.

A key symptom was that in the actlog we were seeing messages to the
effect
of "Recovery log is at 84%, transactions will be delayed by 3ms" (or
something like that).

In opening a ticket, it was found there was an issue in the 5.5.0.0 code
where, when the recovery log gets too full, it would start delaying
transactions in order to prevent the log from filling too quickly.
However,
the side effect was that the pinning transaction would also get delayed,
causing a bit of a never-ending loop.  Transactions keep getting
delayed,
pinned transaction never gets to finish, and everything would just grind
to
a halt.  I would have to halt the server and restart in order to get
things
back to normal.

IBM recognized this as an issue and recommended going to any 5.5.5.0
level
of code, where the problem was supposed to be fixed.  I installed
5.5.5.2,
and the problem has indeed gone away.   It was supposed to be fixed at
5.5.5.0, though perhaps they didn't quite get it done at that code level
as
they hoped?  I'd try installing 5.5.5.2 and see what happens....

regards,

Paul


On Wed, May 11, 2011 at 9:03 AM, Loon, EJ van - SPLXO
<Eric-van.Loon AT klm DOT com
> wrote:

> Hi Robert!
> Thanks you very much for your reply! Several others on this list
> reported this behavior and (as far as I know) three other users opened
a
> PMR too. I hope they have more luck, because I'm stuck. Level 2 keeps
on
> saying that the log keeps on growing because of slow running client
> sessions. Indeed I see slow running client sessions, but they are
slowed
> down by the fact that TSM is delaying all transactions because the log
> is used for more that 80% during a large part of the backup window!
Now
> they refuse to help me, unless I buy a Passport Advantage
contract!!!!!!
> Kind regards,
> Eric van Loon
> KLM Royal Dutch Airlines
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
Of
> Robert Clark
> Sent: woensdag 11 mei 2011 16:05
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: TSM Recovery log is pinning since upgrade to 5.5.5.0 code
>
> I believe we've been seeing this problem as well.
>
> One night in the busiest backup period, I issued a "q actlog
> begint=now-00:30", and got no results back but an error.
>
> I started dsmadmc -console on that server, and could see that the
> console
> output was most of an hour behind. (And the console output was
scrolling
> so fast that it could barely be read.)
>
> In that case, I think we determined that SQL LiteSpeed was set to some
> rediculously small transaction size, and this was causing way too many
> actlog entries.
>
> I think I also noted that the session number incremented by something
> like
> 100,000 in an hour.
>
> Asking the users of SQL LiteSpeed to make some changes was enough to
> rememdy this problem, although we continue to fight with the logs
> getting
> full.
>
> Thanks,
> [RC]
>
>
>
> From:   "Loon, EJ van - SPLXO" <Eric-van.Loon AT KLM DOT COM>
> To:     ADSM-L AT VM.MARIST DOT EDU
> Date:   05/11/2011 02:05 AM
> Subject:        Re: [ADSM-L] TSM Recovery log is pinning since upgrade
> to
> 5.5.5.0 code
> Sent by:        "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
>
>
> Hi TSM-ers!
> Here is a small follow up on my PMR about the recovery log
utilization.
> I'm at TSM level 2, still trying to convince them that there is
> something broken in the TSM server code. To convince them, I have
> changed the logmode to normal on one of my servers. I created a graph
> (through TSMManager) which shows the recovery log utilization during
> last night's client backup window and is doesn't differ much from the
> night before, with logmode rollforward. When running in normal mode,
TSM
> should only use the recovery log for uncommitted transactions, so
> utilization should be very low. My log is 12 Gb and the backuptrigger
> value (75%) was still hit twice!
> This clearly shows that there is something wrong with TSM, let's hope
I
> can convince Level 2 too, so my case gets forwarded to the lab.
> I'll keep you guys posted!
> Kind regards,
> Eric van Loon
> KLM Royal Dutch Airlines
> ********************************************************
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain
> confidential and privileged material intended for the addressee only.
If
> you are not the addressee, you are notified that no part of the e-mail
> or
> any attachment may be disclosed, copied or distributed, and that any
> other
> action related to this e-mail or attachment is strictly prohibited,
and
> may be unlawful. If you have received this e-mail by error, please
> notify
> the sender immediately by return e-mail, and delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
> its
> employees shall not be liable for the incorrect or incomplete
> transmission
> of this e-mail or any attachments, nor responsible for any delay in
> receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with
registered
> number 33014286
> ********************************************************
>
>
>
> U.S. BANCORP made the following annotations
> ---------------------------------------------------------------------
> Electronic Privacy Notice. This e-mail, and any attachments, contains
> information that is, or may be, covered by electronic communications
> privacy laws, and is also confidential and proprietary in nature. If
you
> are not the intended recipient, please be advised that you are legally
> prohibited from retaining, using, copying, distributing, or otherwise
> disclosing this information in any manner. Instead, please reply to
the
> sender that you have received this communication in error, and then
> immediately delete it. Thank you in advance for your cooperation.
>
>
>
> ---------------------------------------------------------------------
> ********************************************************
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain
> confidential and privileged material intended for the addressee only.
If you
> are not the addressee, you are notified that no part of the e-mail or
any
> attachment may be disclosed, copied or distributed, and that any other
> action related to this e-mail or attachment is strictly prohibited,
and may
> be unlawful. If you have received this e-mail by error, please notify
the
> sender immediately by return e-mail, and delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
its
> employees shall not be liable for the incorrect or incomplete
transmission
> of this e-mail or any attachments, nor responsible for any delay in
receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with
registered
> number 33014286
> ********************************************************
>
>
********************************************************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee only. If
you are not the addressee, you are notified that no part of the e-mail
or any attachment may be disclosed, copied or distributed, and that any
other action related to this e-mail or attachment is strictly
prohibited, and may be unlawful. If you have received this e-mail by
error, please notify the sender immediately by return e-mail, and delete
this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
its employees shall not be liable for the incorrect or incomplete
transmission of this e-mail or any attachments, nor responsible for any
delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
Airlines) is registered in Amstelveen, The Netherlands, with registered
number 33014286
********************************************************



Please consider the environment before printing this Email.

CONFIDENTIALITY AND WAIVER: The information contained in this electronic
mail message and any attachments hereto may be legally privileged and
confidential. The information is intended only for the recipient(s)
named in this message. If you are not the intended recipient you are
notified that any use, disclosure, copying or distribution is
prohibited. If you have received this in error please contact the sender
and delete this message and any attachments from your computer system.
We do not guarantee that this message or any attachment to it is secure
or free from errors, computer viruses or other conditions that may
damage or interfere with data, hardware or software.
********************************************************
For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message. 

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt. 
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286
********************************************************
                        

<Prev in Thread] Current Thread [Next in Thread>