ADSM-L

Re: [ADSM-L] sql tdp hang issue

2007-12-14 11:17:35
Subject: Re: [ADSM-L] sql tdp hang issue
From: Fred Johanson <Fred AT UCHICAGO DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Fri, 14 Dec 2007 09:55:31 -0600
This sounds like PMR#64846-122, which is still open after months of
work.  The last thing in the traces we ran is a TCPFLUSH.  After that
things hang forever.

Fred Johanson
TSM Administrator
University of Chicago

773-702-8464

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Park, Rod
Sent: Friday, December 14, 2007 9:50 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] sql tdp hang issue

I don't see anything in the activity log. The backups are going straight
to diskpools and the backups are single threaded. I did add disk space
to my diskpool this week in case that was the issue. Looking at my
diskvols, data doesn't seem to be striped across all disk evenly, would
that be that problem if some fill up and others don't, causing it to
wait on tsm?

tsm: TSMCORP1>q vol stg=sql_diskpool

Volume Name                  Storage         Device         Estimated
Pct      Volume 
                             Pool Name       Class Name      Capacity
Util      Status 
------------------------     -----------     ----------     ---------
-----     --------
/dev/rsqldiskvol1            SQL_DISKPO-     DISK            51,072.0
0.2     On-Line 
                              OL

/dev/rsqldiskvol10           SQL_DISKPO-     DISK            51,072.0
31.4     On-Line 
                              OL

/dev/rsqldiskvol11           SQL_DISKPO-     DISK            51,072.0
100.0     On-Line 
                              OL

/dev/rsqldiskvol12           SQL_DISKPO-     DISK            51,072.0
36.6     On-Line 
                              OL

/dev/rsqldiskvol13           SQL_DISKPO-     DISK            51,072.0
100.0     On-Line 
                              OL

/dev/rsqldiskvol14           SQL_DISKPO-     DISK            51,072.0
92.8     On-Line 
                              OL

/dev/rsqldiskvol15           SQL_DISKPO-     DISK            51,072.0
98.8     On-Line 
                              OL

/dev/rsqldiskvol16           SQL_DISKPO-     DISK            51,072.0
81.9     On-Line 
                              OL

/dev/rsqldiskvol2            SQL_DISKPO-     DISK            51,072.0
3.2     On-Line 
                              OL

/dev/rsqldiskvol3            SQL_DISKPO-     DISK            51,072.0
91.3     On-Line 
                              OL

/dev/rsqldiskvol4            SQL_DISKPO-     DISK            51,072.0
21.0     On-Line 
                              OL

/dev/rsqldiskvol5            SQL_DISKPO-     DISK            51,072.0
11.5     On-Line 
                              OL

/dev/rsqldiskvol6            SQL_DISKPO-     DISK            51,072.0
69.4     On-Line 
                              OL

/dev/rsqldiskvol7            SQL_DISKPO-     DISK            51,072.0
15.4     On-Line 
                              OL

/dev/rsqldiskvol8            SQL_DISKPO-     DISK            51,072.0
99.9     On-Line 
                              OL

/dev/rsqldiskvol9            SQL_DISKPO-     DISK            51,072.0
44.3     On-Line

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Del Hoobler
Sent: Friday, December 14, 2007 9:45 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] sql tdp hang issue

Did you examine the TSM Server activity log closely?
Was there a problem where a tape did not get mounted in time?
How many "stripes" are you using for the backup?
If you are using multiple stripes and one of those stripes
is waiting for the TSM server to mount a tape, it can cause this.
I have seen this when the TSM Server does not have enough available
tapes mounted to satisfy the DP/SQL request.
The "Waiting on TSM server...." indicates that DP/SQL is waiting
for some event on the TSM Server to occur before it can continue
sending any more data. Is it possible that some TSM Server event
pre-empted the DP/SQL backup and took one of the tapes that was
needed to satisfy a multiple stripe DP/SQL backup request?

You should start on the TSM Server... in the activity log to
find out what it thinks happened.

Thanks,

Del

----------------------------------------------------

"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 12/14/2007
09:52:59 AM:

> I have a dba that has one sql server that has "tsm hang" issues
> according to him. They have a full backup job that backs up 7 sql
db's.
> The size is about 80GB total. Sometime it backs them all up with no
> issues, sometime it says "waiting on tsm server" but if I look at "q
se"
> there are no active sessions. Right now there is one running from last
> night and it has only backed up 12.5GB in 9 hours. below are some
client
> details and log details. Anyone have any ideas. Other sql backups run
> just fine along with the other 15TB that backup each night, but of
> course DBA's say it's a TSM issue.
> Here is what they see when it hangs:
>
> Full: 3   Read: 658505728  Written: 655360000  Rate: 480.95 Kb/Sec
> Full: 3   Read: 658505728  Written: 655360000  Rate: 480.59 Kb/Sec
> Full: 3   Read: 658505728  Written: 655360000  Rate: 480.23 Kb/Sec
> Full: 3   Read: 658505728  Written: 655360000  Rate: 479.87 Kb/Sec
>
> Waiting for TSM
>
server..................................................................
> ......................
>
> clint info:
>
> Node Name: WHQWPSQLCL1RS01_SQL
>                       Platform: TDP MSSQL Win32
>                Client OS Level: 5.02
>                 Client Version: Version 5, Release 3, Level 3.0
>             Policy Domain Name: SQL
>          Last Access Date/Time: 12/14/07   08:00:05
>         Days Since Last Access: <1
>
>
> thanks,
> Rod

This email and any files transmitted with it are confidential and
intended solely for the use of the addressee. If you are not the
intended addressee, then you have received this email in error and any
use, dissemination, forwarding, printing, or copying of this email is
strictly prohibited. Please notify us immediately of your unintended
receipt by reply and then delete this email and your reply. Tyson Foods,
Inc. and its subsidiaries and affiliates will not be held liable to any
person resulting from the unintended or unauthorized use of any
information contained in this email or as a result of any additions or
deletions of information originally contained in this email.