Zoltan, I need to look at the complete output from a "show threads" command.
Rather than filling up the listserv with stuff, send it to my work ID. (
ddcanan AT us.ibm DOT com). We need to figure out what is hanging up the import
from canceling.
Dave Canan
IBM ATS TSM Performance
ddcananATUSDOTIBMDOTCOM
916-723-2410
>
On Mon, May 2, 2011 at 12:01 PM, Zoltan Forray <zforray.vcu AT gmail DOT
com>wrote:
> Unless you have some kinda "force" command, it will not cancel. Tried
> numerous times. A "q sess" shows this. The numbers are misleading since
> at
> one point it was so large (remember, this has been running/hung since
> 03/07/2011) it has now rolled-over and started again.
>
> 2:53:38 PM SUN : q sess
> Sess Comm. Sess Wait Bytes Bytes Sess Platform Client Name
>
> Number Method State Time Sent Recvd Type
> ------- ------ ------ ------ ------- ------- ----- --------
> -------------------
> 55,704 Tcp/Ip Run 0 S 461.4 K 4.5 T Admin Linux/x- ZFORRAY
>
> 86_64
>
> 55,705 Memory IdleW 111.1 883.8 K 4.9 M Admin Server ZFORRAY
>
> IPC H
>
>
> 2:55:57 PM SUN : q sess 55705 f=d
> Sess Number: 55,705
> Comm. Method: Memory IPC
> Sess State: IdleW
> Wait Time: 111.1 H
> Bytes Sent: 883.8 K
> Bytes Recvd: 4.9 M
> Sess Type: Admin
> Platform: Server
> Client Name: ZFORRAY
> Media Access Status:
> User Name:
> Date/Time First Data Sent:
> Proxy By Storage Agent:
> Actions:
>
> Yes it is currently happening. Log usage is up to 50% of 60GB. We run DB
> backups daily. It has been slowly creeping up.
>
> 2:58:00 PM SUN : q log f=d
> Total Space(MB): 65,536
> Used Space(MB): 34,224
> Free Space(MB): 30,992
> Active Log Directory: /tsmlog
> Archive Log Directory: /tsmarchlog
> Mirror Log Directory:
> Archive Failover Log Directory:
>
>
>
> On Mon, May 2, 2011 at 2:48 PM, Dave Canan <ddcanan AT gmail DOT com> wrote:
>
> > So from this, it looks like session # 55705 would be the one we would
> want
> > to cancel in order to hopefully clear this out. So is the problem
> currently
> > happening? If it is, the next question is whether or not this session is
> > going to cancel. Let me know what happens. If it won't cancel let me
> know.
> >
> >
> > Dave Canan
> > IBM ATS TSM Performance
> > ddcananATUSDOTIBMDOTCOM
> > 916-723-2410
> >
> > On Mon, May 2, 2011 at 10:27 AM, Zoltan Forray <zforray.vcu AT gmail DOT com
> > >wrote:
> >
> > > Aha - that is why there was so much info. Here is just that piece plus
> > the
> > > thread for the "parent" and his "parent"
> > >
> > > Thread 94, Parent 1: AcceptorThread, Storage 77938528, AllocCnt 420814
> > > HighWaterAmt 77938528
> > > tid=1116109120, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> sess=0
> > >
> > > Thread 137116, Parent 94: psSessionThread, Storage 41352, AllocCnt
> > 2464414
> > > HighWaterAmt 1424944
> > > tid=1203480896, ptid=1116109120, det=1, zomb=0, join=0, result=0,
> > > sess=55704
> > >
> > > Thread 137117, Parent 137116: XiSessionThread, Storage 406459533,
> > AllocCnt
> > > 204215 HighWaterAmt 2147307969
> > > tid=1237166400, ptid=1203480896, det=1, zomb=0, join=0, result=0,
> > > sess=55705
> > > Awaiting cond xiMaster->cSent (0x0x2b787908e870), using mutex
> > > xiMaster->mutex
> > > (0x0xff84ae8), at xicomm.c(551)
> > >
> > > On Mon, May 2, 2011 at 1:14 PM, Dave Canan <ddcanan AT gmail DOT com>
> > > wrote:
> > >
> > > > There are no parameters on the "show threads" command; that is the
> > > complete
> > > > command. When you issue that command, do you see any lines that start
> > > with
> > > > "Thread 137117" ? That will be the output we are looking for. . Then
> > > look
> > > > for the "sess=" part of the output from that thread. That is the
> > session
> > > we
> > > > are looking for.
> > > >
> > > >
> > > > Dave Canan
> > > > IBM ATS TSM Performance
> > > > ddcananATUSDOTIBMDOTCOM
> > > > 916-723-2410
> > > > >
> > > >
> > > > On Mon, May 2, 2011 at 9:50 AM, Zoltan Forray <zforray.vcu AT gmail DOT
> > > > com
> >
> > > > wrote:
> > > >
> > > > > Dave,
> > > > >
> > > > > Thanks for the response.
> > > > >
> > > > > As I suspect, the oldest thread is from 03/07/2011. I have
> 2-IMPORT
> > > > > processes that hung (one started 03/07/2011 the other 04/19/2011)
> > > since
> > > > > the
> > > > > exporting server had a tape-mount problem and the exports died. I
> > have
> > > > > long
> > > > > since tried to cancel the imports, to no avail. A "q proc" just
> > shows
> > > > > "cancel in process".
> > > > >
> > > > > I wasn't sure of the proper syntax for the "show threads" command
> and
> > > > just
> > > > > used the ThreadID and used it. In this case:
> > > > >
> > > > > Start ThreadId=137117, Timestamp=03/07/2011 12:35:37 PM,
> > > > > (the oldest)
> > > > >
> > > > > and did a "show threads 137117"
> > > > >
> > > > > to get:
> > > > >
> > > > > -------------------------------------------------
> > > > > 12:34:20 PM SUN : show threads 137117
> > > > > Server PID: 8520
> > > > > Active threads: 126
> > > > > Zombie threads: 0
> > > > > Cached descriptors: 6
> > > > > Thread 1, Parent 0: main, Storage 6203696, AllocCnt 8342
> HighWaterAmt
> > > > > 6203696
> > > > > tid=1971136576, ptid=<none>, det=0, zomb=0, join=0, result=0,
> sess=0
> > > > > Thread 2, Parent 1: TmDeadlockDetector, Storage 0, AllocCnt 0
> > > > HighWaterAmt
> > > > > 0
> > > > > tid=1086736704, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Thread 3, Parent 1: TbCacheMonitorThread, Storage 0, AllocCnt
> 2081959
> > > > > HighWaterAmt 0
> > > > > tid=1099311424, ptid=1971136576, det=0, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Thread 4, Parent 1: AdmActivityLogThread, Storage 0, AllocCnt
> > 20132093
> > > > > HighWaterAmt 114297
> > > > > tid=1105582400, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Awaiting cond descP->outReady (0x0x2b7879076ee0), using mutex
> > > > OUTV->mutex
> > > > > (0x0xb5de3e8), at output.c(3292)
> > > > > Thread 13, Parent 1: NaDiscoverThread, Storage 733, AllocCnt 14
> > > > > HighWaterAmt
> > > > > 37761
> > > > > tid=1100364096, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Awaiting cond NAV->checkConfig (0x0x2b78790770a0), using mutex
> > > > NAV->mutex
> > > > > (0x0xb7be7b8), at nadiscvr.c(2137)
> > > > > Thread 14, Parent 1: RemAgent, Storage 0, AllocCnt 0 HighWaterAmt 0
> > > > > tid=1102469440, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Awaiting cond MMSV->remCond (0x0x2b7879077180), using mutex
> > > MMSV->mutex
> > > > > (0x0x2aaab0009638), at mmsop.c(1409)
> > > > > Thread 15, Parent 1: PollAgent, Storage 0, AllocCnt 0 HighWaterAmt
> 0
> > > > > tid=1101416768, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > > sess=0
> > > > >
> > > > > (lot of snippage to not anger the listserv
> gods....................)
> > > > >
> > > > >
> > > > > Awaiting cond descP->outReady (0x0x2b78790900f0), using mutex
> > > > OUTV->mutex
> > > > > (0x0xb5de3e8), at output.c(3292)
> > > > > Thread 517131, Parent 517130: SmAdminCommandThread, Storage 253205,
> > > > > AllocCnt
> > > > > 4574 HighWaterAmt 321609
> > > > > tid=1120319808, ptid=1201375552, det=0, zomb=0, join=0, result=0,
> > > sess=0
> > > > > Holding mutex THRV->mutex (0x0xb5ea958), acquired at
> > pkthread.c(2397)
> > > > > Server PID: 8520
> > > > > Active threads: 126
> > > > > Zombie threads: 0
> > > > > Cached descriptors: 6
> > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > >
> > > > > I know there are fixes to import in 6.2.2.2 so I am going straight
> to
> > > > > 6.2.2.30
> > > > >
> > > > > On Mon, May 2, 2011 at 12:26 PM, Dave Canan <ddcanan AT gmail DOT com>
> > wrote:
> > > > >
> > > > > > Zoltan, there is no "show logpinned" command in V6.2.2, but it is
> > > still
> > > > > > possible to figure out which session and transaction is causing
> the
> > > log
> > > > > to
> > > > > > be pinned. Here is an example of what you would need to do:
> > > > > > This isn't quite as straight-forward as the old show command, but
> > > this
> > > > > will
> > > > > > work. Call or re-post if you have questions.
> > > > > >
> > > > > >
> > > > > > 1. You need to start with doing a "show txnt" command. Here is an
> > > > example
> > > > > > of
> > > > > > one:
> > > > > >
> > > > > >
> > > > > > Session established with server xxxxxxx: AIX
> > > > > > Server Version 6, Release 2, Level 2.0
> > > > > > Server date/time: 05/02/11 12:09:43 Last access: 05/02/11
> > > > 12:09:30
> > > > > >
> > > > > >
> > > > > > tsm: xxxxxxx>show txnt
> > > > > >
> > > > > > Transaction hash table contents (slots=256):
> > > > > > slot -> 38:
> > > > > > Tsn=0:688678, Resurrected=False, InFlight=True,
> Distributed=False,
> > > > > > Persistent=True, Addr 113001ab8
> > > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06,
> > > > > Creator=imdmgr.c(3552)
> > > > > > Last known in use by ThreadId=17986
> > > > > > Participants=1, summaryVote=ReadOnly
> > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount
> 0,
> > > > > > mustAbort
> > > > > > False.
> > > > > > Participant DB: voteReceived=False, ackReceived=False
> > > > > > DB: Txn 11295a158, ReadOnly(YES), connP=1128ab1f8,
> > > > applHandle=23208,
> > > > > > openTbls=2:
> > > > > > DB: --> OpenP=115facf38 for table=Management.Classes.
> > > > > > DB: --> OpenP=116eba978 for table=Management.Class.Ids.
> > > > > > DB: --> RegSqlId=0x03000000 SELECT for table=Backup.Objects,
> > > > > > executed(Yes).
> > > > > > slot -> 39:
> > > > > > Tsn=0:688679, Resurrected=False, InFlight=True,
> Distributed=False,
> > > > > > Persistent=True, Addr 115646a78
> > > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06,
> > > > > Creator=imdmgr.c(3673)
> > > > > > Last known in use by ThreadId=17986
> > > > > > Participants=2, summaryVote=ReadOnly
> > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount
> 0,
> > > > > > mustAbort
> > > > > > False.
> > > > > > Participant DB: voteReceived=False, ackReceived=False
> > > > > > DB: Txn 112f855b8, ReadOnly(NO), connP=112ef88f8,
> > > > applHandle=23205,
> > > > > > openTbls=24:
> > > > > > DB: --> OpenP=1147eded8 for table=Activity.Summary.
> > > > > > DB: --> OpenP=118b7e038 for table=Group.Objects.ByLeader.
> > > > > > DB: --> OpenP=112fbdcf8 for table=Backup.Objects.
> > > > > > DB: --> OpenP=11300d3d8 for table=Group.Leaders.
> > > > > > DB: --> OpenP=11730b3b8 for table=AS.Volume.Status.
> > > > > > DB: --> OpenP=11307a7b8 for table=AF.Vol.Clusters.
> > > > > > DB: --> OpenP=11743acd8 for table=AF.Clusters.
> > > > > > DB: --> OpenP=117335198 for table=BF.Dereferenced.Chunks.
> > > > > > DB: --> OpenP=112fa2bb8 for table=DF.Segments.
> > > > > > DB: --> OpenP=113084658 for table=AS.Segments.
> > > > > > DB: --> OpenP=1160b65b8 for table=AF.Damaged.
> > > > > > DB: --> OpenP=112fe8b78 for table=BF.Bitfile.Extents.
> > > > > > DB: --> OpenP=115641438 for table=AF.Segments.
> > > > > > DB: --> OpenP=115f7d578 for table=AF.Bitfiles.
> > > > > > DB: --> OpenP=114536ad8 for table=DF.Bitfiles.
> > > > > > DB: --> OpenP=11303ff78 for table=BF.Aggregate.Attributes.
> > > > > > DB: --> OpenP=118c62eb8 for table=BF.Aggregated.Bitfiles.
> > > > > > DB: --> OpenP=1170581d8 for table=ARCH_EXPDIRS.
> > > > > > DB: --> OpenP=1147ec498 for table=Archive.Objects.
> > > > > > DB: --> OpenP=11604ddb8 for table=Management.Class.Ids.
> > > > > > DB: --> OpenP=116e91c98 for table=Management.Classes.
> > > > > > DB: --> OpenP=113034e98 for table=Filespaces.
> > > > > > DB: --> OpenP=1144d3e38 for table=Nodes.
> > > > > > DB: --> OpenP=114d1ce58 for table=Policy.Domain.Members.
> > > > > > Participant IM: voteReceived=False, ackReceived=False
> > > > > > Locks held by Tsn=0:688679 :
> > > > > > Type=19002(im node filespace), NameSpace=0, SummMode=xLock,
> > > > > Mode=xLock,
> > > > > > Key='154.1'
> > > > > > Type=19001(im node), NameSpace=0, SummMode=ixLock,
> Mode=ixLock,
> > > > > > Key='154'
> > > > > > Type=19000(im universe), NameSpace=0, SummMode=ixLock,
> > > Mode=ixLock,
> > > > > > Key=''
> > > > > > slot -> 165:
> > > > > > Tsn=0:692901, Resurrected=False, InFlight=True,
> Distributed=False,
> > > > > > Persistent=True, Addr 114532178
> > > > > > Start ThreadId=17984, Timestamp=05/02/11 12:10:07,
> > > > > Creator=imdmgr.c(2460)
> > > > > > Last known in use by ThreadId=17984
> > > > > > Participants=1, summaryVote=ReadOnly
> > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount
> 0,
> > > > > > mustAbort
> > > > > > False.
> > > > > >
> > > > > > 2. What you're going to want to do is look at the lines that
> state
> > > > "Start
> > > > > > Threadid=nnnnn, Timestamp=". You need to find the oldest
> Timestamp.
> > > > That
> > > > > > will also give you the threadid. From there you do a "show
> threads"
> > > > > > command.
> > > > > > Here is a piece from that command :
> > > > > >
> > > > > > Thread 17980, Parent 42: psSessionThread, Storage 0,
> AllocCnt
> > > 490
> > > > > > HighWaterAmt
> > > > > > 174080
> > > > > > tid=253c, ptid=1e2a, det=1, zomb=0, join=0, result=0, sess=4612
> > > > > > Holding mutex OUTV->mutex (0x112018018), acquired at
> > output.c(3267)
> > > > > >
> > > > > > Stack trace:
> > > > > > 0x0900000000595f30 _cond_wait_global
> > > > > > 0x0900000000596abc _cond_wait
> > > > > > 0x09000000005977ac pthread_cond_wait
> > > > > > 0x0000000100008414 pkWaitConditionTracked
> > > > > > 0x00000001000130f8 outGetNext
> > > > > > 0x0000000100870e58 SmAdminOutput
> > > > > > 0x000000010086ffb8 SmExecuteAdminCommand
> > > > > > 0x000000010086f888 SmAdminSession
> > > > > > 0x00000001003f123c DoAdminGeneral
> > > > > > 0x00000001003ec9bc smExecuteSession
> > > > > > 0x000000010009ad48 psSessionThread
> > > > > > 0x000000010001b688 StartThread
> > > > > >
> > > > > >
> > > > > > 3. Look for the thread Id for the longest running transaction in
> > the
> > > > > first
> > > > > > step. This will also point you to the session matching that
> thread.
> > > > From
> > > > > > there, the session (that is the longest running transaction), can
> > be
> > > > > > canceled.
> > > > > >
> > > > > >
> > > > > > Dave Canan
> > > > > > IBM ATS TSM Performance
> > > > > > ddcananATUSDOTIBMDOTCOM
> > > > > > 916-723-2410
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 29, 2011 at 5:50 AM, Zoltan Forray <
> > > zforray.vcu AT gmail DOT com
> > > > > > >wrote:
> > > > > >
> > > > > > > Yes, I realize 6.x is DB2 but I was wondering is there is an
> > > > equivalent
> > > > > > to
> > > > > > > "show logpinned". I seem to be in some kind of situation where
> > the
> > > > logs
> > > > > > are
> > > > > > > not being released and growing daily, eventhough I have
> performed
> > > > > > numerous
> > > > > > > full DB backups and backup volhist.
> > > > > > >
> > > > > > > I am going to have to bounce it soon, if I can't kill whatever
> is
> > > > > holding
> > > > > > > things up.
> > > > > > >
> > > > > > > My suspicion is a hung import. I have been moving numerous
> nodes
> > > > from
> > > > > an
> > > > > > > old 5.5 server and along the way at least 2-export/imports have
> > > > crashed
> > > > > > for
> > > > > > > various reasons and left zombie imports on the 6.2.2 server
> (yes
> > I
> > > > > tried
> > > > > > > cancel proc with no results other than "cancel in progress")
> > > > > > >
> > > > > > > --
> > > > > > > Zoltan Forray
> > > > > > > TSM Software & Hardware Administrator
> > > > > > > Virginia Commonwealth University
> > > > > > > UCC/Office of Technology Services
> > > > > > > zforray AT vcu DOT edu - 804-828-4807
> > > > > > > Don't be a phishing victim - VCU and other reputable
> > organizations
> > > > will
> > > > > > > never use email to request that you reply with your password,
> > > social
> > > > > > > security number or confidential personal information. For more
> > > > details
> > > > > > > visit
> > > > > > > http://infosecurity.vcu.edu/phishing.html
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Zoltan Forray
> > > > > TSM Software & Hardware Administrator
> > > > > Virginia Commonwealth University
> > > > > UCC/Office of Technology Services
> > > > > zforray AT vcu DOT edu - 804-828-4807
> > > > > Don't be a phishing victim - VCU and other reputable organizations
> > will
> > > > > never use email to request that you reply with your password,
> social
> > > > > security number or confidential personal information. For more
> > details
> > > > > visit
> > > > > http://infosecurity.vcu.edu/phishing.html
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Zoltan Forray
> > > TSM Software & Hardware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > zforray AT vcu DOT edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more details
> > > visit
> > > http://infosecurity.vcu.edu/phishing.html
> > >
> >
>
>
>
> --
> Zoltan Forray
> TSM Software & Hardware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit
> http://infosecurity.vcu.edu/phishing.html
>
|
