Unless you have some kinda "force" command, it will not cancel. Tried
numerous times. A "q sess" shows this. The numbers are misleading since at
one point it was so large (remember, this has been running/hung since
03/07/2011) it has now rolled-over and started again.
2:53:38 PM SUN : q sess
Sess Comm. Sess Wait Bytes Bytes Sess Platform Client Name
Number Method State Time Sent Recvd Type
------- ------ ------ ------ ------- ------- ----- --------
-------------------
55,704 Tcp/Ip Run 0 S 461.4 K 4.5 T Admin Linux/x- ZFORRAY
86_64
55,705 Memory IdleW 111.1 883.8 K 4.9 M Admin Server ZFORRAY
IPC H
2:55:57 PM SUN : q sess 55705 f=d
Sess Number: 55,705
Comm. Method: Memory IPC
Sess State: IdleW
Wait Time: 111.1 H
Bytes Sent: 883.8 K
Bytes Recvd: 4.9 M
Sess Type: Admin
Platform: Server
Client Name: ZFORRAY
Media Access Status:
User Name:
Date/Time First Data Sent:
Proxy By Storage Agent:
Actions:
Yes it is currently happening. Log usage is up to 50% of 60GB. We run DB
backups daily. It has been slowly creeping up.
2:58:00 PM SUN : q log f=d
Total Space(MB): 65,536
Used Space(MB): 34,224
Free Space(MB): 30,992
Active Log Directory: /tsmlog
Archive Log Directory: /tsmarchlog
Mirror Log Directory:
Archive Failover Log Directory:
On Mon, May 2, 2011 at 2:48 PM, Dave Canan <ddcanan AT gmail DOT com> wrote:
> So from this, it looks like session # 55705 would be the one we would want
> to cancel in order to hopefully clear this out. So is the problem currently
> happening? If it is, the next question is whether or not this session is
> going to cancel. Let me know what happens. If it won't cancel let me know.
>
>
> Dave Canan
> IBM ATS TSM Performance
> ddcananATUSDOTIBMDOTCOM
> 916-723-2410
>
> On Mon, May 2, 2011 at 10:27 AM, Zoltan Forray <zforray.vcu AT gmail DOT com
> >wrote:
>
> > Aha - that is why there was so much info. Here is just that piece plus
> the
> > thread for the "parent" and his "parent"
> >
> > Thread 94, Parent 1: AcceptorThread, Storage 77938528, AllocCnt 420814
> > HighWaterAmt 77938528
> > tid=1116109120, ptid=1971136576, det=1, zomb=0, join=0, result=0, sess=0
> >
> > Thread 137116, Parent 94: psSessionThread, Storage 41352, AllocCnt
> 2464414
> > HighWaterAmt 1424944
> > tid=1203480896, ptid=1116109120, det=1, zomb=0, join=0, result=0,
> > sess=55704
> >
> > Thread 137117, Parent 137116: XiSessionThread, Storage 406459533,
> AllocCnt
> > 204215 HighWaterAmt 2147307969
> > tid=1237166400, ptid=1203480896, det=1, zomb=0, join=0, result=0,
> > sess=55705
> > Awaiting cond xiMaster->cSent (0x0x2b787908e870), using mutex
> > xiMaster->mutex
> > (0x0xff84ae8), at xicomm.c(551)
> >
> > On Mon, May 2, 2011 at 1:14 PM, Dave Canan <ddcanan AT gmail DOT com> wrote:
> >
> > > There are no parameters on the "show threads" command; that is the
> > complete
> > > command. When you issue that command, do you see any lines that start
> > with
> > > "Thread 137117" ? That will be the output we are looking for. . Then
> > look
> > > for the "sess=" part of the output from that thread. That is the
> session
> > we
> > > are looking for.
> > >
> > >
> > > Dave Canan
> > > IBM ATS TSM Performance
> > > ddcananATUSDOTIBMDOTCOM
> > > 916-723-2410
> > > >
> > >
> > > On Mon, May 2, 2011 at 9:50 AM, Zoltan Forray <zforray.vcu AT gmail DOT
> > > com>
> > > wrote:
> > >
> > > > Dave,
> > > >
> > > > Thanks for the response.
> > > >
> > > > As I suspect, the oldest thread is from 03/07/2011. I have 2-IMPORT
> > > > processes that hung (one started 03/07/2011 the other 04/19/2011)
> > since
> > > > the
> > > > exporting server had a tape-mount problem and the exports died. I
> have
> > > > long
> > > > since tried to cancel the imports, to no avail. A "q proc" just
> shows
> > > > "cancel in process".
> > > >
> > > > I wasn't sure of the proper syntax for the "show threads" command and
> > > just
> > > > used the ThreadID and used it. In this case:
> > > >
> > > > Start ThreadId=137117, Timestamp=03/07/2011 12:35:37 PM,
> > > > (the oldest)
> > > >
> > > > and did a "show threads 137117"
> > > >
> > > > to get:
> > > >
> > > > -------------------------------------------------
> > > > 12:34:20 PM SUN : show threads 137117
> > > > Server PID: 8520
> > > > Active threads: 126
> > > > Zombie threads: 0
> > > > Cached descriptors: 6
> > > > Thread 1, Parent 0: main, Storage 6203696, AllocCnt 8342 HighWaterAmt
> > > > 6203696
> > > > tid=1971136576, ptid=<none>, det=0, zomb=0, join=0, result=0, sess=0
> > > > Thread 2, Parent 1: TmDeadlockDetector, Storage 0, AllocCnt 0
> > > HighWaterAmt
> > > > 0
> > > > tid=1086736704, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > sess=0
> > > > Thread 3, Parent 1: TbCacheMonitorThread, Storage 0, AllocCnt 2081959
> > > > HighWaterAmt 0
> > > > tid=1099311424, ptid=1971136576, det=0, zomb=0, join=0, result=0,
> > sess=0
> > > > Thread 4, Parent 1: AdmActivityLogThread, Storage 0, AllocCnt
> 20132093
> > > > HighWaterAmt 114297
> > > > tid=1105582400, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > sess=0
> > > > Awaiting cond descP->outReady (0x0x2b7879076ee0), using mutex
> > > OUTV->mutex
> > > > (0x0xb5de3e8), at output.c(3292)
> > > > Thread 13, Parent 1: NaDiscoverThread, Storage 733, AllocCnt 14
> > > > HighWaterAmt
> > > > 37761
> > > > tid=1100364096, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > sess=0
> > > > Awaiting cond NAV->checkConfig (0x0x2b78790770a0), using mutex
> > > NAV->mutex
> > > > (0x0xb7be7b8), at nadiscvr.c(2137)
> > > > Thread 14, Parent 1: RemAgent, Storage 0, AllocCnt 0 HighWaterAmt 0
> > > > tid=1102469440, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > sess=0
> > > > Awaiting cond MMSV->remCond (0x0x2b7879077180), using mutex
> > MMSV->mutex
> > > > (0x0x2aaab0009638), at mmsop.c(1409)
> > > > Thread 15, Parent 1: PollAgent, Storage 0, AllocCnt 0 HighWaterAmt 0
> > > > tid=1101416768, ptid=1971136576, det=1, zomb=0, join=0, result=0,
> > sess=0
> > > >
> > > > (lot of snippage to not anger the listserv gods....................)
> > > >
> > > >
> > > > Awaiting cond descP->outReady (0x0x2b78790900f0), using mutex
> > > OUTV->mutex
> > > > (0x0xb5de3e8), at output.c(3292)
> > > > Thread 517131, Parent 517130: SmAdminCommandThread, Storage 253205,
> > > > AllocCnt
> > > > 4574 HighWaterAmt 321609
> > > > tid=1120319808, ptid=1201375552, det=0, zomb=0, join=0, result=0,
> > sess=0
> > > > Holding mutex THRV->mutex (0x0xb5ea958), acquired at
> pkthread.c(2397)
> > > > Server PID: 8520
> > > > Active threads: 126
> > > > Zombie threads: 0
> > > > Cached descriptors: 6
> > > >
> > > >
> ----------------------------------------------------------------------
> > > >
> > > > I know there are fixes to import in 6.2.2.2 so I am going straight to
> > > > 6.2.2.30
> > > >
> > > > On Mon, May 2, 2011 at 12:26 PM, Dave Canan <ddcanan AT gmail DOT com>
> wrote:
> > > >
> > > > > Zoltan, there is no "show logpinned" command in V6.2.2, but it is
> > still
> > > > > possible to figure out which session and transaction is causing the
> > log
> > > > to
> > > > > be pinned. Here is an example of what you would need to do:
> > > > > This isn't quite as straight-forward as the old show command, but
> > this
> > > > will
> > > > > work. Call or re-post if you have questions.
> > > > >
> > > > >
> > > > > 1. You need to start with doing a "show txnt" command. Here is an
> > > example
> > > > > of
> > > > > one:
> > > > >
> > > > >
> > > > > Session established with server xxxxxxx: AIX
> > > > > Server Version 6, Release 2, Level 2.0
> > > > > Server date/time: 05/02/11 12:09:43 Last access: 05/02/11
> > > 12:09:30
> > > > >
> > > > >
> > > > > tsm: xxxxxxx>show txnt
> > > > >
> > > > > Transaction hash table contents (slots=256):
> > > > > slot -> 38:
> > > > > Tsn=0:688678, Resurrected=False, InFlight=True, Distributed=False,
> > > > > Persistent=True, Addr 113001ab8
> > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06,
> > > > Creator=imdmgr.c(3552)
> > > > > Last known in use by ThreadId=17986
> > > > > Participants=1, summaryVote=ReadOnly
> > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount 0,
> > > > > mustAbort
> > > > > False.
> > > > > Participant DB: voteReceived=False, ackReceived=False
> > > > > DB: Txn 11295a158, ReadOnly(YES), connP=1128ab1f8,
> > > applHandle=23208,
> > > > > openTbls=2:
> > > > > DB: --> OpenP=115facf38 for table=Management.Classes.
> > > > > DB: --> OpenP=116eba978 for table=Management.Class.Ids.
> > > > > DB: --> RegSqlId=0x03000000 SELECT for table=Backup.Objects,
> > > > > executed(Yes).
> > > > > slot -> 39:
> > > > > Tsn=0:688679, Resurrected=False, InFlight=True, Distributed=False,
> > > > > Persistent=True, Addr 115646a78
> > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06,
> > > > Creator=imdmgr.c(3673)
> > > > > Last known in use by ThreadId=17986
> > > > > Participants=2, summaryVote=ReadOnly
> > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount 0,
> > > > > mustAbort
> > > > > False.
> > > > > Participant DB: voteReceived=False, ackReceived=False
> > > > > DB: Txn 112f855b8, ReadOnly(NO), connP=112ef88f8,
> > > applHandle=23205,
> > > > > openTbls=24:
> > > > > DB: --> OpenP=1147eded8 for table=Activity.Summary.
> > > > > DB: --> OpenP=118b7e038 for table=Group.Objects.ByLeader.
> > > > > DB: --> OpenP=112fbdcf8 for table=Backup.Objects.
> > > > > DB: --> OpenP=11300d3d8 for table=Group.Leaders.
> > > > > DB: --> OpenP=11730b3b8 for table=AS.Volume.Status.
> > > > > DB: --> OpenP=11307a7b8 for table=AF.Vol.Clusters.
> > > > > DB: --> OpenP=11743acd8 for table=AF.Clusters.
> > > > > DB: --> OpenP=117335198 for table=BF.Dereferenced.Chunks.
> > > > > DB: --> OpenP=112fa2bb8 for table=DF.Segments.
> > > > > DB: --> OpenP=113084658 for table=AS.Segments.
> > > > > DB: --> OpenP=1160b65b8 for table=AF.Damaged.
> > > > > DB: --> OpenP=112fe8b78 for table=BF.Bitfile.Extents.
> > > > > DB: --> OpenP=115641438 for table=AF.Segments.
> > > > > DB: --> OpenP=115f7d578 for table=AF.Bitfiles.
> > > > > DB: --> OpenP=114536ad8 for table=DF.Bitfiles.
> > > > > DB: --> OpenP=11303ff78 for table=BF.Aggregate.Attributes.
> > > > > DB: --> OpenP=118c62eb8 for table=BF.Aggregated.Bitfiles.
> > > > > DB: --> OpenP=1170581d8 for table=ARCH_EXPDIRS.
> > > > > DB: --> OpenP=1147ec498 for table=Archive.Objects.
> > > > > DB: --> OpenP=11604ddb8 for table=Management.Class.Ids.
> > > > > DB: --> OpenP=116e91c98 for table=Management.Classes.
> > > > > DB: --> OpenP=113034e98 for table=Filespaces.
> > > > > DB: --> OpenP=1144d3e38 for table=Nodes.
> > > > > DB: --> OpenP=114d1ce58 for table=Policy.Domain.Members.
> > > > > Participant IM: voteReceived=False, ackReceived=False
> > > > > Locks held by Tsn=0:688679 :
> > > > > Type=19002(im node filespace), NameSpace=0, SummMode=xLock,
> > > > Mode=xLock,
> > > > > Key='154.1'
> > > > > Type=19001(im node), NameSpace=0, SummMode=ixLock, Mode=ixLock,
> > > > > Key='154'
> > > > > Type=19000(im universe), NameSpace=0, SummMode=ixLock,
> > Mode=ixLock,
> > > > > Key=''
> > > > > slot -> 165:
> > > > > Tsn=0:692901, Resurrected=False, InFlight=True, Distributed=False,
> > > > > Persistent=True, Addr 114532178
> > > > > Start ThreadId=17984, Timestamp=05/02/11 12:10:07,
> > > > Creator=imdmgr.c(2460)
> > > > > Last known in use by ThreadId=17984
> > > > > Participants=1, summaryVote=ReadOnly
> > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount 0,
> > > > > mustAbort
> > > > > False.
> > > > >
> > > > > 2. What you're going to want to do is look at the lines that state
> > > "Start
> > > > > Threadid=nnnnn, Timestamp=". You need to find the oldest Timestamp.
> > > That
> > > > > will also give you the threadid. From there you do a "show threads"
> > > > > command.
> > > > > Here is a piece from that command :
> > > > >
> > > > > Thread 17980, Parent 42: psSessionThread, Storage 0, AllocCnt
> > 490
> > > > > HighWaterAmt
> > > > > 174080
> > > > > tid=253c, ptid=1e2a, det=1, zomb=0, join=0, result=0, sess=4612
> > > > > Holding mutex OUTV->mutex (0x112018018), acquired at
> output.c(3267)
> > > > >
> > > > > Stack trace:
> > > > > 0x0900000000595f30 _cond_wait_global
> > > > > 0x0900000000596abc _cond_wait
> > > > > 0x09000000005977ac pthread_cond_wait
> > > > > 0x0000000100008414 pkWaitConditionTracked
> > > > > 0x00000001000130f8 outGetNext
> > > > > 0x0000000100870e58 SmAdminOutput
> > > > > 0x000000010086ffb8 SmExecuteAdminCommand
> > > > > 0x000000010086f888 SmAdminSession
> > > > > 0x00000001003f123c DoAdminGeneral
> > > > > 0x00000001003ec9bc smExecuteSession
> > > > > 0x000000010009ad48 psSessionThread
> > > > > 0x000000010001b688 StartThread
> > > > >
> > > > >
> > > > > 3. Look for the thread Id for the longest running transaction in
> the
> > > > first
> > > > > step. This will also point you to the session matching that thread.
> > > From
> > > > > there, the session (that is the longest running transaction), can
> be
> > > > > canceled.
> > > > >
> > > > >
> > > > > Dave Canan
> > > > > IBM ATS TSM Performance
> > > > > ddcananATUSDOTIBMDOTCOM
> > > > > 916-723-2410
> > > > >
> > > > >
> > > > > On Fri, Apr 29, 2011 at 5:50 AM, Zoltan Forray <
> > zforray.vcu AT gmail DOT com
> > > > > >wrote:
> > > > >
> > > > > > Yes, I realize 6.x is DB2 but I was wondering is there is an
> > > equivalent
> > > > > to
> > > > > > "show logpinned". I seem to be in some kind of situation where
> the
> > > logs
> > > > > are
> > > > > > not being released and growing daily, eventhough I have performed
> > > > > numerous
> > > > > > full DB backups and backup volhist.
> > > > > >
> > > > > > I am going to have to bounce it soon, if I can't kill whatever is
> > > > holding
> > > > > > things up.
> > > > > >
> > > > > > My suspicion is a hung import. I have been moving numerous nodes
> > > from
> > > > an
> > > > > > old 5.5 server and along the way at least 2-export/imports have
> > > crashed
> > > > > for
> > > > > > various reasons and left zombie imports on the 6.2.2 server (yes
> I
> > > > tried
> > > > > > cancel proc with no results other than "cancel in progress")
> > > > > >
> > > > > > --
> > > > > > Zoltan Forray
> > > > > > TSM Software & Hardware Administrator
> > > > > > Virginia Commonwealth University
> > > > > > UCC/Office of Technology Services
> > > > > > zforray AT vcu DOT edu - 804-828-4807
> > > > > > Don't be a phishing victim - VCU and other reputable
> organizations
> > > will
> > > > > > never use email to request that you reply with your password,
> > social
> > > > > > security number or confidential personal information. For more
> > > details
> > > > > > visit
> > > > > > http://infosecurity.vcu.edu/phishing.html
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Zoltan Forray
> > > > TSM Software & Hardware Administrator
> > > > Virginia Commonwealth University
> > > > UCC/Office of Technology Services
> > > > zforray AT vcu DOT edu - 804-828-4807
> > > > Don't be a phishing victim - VCU and other reputable organizations
> will
> > > > never use email to request that you reply with your password, social
> > > > security number or confidential personal information. For more
> details
> > > > visit
> > > > http://infosecurity.vcu.edu/phishing.html
> > > >
> > >
> >
> >
> >
> > --
> > Zoltan Forray
> > TSM Software & Hardware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > zforray AT vcu DOT edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit
> > http://infosecurity.vcu.edu/phishing.html
> >
>
--
Zoltan Forray
TSM Software & Hardware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details visit
http://infosecurity.vcu.edu/phishing.html
|
