Expiration getting hung at the end. Doing manual cancel expiration.

[mdfazu1984]

I'm getting below error from db2diag. could someone guide me how to fix this expiration and clear below error log.


OSERR : ENOSPC (28) "No space left on device"
DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
File Handle = 248
File System Block Size = 4096 bytes
File System Type = jfs2
File Handle Flags :
Require Sector Align = No
DIO/CIO Mode = Yes
Raw Block Device = No
Reserved Handle = No
Flush On Close = Yes
Thread-Level Lock = Yes
Write-through Mode = No
File Not Tracked = No
DATA #2 : unsigned integer, 8 bytes
134217728
DATA #3 : signed integer, 8 bytes
110968832
DATA #4 : String, 105 bytes
Search for ossError*Analysis probe point after this log entry for further
self-diagnosis of this problem.

 

[Trident]

Hi,

OSERR : ENOSPC (28) "No space left on device"

Do you have enough free space?

 

[mdfazu1984]

Hi,

OSERR : ENOSPC (28) "No space left on device"

Do you have enough free space?

where ?

tsm: TSMSERV>q log f=d

Active Log Directory: /activelog/tsmactlog
Total Space(MB): 128,000
Used Space(MB): 1,808
Free Space(MB): 126,192
Total Space of File System (MB): 128,000.00
Used Space on File System (MB): 127,510.18
Free Space on File System (MB): 489.82
Archive Log Directory: /archivelog/tsmarchlog
Total Space of File System (MB): 204,800.00
Used Space on File System (MB): 81,234.62
Free Space on File System (MB): 123,565.38
Archive Log Compressed: No
Mirror Log Directory:
Total Space of File System (MB):
Used Space on File System (MB):
Free Space on File System (MB):
Archive Failover Log Directory: /archlog-failover/tsmarch_fail
Total Space of File System (MB): 153,600.00
Used Space on File System (MB): 535.96
Free Space on File System (MB): 153,064.04


tsm: TSMSERV>q db f=d

Database Name: TSMDB1
Total Space of File System (MB): 816,128
Space Used on File System(MB): 609,482
Space Used by Database(MB): 609,335
Free Space Available (MB): 206,646
Total Pages: 41,120,543
Usable Pages: 41,118,491
Used Pages: 40,398,035
Free Pages: 720,456
Buffer Pool Hit Ratio: 97.5
Total Buffer Requests: 19,912,169,367
Sort Overflows: 0
Package Cache Hit Ratio: 99.9
Last Database Reorganization: 05/12/20 19:15:03
Full Device Class Name: C3583
Number of Database Backup Streams: 1
Incrementals Since Last Full: 0
Last Complete Backup Date/Time: 05/13/20 23:00:09
Compress Database Backups: No
Protect Master Encryption Key: No

 

[Trident]

Hi,

You shuld have more space in active log.

https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.5/srv.admin/t_adm_invcap_mng.html

Try adding 20 Gb to that file system.

 

[mdfazu1984]

in show tread im getting below log for expiration.. is any thing related for expiration hung


Thread context:
COPY_TYPE: ARCHIVE
COMMAND: EXPIRE INVENTORY
SCHEDULE_TYPE: ADMIN
SCHEDULE_NAME: EXPIRING_INVENTORY
THREAD_TYPE: PROCESS
PROCESS_DESC: EXPIRE INVENTORY
PROCESS_NUMBER: 1167
DOMAIN_NAME: WIN_DOM
SCHEDULED: YES
NODE_NAME:
FILESPACE_NAME: SQJIRAP01\SystemState\NULL\System State\SystemState

Thread 75499, Parent 75494: ExpirationProcessThread, Storage 226967, AllocCnt 3308886 HighWaterAmt 2078143
tid=2b3eb, ptid=299e6, det=1, zomb=0, join=0, result=0, sess=0, procToken=1167, sessToken=29336
Stack trace:
0x09000000001fb000 semop
0x090000001cd051e8 sqloSSemP
0x090000001cd04d08 sqlccipcrecv__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
0x090000001cd0460c sqlccrecv
0x090000001cd047f0 sqljcReceive__FP10sqljCmnMgr
0x090000001cd16c60 sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
0x090000001cce8680 CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001c7ecd04 SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cb8786c SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cce52fc SQLExecute
0x000000010019de80 tbRegExecEx
0x0000000100aabe1c IPRA.$ImDeleteDependents
0x0000000100aab7e0 IPRA.$ImDeleteDependents
0x0000000100aab7e0 IPRA.$ImDeleteDependents
0x0000000100ab1d6c ImDeleteDependents
0x00000001001d9688 IPRA.$imDeleteObject
0x00000001001f8338 imDeleteObject
0x0000000100f64698 ExpireBackupData
0x0000000100f5cba4 ExpirationProcessThread
0x000000010000da94 StartThread
Thread context:
COPY_TYPE: ARCHIVE
COMMAND: EXPIRE INVENTORY
SCHEDULE_TYPE: ADMIN
SCHEDULE_NAME: EXPIRING_INVENTORY
THREAD_TYPE: PROCESS
PROCESS_DESC: EXPIRE INVENTORY
PROCESS_NUMBER: 1167
DOMAIN_NAME: WIN_DOM
SCHEDULED: YES
NODE_NAME:
FILESPACE_NAME: APTABP01\SystemState\NULL\System State\SystemState

Thread 75500, Parent 75494: ExpirationProcessThread, Storage 91688, AllocCnt 4058942 HighWaterAmt 1924367
tid=2b9ec, ptid=299e6, det=1, zomb=0, join=0, result=0, sess=0, procToken=1167, sessToken=29336
Stack trace:
0x09000000001fb000 semop
0x090000001cd051e8 sqloSSemP
0x090000001cd04d08 sqlccipcrecv__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
0x090000001cd0460c sqlccrecv
0x090000001cd047f0 sqljcReceive__FP10sqljCmnMgr
0x090000001cd16c60 sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
0x090000001cce8680 CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001c7ecd04 SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cb8786c SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cce52fc SQLExecute
0x000000010019de80 tbRegExecEx
0x0000000100aabe1c IPRA.$ImDeleteDependents
0x0000000100aab7e0 IPRA.$ImDeleteDependents
0x0000000100ab1d6c ImDeleteDependents
0x00000001001d9688 IPRA.$imDeleteObject
0x00000001001f8338 imDeleteObject
0x0000000100f64698 ExpireBackupData
0x0000000100f5cba4 ExpirationProcessThread
0x000000010000da94 StartThread
Thread context:
COPY_TYPE: ARCHIVE
COMMAND: EXPIRE INVENTORY
SCHEDULE_TYPE: ADMIN
SCHEDULE_NAME: EXPIRING_INVENTORY
THREAD_TYPE: PROCESS
PROCESS_DESC: EXPIRE INVENTORY
PROCESS_NUMBER: 1167
DOMAIN_NAME: WIN_DOM
SCHEDULED: YES
NODE_NAME:
FILESPACE_NAME: APJIRAP2\SystemState\NULL\System State\SystemState

Thread 75501, Parent 75494: ExpirationProcessThread, Storage 6211374, AllocCnt 4885295 HighWaterAmt 6220182
tid=2baed, ptid=299e6, det=1, zomb=0, join=0, result=0, sess=0, procToken=1167, sessToken=29336
Stack trace:
0x09000000001fb000 semop
0x090000001cd051e8 sqloSSemP
0x090000001cd04d08 sqlccipcrecv__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
0x090000001cd0460c sqlccrecv
0x090000001cd047f0 sqljcReceive__FP10sqljCmnMgr
0x090000001c757db0 sqljrDrdaArOpen__FP14db2UCinterfaceP15db2UCCursorInfo
0x090000001c1fdba0 csmOpen__FP14db2UCinterfaceP15db2UCCursorInfo
0x090000001cbffdac CLI_sqlOpen__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cce63a8 SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000001cce52fc SQLExecute
0x0000000100189a00 RdbPrepareAndExecuteStmt
0x0000000100183d68 RdbCliFetch
0x0000000100182da4 tbCliSRFetch
0x0000000100aaac18 IPRA.$ImDeleteDependents
0x0000000100aab7e0 IPRA.$ImDeleteDependents
0x0000000100aab7e0 IPRA.$ImDeleteDependents
0x0000000100ab1d6c ImDeleteDependents
0x00000001001d9688 IPRA.$imDeleteObject
0x00000001001f8338 imDeleteObject
0x0000000100f64698 ExpireBackupData
0x0000000100f5cba4 ExpirationProcessThread
0x000000010000da94 StartThread
Holding mutex txnP->mutex (0x1186a0688), acquired at tbcli.c(956)
Thread context:
COPY_TYPE: ARCHIVE
COMMAND: EXPIRE INVENTORY
SCHEDULE_TYPE: ADMIN
SCHEDULE_NAME: EXPIRING_INVENTORY
THREAD_TYPE: PROCESS
PROCESS_DESC: EXPIRE INVENTORY
PROCESS_NUMBER: 1167
DOMAIN_NAME: WIN_DOM
SCHEDULED: YES
NODE_NAME:
FILESPACE_NAME: APECMD2\SystemState\NULL\System State\SystemState
Server PID: 33161266
Active threads: 662
Zombie threads: 0
Cached descriptors: 6