Within the past week, the DB backups have been failing on this server.
Looking in the db2diag.log file, I am seeing:
2011-10-06-10.31.30.820265-240 E5649679E455 LEVEL: Info
PID : 14396 TID : 47256765983040PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-27731 APPID: *LOCAL.tsminst1.111006143025
AUTHID : TSMINST1
EDUID : 5622 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:1603
MESSAGE : Starting an online db backup.
2011-10-06-10.31.30.913302-240 I5650135E791 LEVEL: Error (OS)
PID : 22093 TID : 47249703067664PROC : db2vend (db2med
- 5740 (TSMDB1))
INSTANCE: tsminst1 NODE : 000
FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr, probe:180
MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
"No Storage Available for allocation"
DIA8305C Memory allocation failure occurred.
CALLED : OS, -, shmat OSERR: EINVAL (22)
DATA #1 : Memory set handle, PD_TYPE_OSS_MEM_SET_HDL, 48 bytes
0x00007FFF1D134550 : 0060 F4DE F52A 0000 0060 F4DE F52A 0000
.`...*...`...*..
0x00007FFF1D134560 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x00007FFF1D134570 : 0000 488D 0000 0000 0800 6E20 0100 0000 ..H.......n
....
2011-10-06-10.31.30.913659-240 I5650927E500 LEVEL: Error
PID : 22093 TID : 47249703067664PROC : db2vend (db2med
- 5740 (TSMDB1))
INSTANCE: tsminst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, DB2VEND_main, probe:1020
MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
"No Storage Available for allocation"
DIA8305C Memory allocation failure occurred.
DATA #1 : String, 57 bytes
Fenced vendor process unable to connect to shared memory.
2011-10-06-10.31.30.919246-240 I5656762E1150 LEVEL: Error
PID : 14396 TID : 47257793587520PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000
EDUID : 5740 EDUNAME: db2med.5622.0 (TSMDB1) 0
FUNCTION: DB2 UDB, database utilities, sqluVendorInitProcess, probe:133
MESSAGE : SQL2043N Unable to start a child process or thread.
DATA #1 : ZRC, PD_TYPE_ZRC, 4 bytes
0x800F00AF
DATA #2 : signed integer, 4 bytes
0
CALLSTCK:
[0] 0x00002AF5B48DE7C6 pdLog + 0x398
[1] 0x00002AF5B593FA64
_Z21sqluVendorInitProcessmP12SQLO_MEM_SETP13SQLO_MEM_POOLPKcS4_PKvmP28sqluFencedVendorProcessStateP18SQLO_VENDOR_HANDLEPi
+ 0x1CA
[2] 0x00002AF5B58DA38E _Z23sqluMCInitVendorProcessP13SQLUMC_IBLK_T +
0x3B8
[3] 0x00002AF5B58E013A _Z24sqluMCTestDevType4BackupP13SQLUMC_IBLK_T +
0xCA
[4] 0x00002AF5B58E0A07 _Z25sqluMCProcessBackupStatesP13SQLUMC_IBLK_T +
0x72D
[5] 0x00002AF5B58E0E24 _Z32sqluMCStartBackupMediaControllerPcj + 0x390
[6] 0x00002AF5B4E31D62 sqloEDUEntry + 0x2F4
[7] 0x0000003DB540673D /lib64/libpthread.so.0 + 0x673D
[8] 0x0000003DB4CD3F6D clone + 0x6D
[9] 0x0000000000000000 ?unknown + 0x0
2011-10-06-10.31.30.922934-240 E5657913E478 LEVEL: Severe
PID : 14396 TID : 47257793587520PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000
EDUID : 5740 EDUNAME: db2med.5622.0 (TSMDB1) 0
FUNCTION: DB2 UDB, database utilities, sqluMCInitVendorProcess, probe:2382
MESSAGE : SQL2043N Unable to start a child process or thread.
DATA #1 : String, 35 bytes
/home/tsminst1/sqllib/adsm/libtsm.a
DATA #2 : signed integer, 4 bytes
This servers DB is very small (9GB) since most backups are TDP. The
machine has 48GB RAM so that isn't it. Most ulimit values are
"unlimited".
I did a few Google searches with no success........
Server/machine has not been restarted since installing 6.2.2.30 on
07/21/2011
Thoughts/suggestions?
Zoltan Forray
TSM Software & Hardware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://infosecurity.vcu.edu/phishing.html
|