ADSM-L

[ADSM-L] V5->V6 conversion failed on INSERTDB with File system full

2014-04-01 18:24:39
Subject: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system full
From: Bill Boyer <bjdboyer AT COMCAST DOT NET>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 1 Apr 2014 18:22:41 -0400
V5.5.7.0 on AIX converting over the network to V6.3.4.0 on Linux RedHat.

The V5 DB is 260GB at 90%.



The TSM DB is defined on 4x100GB file systems /tsmdb01 - /tsmdb04.  Active
log is 32GB and archive log file system is 200GB.

The extractdb phase seems to complete successfully:

ANR0409I Session 1 ended for server $UPGRADETARGET$ (Linux/x86_64).

ANR1382I EXTRACTDB: Process 1, database extract, has completed.

ANR1383I EXTRACTDB: Found 119 database objects.

ANR1384I EXTRACTDB: Processed 75 database objects.

ANR1385I EXTRACTDB: Skipped 44 empty database objects.

ANR1386I EXTRACTDB: Failed to process 0 database objects.

ANR1387I EXTRACTDB: Processed 1,178,563,762 database records.

ANR1388I EXTRACTDB: Read 47,215,261 database pages.

ANR1389I EXTRACTDB: Wrote 128,524,753,412 bytes.

ANR1390I EXTRACTDB: Elapsed time was 2:16:21.

ANR1391I EXTRACTDB: Throughput was 53936.19 megabytes per hour.



After the ExtractDB phase completes in the InsertDB log:

ANR1379I INSERTDB: Read 123,388,191,411 bytes and inserted 1,126,658,997

database entries in 2:10:00 (54310.15 megabytes per hour).

ANR1526I INSERTDB: Building indices and checking table integrity.

ANR0409I Session 2 ended for server $UPGRADESOURCE$ (AIX-RS/6000).

ANR1527I INSERTDB: Checked 69 of 75 database objects in 0:04:03.

The file system is full.

ANR0171I tbcli.c(10847): Error detected on 27:2, database in evaluation
mode.

ANR0131E tbcli.c(10847): Server DB space exhausted.

ANR0162W Supplemental database diagnostic information:  -1:     :-968

([IBM][CLI Driver][DB2/LINUXX8664] SQL0968C  The file system is full.

).



And a bunch more Transaction hash table..



Lock hash table contents (slots=3002):

Note: Enabling trace class TMTIMER will provide additional timing info on
the

following locks

  *** no locks found ***

ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:14:03.

ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:24:03.

ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:34:03.

ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:44:03.

ANR1527I INSERTDB: Checked 70 of 75 database objects in 1:04:03.

ANR1527I INSERTDB: Checked 70 of 75 database objects in 1:24:03.



And then nothing. It just quits. But it appears to have run several hours
after the extract/insert phase had completed. The Linux file systems shows
/tsmdb01 at 48%, /tsmdb02 at 100%, /tsmdb03 at 100% and /tsmdb04 at 43%. 2
of the 4 file systems were at 100%, but I should have still had 100GB of
database space left.



I have since added another /tsmdb05 at 100GB and restarted the process.



The other file systems: /home had 4.7GB of free space. /tmp 6.4GB, /var
4.4GB.



Any ideas? I've got about 2-3 more hours to see if adding the 100GB of
database space will resolve the problem.



Bill Boyer
DSS, Inc.
(610) 927-4407
"Enjoy life. It has an expiration date." - ??