ADSM-L

Re: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system full

2014-04-09 10:02:33
Subject: Re: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system full
From: Bill Boyer <bjdboyer AT COMCAST DOT NET>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 9 Apr 2014 10:01:18 -0400
The first go around failed after about 5-hours with DB2 space shortage. The
V5 has a 260GB DB so I created 4x100GB TSM db file systems. When I got the
failure messages about file system full 2 of them were 100% and the other 2
were at 87%. So we create a 5th and had 500GB of database space. Then it ran
for 12.5 hours and failed with some record conversion error. Opened a PMR
and we got hit with APAR IC97407. Any volumes that are EMPTY or have a
%Utilized of 0 need to be deleted in the V5 server before you start. Or
upgrade to 6.3.4.300. Customer's not comfortable running it on a patch
level, so right now we're preparing for the 3rd attempt and making sure any
EMPTY volumes are deleted and any volumes with PCT_UTILIZED=0 and no content
are removed. This includes all the DISK volumes for the storage pools.
Interesting you can have a volume with PCT_UTILIZED=0 and it still have
content (Q CONTENT returns files). Apparently the rounding routing to
compute the used % doesn't round beyond the 0.0. Migrating off the DIRMC
storage pool had that volume showng %Utilized=0, but there are 100,000's of
file in it. As long as it has content, the volume is OK.

I'm really surprised at how fast this migration/conversion went last time.
Only 2:10 hrs for the extractdb phase. And they have 260GB at 90%.

As far as the AIX to Linux, seems to be just fine.

Ask me tomorrow how this 3rd attempt goes.

Bill

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Prather, Wanda
Sent: Monday, April 07, 2014 3:42 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system
full

Well, at base level 6.3.0 it was verboten, so that's probably what the tech
is referring to:

"Restriction: You cannot upgrade your server to run on an operating system
that is different from the operating system it currently runs on. For
example, you cannot upgrade a server running on an AIX system to a server
running on a Linux system."

But in 6.3.4 the upgrade guide lists Chap 12 as a specific case where you
can go from 5.5 on AIX to 6.3.4 on Linux.
I haven't dug into it to see how the Chap 12 procedure is different from the
standard ones.
Let us know how it comes out - inquiring minds want to know!

"Part 2. Migrating Tivoli Storage Manager V5 servers on AIX, HP-UX, or
Solaris systems to V6.3.4 on Linux A Tivoli Storage Manager V5 server that
runs on an AIX, HP-UX, or Solaris operating system can be migrated to V6.3.4
or later on a Linux x86_64 operating system."



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Bill Boyer
Sent: Thursday, April 03, 2014 5:59 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system
full

Added another 1TB file system to the database on the V6 server. Still
failed, but for a different reason. Turns out I should have been at
6.3.4.300. The 6.3.4.0 insertdb code has a problem with volumes that are
EMPTY, volumes with 0% utilization and Q CONTENT shows nothing. APAR
IC97407: http://www-01.ibm.com/support/docview.wss?uid=swg1IC97407

I couldn't believe I needed 500GB for the conversion. Now that it's done, my
(5) db filespaces are showing 3 at 36% and 2 at 74%. Why they aren't more
even distributed...? Now the customer needs to decide if they want to
upgrade to 6.3.4.300 and do the upgrade again, or attempt the 'local fix' of
making sure to delete all empty and 0% used volumes. DISK, FILE or tape. And
if I miss one...another 12-hours down the drain.

Interesting... after I opened the PMR on the 2nd failure, the tech who
called back said he was pretty sure I couldn't do this from AIX to Linux.

Bill

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Steven Langdale
Sent: Wednesday, April 02, 2014 2:43 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] V5->V6 conversion failed on INSERTDB with File system
full

Bill, what was the outcome?


On 1 April 2014 23:22, Bill Boyer <bjdboyer AT comcast DOT net> wrote:

> V5.5.7.0 on AIX converting over the network to V6.3.4.0 on Linux RedHat.
>
> The V5 DB is 260GB at 90%.
>
>
>
> The TSM DB is defined on 4x100GB file systems /tsmdb01 - /tsmdb04.
> Active log is 32GB and archive log file system is 200GB.
>
> The extractdb phase seems to complete successfully:
>
> ANR0409I Session 1 ended for server $UPGRADETARGET$ (Linux/x86_64).
>
> ANR1382I EXTRACTDB: Process 1, database extract, has completed.
>
> ANR1383I EXTRACTDB: Found 119 database objects.
>
> ANR1384I EXTRACTDB: Processed 75 database objects.
>
> ANR1385I EXTRACTDB: Skipped 44 empty database objects.
>
> ANR1386I EXTRACTDB: Failed to process 0 database objects.
>
> ANR1387I EXTRACTDB: Processed 1,178,563,762 database records.
>
> ANR1388I EXTRACTDB: Read 47,215,261 database pages.
>
> ANR1389I EXTRACTDB: Wrote 128,524,753,412 bytes.
>
> ANR1390I EXTRACTDB: Elapsed time was 2:16:21.
>
> ANR1391I EXTRACTDB: Throughput was 53936.19 megabytes per hour.
>
>
>
> After the ExtractDB phase completes in the InsertDB log:
>
> ANR1379I INSERTDB: Read 123,388,191,411 bytes and inserted
> 1,126,658,997
>
> database entries in 2:10:00 (54310.15 megabytes per hour).
>
> ANR1526I INSERTDB: Building indices and checking table integrity.
>
> ANR0409I Session 2 ended for server $UPGRADESOURCE$ (AIX-RS/6000).
>
> ANR1527I INSERTDB: Checked 69 of 75 database objects in 0:04:03.
>
> The file system is full.
>
> ANR0171I tbcli.c(10847): Error detected on 27:2, database in
> evaluation mode.
>
> ANR0131E tbcli.c(10847): Server DB space exhausted.
>
> ANR0162W Supplemental database diagnostic information:  -1:     :-968
>
> ([IBM][CLI Driver][DB2/LINUXX8664] SQL0968C  The file system is full.
>
> ).
>
>
>
> And a bunch more Transaction hash table..
>
>
>
> Lock hash table contents (slots=3002):
>
> Note: Enabling trace class TMTIMER will provide additional timing info
> on the
>
> following locks
>
>   *** no locks found ***
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:14:03.
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:24:03.
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:34:03.
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 0:44:03.
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 1:04:03.
>
> ANR1527I INSERTDB: Checked 70 of 75 database objects in 1:24:03.
>
>
>
> And then nothing. It just quits. But it appears to have run several
> hours after the extract/insert phase had completed. The Linux file
> systems shows
> /tsmdb01 at 48%, /tsmdb02 at 100%, /tsmdb03 at 100% and /tsmdb04 at
> 43%. 2 of the 4 file systems were at 100%, but I should have still had
> 100GB of database space left.
>
>
>
> I have since added another /tsmdb05 at 100GB and restarted the process.
>
>
>
> The other file systems: /home had 4.7GB of free space. /tmp 6.4GB,
> /var 4.4GB.
>
>
>
> Any ideas? I've got about 2-3 more hours to see if adding the 100GB of
> database space will resolve the problem.
>
>
>
> Bill Boyer
> DSS, Inc.
> (610) 927-4407
> "Enjoy life. It has an expiration date." - ??
>