ADSM-L

Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

2016-09-14 09:55:19
Subject: Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery
From: "Rhodes, Richard L." <rrhodes AT FIRSTENERGYCORP DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 14 Sep 2016 13:54:00 +0000
Our TSM instance resides on a separate lun from AIX.   We try not to add 
filesystems to rootvg, for the most part.  I keep all our TSM and db2 setup in 
a filesystem tree called /tsmdata.  Here is what it looks like for this system:

  /dev/tsmdata            0.06      0.06    1%        6     1% /tsmdata
  /dev/tsmscripts         0.34      0.23   34%     1499     3% 
/tsmdata/tsm_scripts
  /dev/tsmoc             10.16     10.01    2%      759     1% /tsmdata/tsmoc
  /dev/db2activelog1      9.59      1.59   84%       23     1% 
/tsmdata/tsmoc/db2activelog1
  /dev/db2activelog2      9.59      1.59   84%       23     1% 
/tsmdata/tsmoc/db2activelog2
  /dev/db2archivelog     38.22     33.84   12%       18     1% 
/tsmdata/tsmoc/db2archivelog
  /dev/db2data1           4.00      1.95   52%      104     1% 
/tsmdata/tsmoc/db2data1
  /dev/db2data2           4.00      1.95   52%      104     1% 
/tsmdata/tsmoc/db2data2
  /dev/tsmocinstall      12.50      8.74   31%     3146     1% 
/tsmdata/tsmoc/install

The TSM config (dsmserv.opt and other stuff) is in /tsmdata/tsmoc/config.  
The DB2 home dir sqllib is at /tsmdata/tsmoc/config/sqllib.

All these filesystems are on a separate lun in a separate vg on separate lun.

  rsxxxxp:/tsmdata/tsmoc/config==>lsvg
    rootvg
    vgtsmoc

  rsxxxxp:/tsmdata/tsmoc/config==>lsvg -l rootvg
  rootvg:
  LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
  hd5                 boot       1       1       1    closed/syncd  N/A
  hd6                 paging     64      64      1    open/syncd    N/A
  hd8                 jfs2log    1       1       1    open/syncd    N/A
  hd4                 jfs2       1       1       1    open/syncd    /
  hd2                 jfs2       22      22      1    open/syncd    /usr
  hd9var              jfs2       6       6       1    open/syncd    /var
  hd3                 jfs2       12      12      1    open/syncd    /tmp
  hd1                 jfs2       1       1       1    open/syncd    /home
  hd10opt             jfs2       39      39      1    open/syncd    /opt
  lg_dumplv           sysdump    6       6       1    open/syncd    N/A
  scripts             jfs2       1       1       1    open/syncd    /scripts
  usremc              jfs2       8       8       1    open/syncd    /usr/emc

  rsxxxxp:/tsmdata/tsmoc/config==>lsvg -l vgtsmoc
  vgtsmoc:
  LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
  tsmdata             jfs2       2       2       1    open/syncd    /tsmdata
  tsmscripts          jfs2       11      11      1    open/syncd    
/tsmdata/tsm_scripts
  tsmoc               jfs2       325     325     1    open/syncd    
/tsmdata/tsmoc
  db2activelog1       jfs2       307     307     2    open/syncd    
/tsmdata/tsmoc/db2activelog1
  db2activelog2       jfs2       307     307     2    open/syncd    
/tsmdata/tsmoc/db2activelog2
  db2archivelog       jfs2       1223    1223    2    open/syncd    
/tsmdata/tsmoc/db2archivelog
  db2data1            jfs2       128     128     1    open/syncd    
/tsmdata/tsmoc/db2data1
  db2data2            jfs2       128     128     1    open/syncd    
/tsmdata/tsmoc/db2data2
  tsmocinstall        jfs2       400     400     2    open/syncd    
/tsmdata/tsmoc/install
  loglv00             jfs2log    1       1       1    open/syncd    N/A


When we lost the rootvg disk, all the TSM/DB2 config was just fine.

Interestingly, this was a good exercise.  Shortly, we are moving all our TSM 
instances to a new chassis - all new lpars with AIX7.  I can swing the luns 
with the TSM/DB2 filesystems to the new lpar, but I will have an entirely new 
rootvg that doesn't have TSM/DB2 installed.  What we just did to recover tsmoc 
is exactly what we will need to do on the new lpars.  We just got there a 
different way!

Now, we just have to install OC . . . . (not that we actually use it for much).

Thanks!


Rick



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
David Ehresman
Sent: Wednesday, September 14, 2016 9:25 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

Which leads me to ask.  Where (on AIX) does the tsmdb1 instance reside?

David

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rhodes, Richard L.
Sent: Wednesday, September 14, 2016 9:00 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive 
crash/recovery

Thanks!

I added the lines to /etc/services, and tried db2start, and db2 came up.


When I run db2iupdt I get:

  rssymapi2p:/home/root==>/opt/tivoli/tsm/db2/instance/db2iupdt tsmdb1
  DBI1446I  The db2iupdt command is running.
  DB2 installation is being initialized.
  The specified instance "tsmdb1" does not exist. Specify an existing instance 
name.


Thanks!

Rick




-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Paul van Dongen
Sent: Wednesday, September 14, 2016 8:30 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

Richard, 

I have recently moved several TSM servers between AIX systems (without 
restoring the DBs). Some points of attention:

On a system without TSM, you must first install TSM software. You will also 
need some tweaking of the DB registry.

1. correct hostname in <instance home dir>/sqllib/db2nodes.cfg 
2. /etc/services must contain:

DB2_<instance name>      60016/tcp
DB2_<instance name>_1    60017/tcp
DB2_<instance name>_2    60018/tcp
DB2_<instance name>_END  60019/tcp

Ports can vary, but must be in sequence

3. Instance "update": /opt/tivoli/tsm/db2/instance/db2iupdt <instance name>
4. As instance owner: ~/sqllib/adm/db2set -g DB2SYSTEM=<hostname>
5. dsm.sys for DBbackups has to be present (location depends on the TSM server 
version)

If you need more details please PM me.

Met vriendelijke groet / With kind regards,
Paul van Dongen
System Expert


Vancis BV
Science Park 408
1098 XH Amsterdam
W: www.vancis.nl

T:      020 560 6600
M:      06 4181 4446
E:      Paul.vanDongen AT vancis DOT nl



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rhodes, Richard L.
Sent: 14 September 2016 13:48
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive 
crash/recovery

No, DB2 won't start at all.

rssymapi2p:/tsmdata/tsmoc/config==>db2start
SQL6031N  Error in the db2nodes.cfg file at line number "0".  Reason code "11".

I'm not sure how to state this, but I think there is some configuration in DB2 
(that would be in on the boot drive) that points to the TSM database TSMDB1.  I 
think that configuration is missing.

For example, I found out about this cmd fighting a db2 problem some time ago.  
Run on a working TSM system:
   ./db2ilist
   tsmuser          (our db2 owner)
Run on this recovered system:
   ./db2ilist        (no output)


Rick




-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Lee, Gary
Sent: Tuesday, September 13, 2016 3:42 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

Was db2 running for sure? 
In previous installs I had to modify the startup script to do a db2start then 
sleep for 10 secs to get tsm to start correctly fron reboot of the system.  
This was under redhat linux 6.

That was the message I would get.


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rhodes, Richard L.
Sent: Tuesday, September 13, 2016 3:29 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L] tsm/db2 won't start after boot drive crash/recovery

Hello,

We have a TSM v7.1.4 instance called TSMOC.  Its sole purpose in life is to run 
Operations Center.  It runs on an AIX lpar that's used for some other things.

Last Friday the AIX rootvg was lost during a live migration to a new storage 
system.  We decided to recover by installing a mksysb from a partner lpar which 
had the same AIX version and application version, but NOT TSM.  So the lpar got 
back up and running quickly - only took an hour, but it's missing the TSM 
install.  All the TSM instance files (dsmserv.opt, etc) and DB2 filesystems 
(data, logs, sqllib, etc) exist with all their data.  They weren't lost since 
they were on a different lun and volume group.

We installed TSM v7.1.4 and I then created the AIX db2 instance owner and 
group.  We then tried to start TSM.  We got msgs below.

ANR7801I Subsystem process ID is 16056324.
ANR0900I Processing options file /tsmdata/tsmoc/config/dsmserv.opt.
ANR7811I Using instance directory /tsmdata/tsmoc/config.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0172I rdbinst.c(3942): Error encountered performing action UPDATE DBM CFG.
ANR0162W Supplemental database diagnostic information:  -6031:**No State 
Information Available
(-2)**:-6031 (SQL6031N  Error in the db2nodes.cfg file at line number "0".  
Reason code "11".
).
ANR0171I dbiconn.c(1844): Error detected on 0:1, database in evaluation mode.
ANR0162W Supplemental database diagnostic information:  -1:08001:-1032 
([IBM][CLI Driver] SQL1032N  No start database manager command was issued.  
SQLSTATE=57019


I think I've got to tell db2 about the database that exists on disk.
We are going to call support, but figured I'd ask here first!

Any thought/comments are appreciated!

Rick


-----------------------------------------

The information contained in this message is intended only for the personal and 
confidential use of the recipient(s) named above. If the reader of this message 
is not the intended recipient or an agent responsible for delivering it to the 
intended recipient, you are hereby notified that you have received this 
document in error and that any review, dissemination, distribution, or copying 
of this message is strictly prohibited. If you have received this communication 
in error, please notify us immediately, and delete the original message.


-----------------------------------------The information contained in this 
message is intended only for the personal and confidential use of the 
recipient(s) named above. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that you have received this document in error and that 
any review, dissemination, distribution, or copying of this message is strictly 
prohibited. If you have received this communication in error, please notify us 
immediately, and delete the original message.


-----------------------------------------

The information contained in this message is intended only for the personal and 
confidential use of the recipient(s) named above. If the reader of this message 
is not the intended recipient or an agent responsible for delivering it to the 
intended recipient, you are hereby notified that you have received this 
document in error and that any review, dissemination, distribution, or copying 
of this message is strictly prohibited. If you have received this communication 
in error, please notify us immediately, and delete the original message.


-----------------------------------------
The information contained in this message is intended only for the personal and 
confidential use of the recipient(s) named above. If the reader of this message 
is not the intended recipient or an agent responsible for delivering it to the 
intended recipient, you are hereby notified that you have received this 
document in error and that any review, dissemination, distribution, or copying 
of this message is strictly prohibited. If you have received this communication 
in error, please notify us immediately, and delete the original message.