ADSM-L

Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

2016-09-14 09:13:40
Subject: Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery
From: "Rhodes, Richard L." <rrhodes AT FIRSTENERGYCORP DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 14 Sep 2016 13:06:40 +0000
TSM came up!

Of the steps below, I added the entries to /etc/services and ran the db2set 
cmd.  The db2iupdt failed.

But TSM/DB2 came up!




rssymapi2p:/tsmdata/tsmoc/config==>./rc.start_tsm_in_forground.ksh
ANR7800I DSMSERV generated at 17:06:42 on Jan 24 2016.

IBM Tivoli Storage Manager for AIX
Version 7, Release 1, Level 4.100

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2015.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 11272436.
ANR0900I Processing options file /tsmdata/tsmoc/config/dsmserv.opt.
ANR7811I Using instance directory /tsmdata/tsmoc/config.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0152I Database manager successfully started.
ANR1628I The database manager is using port 51500 for server connections.
ANR1636W The server machine GUID changed: old value 
(73.28.0e.8a.af.16.11.dc.84.de.08.63.ac.11.10.4e), new value ().
ANR2100I Activity log process has started.
ANR2102I Activity log pruning started: removing entries prior to 08/15/16 
00:00:00.
ANR2741I Alert monitor has started.
ANR4726I The NAS-NDMP support module has been loaded.
ANR1794W TSM SAN discovery is disabled by options.
ANR2803I License manager started.
ANR8285I Shared Memory driver ready for connection with clients on port 1510
ANR9613W Error loading /opt/tivoli/tsm/server/bin/dsmlicense for Licensing 
function.
ANR8200I TCP/IP Version 4 driver ready for connection with clients on port 1500.
ANR2812W License Audit completed - ATTENTION: Server is NOT in compliance with 
license terms.
ANR2560I Schedule manager started.
ANR0984I Process 1 for EXPIRE INVENTORY (Automatic) started in the BACKGROUND 
at 09:03:17.
ANR0811I Inventory client file expiration started as process 1.
ANR0984I Process 2 for AUDIT LICENSE started in the BACKGROUND at 09:03:17.
ANR2820I Automatic license audit started as process 2.
ANR0167I Inventory file expiration process 1 processed for 0 minutes.
ANR0812I Inventory file expiration process 1 completed: processed 0 nodes, 
examined 0 objects, deleting 0 backup objects, 0 archive objects,
0 DB backup volumes, and 0 recovery plan files. 0 objects were retried and 0 
errors were encountered.
ANR0985I Process 1 for EXPIRE INVENTORY (Automatic) running in the BACKGROUND 
completed with completion state SUCCESS at 09:03:17.
ANR2825I License audit process 2 completed successfully - 0 nodes audited.
ANR0985I Process 2 for AUDIT LICENSE running in the BACKGROUND completed with 
completion state SUCCESS at 09:03:21.
ANR2812W License Audit completed - ATTENTION: Server is NOT in compliance with 
license terms.
ANR2103I Activity log pruning completed: 13532 records removed.
ANR2244I Activity Summary pruning started: removing entries prior to 08/15/16 
00:00:00.
ANR2245I Activity summary pruning completed: 347 records removed.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 3 started for server TSM2 (AIX) (Tcp/Ip) for remote monitoring.
ANR1990I Remote monitoring started for server TSM2.
ANR0408I Session 1 started for server TSMLM2 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSMLM2.
ANR0408I Session 2 started for server TSMSAP1 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSMSAP1.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 4 started for server TSM1 (AIX) (Tcp/Ip) for remote monitoring.
ANR1990I Remote monitoring started for server TSM1.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 5 started for server TSMLM1 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSMLM1.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 6 started for server TSMSAP2 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSMSAP2.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 7 started for server TSM9 (AIX) (Tcp/Ip) for remote monitoring.
ANR1990I Remote monitoring started for server TSM9.
ANR0408I Session 8 started for server TSM3 (AIX) (Tcp/Ip) for remote monitoring.
ANR1990I Remote monitoring started for server TSM3.
ANR1956W The password for the administrator IBM-OC-TSMOC is expired.
ANR0408I Session 9 started for server TSM8 (AIX) (Tcp/Ip) for remote monitoring.
ANR1990I Remote monitoring started for server TSM8.
ANR0408I Session 10 started for server TSM4 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSM4.
ANR0408I Session 11 started for server TSM7 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSM7.
ANR0408I Session 12 started for server TSM6 (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSM6.
ANR0408I Session 13 started for server TSMAPP (AIX) (Tcp/Ip) for remote 
monitoring.
ANR1990I Remote monitoring started for server TSMAPP.

ANR0098W This system does not meet the minimum memory requirements.
ANR0993I Server initialization complete.
ANR0916I TIVOLI STORAGE MANAGER distributed by Tivoli is now ready for use.
TSM:TSMOC>
Storage Management Server for AIX - Version 7, Release 1, Level 4.100

TSM:TSMOC>
ANR2812W License Audit completed - ATTENTION: Server is NOT in compliance with 
license terms.
ANR1959I Status monitor collecting current data at 09:03:57.
ANR1960I Status monitor finished collecting data at 09:04:01 and will sleep for 
60 minutes.

Storage Management Server for AIX - Version 7, Release 1, Level 4.100

TSM:TSMOC>








-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Paul van Dongen
Sent: Wednesday, September 14, 2016 8:30 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

Richard, 

I have recently moved several TSM servers between AIX systems (without 
restoring the DBs). Some points of attention:

On a system without TSM, you must first install TSM software. You will also 
need some tweaking of the DB registry.

1. correct hostname in <instance home dir>/sqllib/db2nodes.cfg 
2. /etc/services must contain:

DB2_<instance name>      60016/tcp
DB2_<instance name>_1    60017/tcp
DB2_<instance name>_2    60018/tcp
DB2_<instance name>_END  60019/tcp

Ports can vary, but must be in sequence

3. Instance "update": /opt/tivoli/tsm/db2/instance/db2iupdt <instance name>
4. As instance owner: ~/sqllib/adm/db2set -g DB2SYSTEM=<hostname>
5. dsm.sys for DBbackups has to be present (location depends on the TSM server 
version)

If you need more details please PM me.

Met vriendelijke groet / With kind regards,
Paul van Dongen
System Expert


Vancis BV
Science Park 408
1098 XH Amsterdam
W: www.vancis.nl

T:      020 560 6600
M:      06 4181 4446
E:      Paul.vanDongen AT vancis DOT nl



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rhodes, Richard L.
Sent: 14 September 2016 13:48
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] *EXTERNAL* Re: tsm/db2 won't start after boot drive 
crash/recovery

No, DB2 won't start at all.

rssymapi2p:/tsmdata/tsmoc/config==>db2start
SQL6031N  Error in the db2nodes.cfg file at line number "0".  Reason code "11".

I'm not sure how to state this, but I think there is some configuration in DB2 
(that would be in on the boot drive) that points to the TSM database TSMDB1.  I 
think that configuration is missing.

For example, I found out about this cmd fighting a db2 problem some time ago.  
Run on a working TSM system:
   ./db2ilist
   tsmuser          (our db2 owner)
Run on this recovered system:
   ./db2ilist        (no output)


Rick




-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Lee, Gary
Sent: Tuesday, September 13, 2016 3:42 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: *EXTERNAL* Re: tsm/db2 won't start after boot drive crash/recovery

Was db2 running for sure? 
In previous installs I had to modify the startup script to do a db2start then 
sleep for 10 secs to get tsm to start correctly fron reboot of the system.  
This was under redhat linux 6.

That was the message I would get.


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rhodes, Richard L.
Sent: Tuesday, September 13, 2016 3:29 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L] tsm/db2 won't start after boot drive crash/recovery

Hello,

We have a TSM v7.1.4 instance called TSMOC.  Its sole purpose in life is to run 
Operations Center.  It runs on an AIX lpar that's used for some other things.

Last Friday the AIX rootvg was lost during a live migration to a new storage 
system.  We decided to recover by installing a mksysb from a partner lpar which 
had the same AIX version and application version, but NOT TSM.  So the lpar got 
back up and running quickly - only took an hour, but it's missing the TSM 
install.  All the TSM instance files (dsmserv.opt, etc) and DB2 filesystems 
(data, logs, sqllib, etc) exist with all their data.  They weren't lost since 
they were on a different lun and volume group.

We installed TSM v7.1.4 and I then created the AIX db2 instance owner and 
group.  We then tried to start TSM.  We got msgs below.

ANR7801I Subsystem process ID is 16056324.
ANR0900I Processing options file /tsmdata/tsmoc/config/dsmserv.opt.
ANR7811I Using instance directory /tsmdata/tsmoc/config.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0172I rdbinst.c(3942): Error encountered performing action UPDATE DBM CFG.
ANR0162W Supplemental database diagnostic information:  -6031:**No State 
Information Available
(-2)**:-6031 (SQL6031N  Error in the db2nodes.cfg file at line number "0".  
Reason code "11".
).
ANR0171I dbiconn.c(1844): Error detected on 0:1, database in evaluation mode.
ANR0162W Supplemental database diagnostic information:  -1:08001:-1032 
([IBM][CLI Driver] SQL1032N  No start database manager command was issued.  
SQLSTATE=57019


I think I've got to tell db2 about the database that exists on disk.
We are going to call support, but figured I'd ask here first!

Any thought/comments are appreciated!

Rick


-----------------------------------------

The information contained in this message is intended only for the personal and 
confidential use of the recipient(s) named above. If the reader of this message 
is not the intended recipient or an agent responsible for delivering it to the 
intended recipient, you are hereby notified that you have received this 
document in error and that any review, dissemination, distribution, or copying 
of this message is strictly prohibited. If you have received this communication 
in error, please notify us immediately, and delete the original message.


-----------------------------------------The information contained in this 
message is intended only for the personal and confidential use of the 
recipient(s) named above. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that you have received this document in error and that 
any review, dissemination, distribution, or copying of this message is strictly 
prohibited. If you have received this communication in error, please notify us 
immediately, and delete the original message.


-----------------------------------------
The information contained in this message is intended only for the personal and 
confidential use of the recipient(s) named above. If the reader of this message 
is not the intended recipient or an agent responsible for delivering it to the 
intended recipient, you are hereby notified that you have received this 
document in error and that any review, dissemination, distribution, or copying 
of this message is strictly prohibited. If you have received this communication 
in error, please notify us immediately, and delete the original message.