ANR2097E Unable to retrieve the master encryption key

[nathrix]

After upgrading server from 7.6 to 7.1.9.3 the following Error was reported:

ANR2097E Unable to retrieve the master encryption key from the server password file, dsmserv.pwd.

Spectrum Protect (Product Level com.tivoli.dsm.server_7.1.9.20190617_1349)
Spectrum Protect Client installed 8.1.4.1
Operating System AIX 7.1 (OS Version 7200-03-02-1846)

I had a look at this link:
https://www.ibm.com/support/knowledgecenter/en/SSGSG7_7.1.8/srv.common/r_techchg_srv_ekey_718.html

The newly generated master encryption key is stored in a new key database, dsmkeydb.kdb. If the server has an existing master encryption key, the key is migrated from the dsmserv.pwd file to the new key database. The automatic generation of the master encryption key and its storage in the new key database are designed to enhance system security. Server certificates are still stored in the cert.kdb key database and accessed by the stash file cert.sth.

There is no dsmkeydb.kdb file.
As per link. "During the upgrade, the newly generated master encryption key is stored in a new key database, dsmkeydb.kdb."
The key was not migrated from the dsmserv.pwd file to the new key database.


TSM:/home/tsminst1#‌> ls -l cert*
-rw------- 1 tsminst1 tsmsrvrs 80 Oct 10 2018 cert.crl
-rw------- 1 tsminst1 tsmsrvrs 130080 Nov 19 10:41 cert.kdb <--- exact time of upgrade
-rw------- 1 tsminst1 tsmsrvrs 80 Oct 10 2018 cert.rdb
-rw------- 1 tsminst1 tsmsrvrs 129 Oct 10 2018 cert.sth
-rw-r--r-- 1 tsminst1 tsmsrvrs 1164 Oct 10 2018 cert256.arm

TSM:/home/tsminst1#‌> ls -l | grep dsm
-rw-r--r-- 1 tsminst1 tsmsrvrs 0 Nov 19 12:11 .dsmserv.ilock
-rw-r--r-- 1 tsminst1 tsmsrvrs 82353796 Nov 21 05:38 dsmaccnt.log
-rw-r--r-- 1 tsminst1 tsmsrvrs 623993 Nov 20 15:26 dsmffdc.log
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048582 Feb 26 2019 dsmffdc.log.1
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.2
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.3
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.4
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.5
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.6
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.7
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048658 Feb 26 2019 dsmffdc.log.8
-rw-r--r-- 1 tsminst1 tsmsrvrs 1048680 Feb 26 2019 dsmffdc.log.9
-rw-r--r-- 1 tsminst1 tsmsrvrs 27 Oct 10 2018 dsmserv.dbid
-rw-r--r-- 1 tsminst1 tsmsrvrs 257 Oct 3 2018 dsmserv.err
-rw-r--r-- 1 tsminst1 tsmsrvrs 703 Nov 18 10:08 dsmserv.opt
-rw------- 1 tsminst1 tsmsrvrs 181 Oct 3 2018 dsmserv.pwd <--- exact time of upgrade
-rw-r--r-- 1 tsminst1 tsmsrvrs 61 Nov 19 12:11 dsmserv.v6lock

TSM:/home/tsminst1#‌> find / -name dsmkeydb.kdb
find: cannot open < /proc/7733746 >
find: cannot open < /proc/8061438 >
TSM:/home/tsminst1#‌>

Now, the HUGE problem I have is I'm unable to run db backup after upgrading the server upgrade because of those keys and two days of backups have gone through with no issues!

TSM> backup db type=dbsnapshot devclass=LTO5RDC
ANR2270E The PROTECTKEYS parameter is not enabled.

TSM> set dbrecovery lto5rdc protectkeys=no
ANR2784W Specifying PROTECTKEYS=NO requires the server's encryption keys to be backed up manually.
Do you want to proceed? (Yes (Y)/No (N)) y

ANR2782I SET DBRECOVERY completed successfully and device class for automatic DB backup is set to LTO5RDC.

TSM> backup db type=dbsnapshot devclass=LTO5RDC
ANR2270E The PROTECTKEYS parameter is not enabled.

TSM> set dbrecovery lto5rdc protectkeys=yes passw=xxxxxxx
ANR2270E The PROTECTKEYS parameter is not enabled.

I have logged a call with IBM yesterday morning after noting the backup db did not run.
Thought I will post this here as well as any suggestions will be much appreciated and I'm sure there will be some poor folk out there that will run into this same issue.

 

[nathrix]

BTW, I started with this site a couple of months ago taking over from someone else.
The SP server version had vulnerabilities and they wanted it upgraded. I was hoping that they would refresh the environment first so that I can install latest and greatest and then migrate/replicate the old to the new. Unfortunate they are still undecided on the refresh on who will get the backup as a service contract as Comvault are still busy doing POC's.

 

[nathrix]

Update,
Attempted the following, still same issue:

TSM>set dbrecovery LTO5RDC protectkeys=yes password=xxxxxxx
ANR2270E The PROTECTKEYS parameter is not enabled.
ANS8001I Return code11.

ANR2270E The PROTECTKEYS parameter is not enabled.
https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.3/srv.msgs/ANR2270E.html

TSM>SET SERVERPASSWORD xxxxxxxx
ANR2131I Server password set.

TSM>set dbrecovery LTO5RDC protectkeys=yes password=xxxxxxx
ANR2270E The PROTECTKEYS parameter is not enabled.
ANS8001I Return code 11.

TSM>BACKUP DB devcl=lto5rdc protectkeys=yes password=xxxxxx
ANR2270E The PROTECTKEYS parameter is not enabled.
ANS8001I Return code 11.

+/- 21 minutes into the video:

 

[nathrix]

Still waiting anxiously for IBM to get back to me

 

[nathrix]

Thinking back after the upgrade was done, I was logged in as root when I started tsm:

/opt/tivoli/tsm/server/bin/rc.dsmserv -u tsminst1 -i /home/tsminst1

Could this perhaps be the cause of the issues?

 

[nathrix]

OK

Halted tsm and restarted tsm, working now!

TSM>
BACKUP DB devcl=lto5rdc protectkeys=yes password=c0k3ad7f
ANR2017I Administrator SERVER_CONSOLE issued command: BACKUP DB devcl=lto5rdc protectkeys=yes password=?***?
ANR0984I Process 1 for Database Backup started in the BACKGROUND at 10:45:24.
ANR4559I Backup DB is in progress.
ANR2280I Full database backup started as process 1.
TSM>
ANR8337I LTO volume STO190L5 mounted in drive DRIVE5 (/dev/rmt5).
ANR8987W The server will not encrypt the volume STO190L5.
ANR0513I Process 1 opened output volume STO190L5.
ANR1360I Output volume STO190L5 opened (sequence number 1).
ANR4626I Database backup will use 1 streams for processing with the number originally requested 1.
ANR0406I Session 5 started for node $$_TSMDBMGR_$$ (DB2/AIX64) (Tcp/Ip loopback(33037)).

 

[nathrix]

So, I think this is the VERY important part when the server started up this time:

TSM:/#> cd /home/tsminst1
TSM:/home/tsminst1#> /opt/tivoli/tsm/server/bin/rc.dsmserv -u tsminst1 -i /home/tsminst1
ANR7800I DSMSERV generated at 19:52:52 on Jun 17 2019.

IBM Tivoli Storage Manager for AIX
Version 7, Release 1, Level 9.300

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2018.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 16253350.
ANR4979W Use of environment variable DSMSERV_DIR is no longer supported.
ANR0900I Processing options file /home/tsminst1/dsmserv.opt.
ANR7811I Using instance directory /home/tsminst1.
ANR3339I Default Label in key data base is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0152I Database manager successfully started.
ANR1628I The database manager is using port 51500 for server connections.
ANR2278I The server master encryption key was moved from the server password file, dsmserv.pwd, to the server key database.
ANR2279W A server password file, dsmserv.pwd, was found during an upgrade operation. The file was renamed to dsmserv.pwd.20191121104338.deletionsave.
ANR1635I The server machine GUID, 60.99.ed.a4.6d.3f.11.e8.86.67.d2.37.45.c2.d7.0b, has initialized.
ANR2100I Activity log process has started.
ANR2741I Alert monitor has started.


The keys are there now:

TSM:/home/tsminst1#> ls -l dsmkeydb*
-rw------- 1 tsminst1 tsmsrvrs 3358 Nov 21 10:43 dsmkeydb.kdb
-rw------- 1 tsminst1 tsmsrvrs 193 Nov 21 10:43 dsmkeydb.sth