TSM 8.1.0 -> 8.1.17 Upgrade Problem

[KevinP]

We recently upgraded two instances running on the same server, one manages communications with the library and the other handles the backups. The backup instance (TSM1) can no longer communicate with the library (TSMLIB) instance.

02/03/2023 14:39:39 ANR0150E Failed to open Server TSMLIB. There was an error
decrypting the Virtual Volumes password. (SESSION: 815)
02/03/2023 14:39:39 ANR1651E Server information for TSMLIB is not available.
(SESSION: 815)
02/03/2023 14:39:39 ANR4374E Session failure, source server detected an
internal error. (SESSION: 815)

I've tried update server forcesync=yes from both sides, no joy. In the devconf.dat the password and server password are a hashed value.

I've tried resetting the password but get an error.


Protect: TSMLIB>upd server tsm1 password=XXXXXXXXXXXXXXX

Do you wish to proceed? (Yes (Y)/No (N)) y
ANR2032E UPDATE SERVER: Command failed - internal server error detected.
ANR9999D Thread<3310> issued message 2032 from:
ANR9999D Thread<3310> 0x1470efb /opt/tivoli/tsm/server/bin/dsmserv(outTextfTrFL+0x8b)
ANR9999D Thread<3310> 0x6d3ac2 /opt/tivoli/tsm/server/bin/dsmserv(AdmUpdateServer+0x20b2)
ANR9999D Thread<3310> 0x70c5df /opt/tivoli/tsm/server/bin/dsmserv(AdmUseExtCmdTab+0x79f)
ANR9999D Thread<3310> 0x584f68 /opt/tivoli/tsm/server/bin/dsmserv(AdmCommandLocal+0x488)
ANR9999D Thread<3310> 0x58713c /opt/tivoli/tsm/server/bin/dsmserv(admCommand+0x186c)
ANR9999D Thread<3310> 0x1101432 /opt/tivoli/tsm/server/bin/dsmserv(SmAdminCommandThread+0x22)
ANR9999D Thread<3310> 0x1519ae3 /opt/tivoli/tsm/server/bin/dsmserv(StartThread+0x93)
ANR9999D Thread<3310> 0x7ffff78c4ea5 /lib64/libpthread.so.0(*UNKNOWN*)
ANR9999D Thread<3310> 0x7ffff3d83b0d /lib64/libc.so.6(clone+0x6d)
ANS8001I Return code 4.

Anybody have any ideas?

 

[LED888]

If we issue QUERY SERVER * on the Library Manager and Library Client.
Is there an out put or an error?

Good Luck,
Sias

 

[KevinP]

Hi, there's output for both.


Protect: TSM1>q server *

Server Comm. High-level Low-leve- Days Server Virtual Allow
Name Method Address l Address Since Password Volume Replacement
Last Set Password
Access Set
-------- ------ ------------- --------- ------ ---------- ---------- -----------
TSMLIB TCPIP 10.31.0.211 1505 6 Yes Yes No


Protect: TSMLIB>q server *

Server Comm. High-level Low-leve- Days Server Virtual Allow
Name Method Address l Address Since Password Volume Replacement
Last Set Password
Access Set
-------- ------ ------------- --------- ------ ---------- ---------- -----------
TSM1 TCPIP 10.31.0.211 1500 3 No No No

 

[LED888]

In the actlog, what are the messages leading up to
ANR2032E UPDATE SERVER: Command failed - internal server error detected.
and after
ANR9999D Thread<3310> 0x7ffff3d83b0d /lib64/libc.so.6(clone+0x6d)
ANS8001I Return code 4.


- Log in as the instance user on library client collect the following.
- db2 connect to TSMDB1
- db2 set schema TSMDB1
- db2 "select * from server_connect"
- db2 "select * from server_connect_info"

What are the out put for server_connect and server_connect_info?
Do we see the ServerName for the Library Manager?


Good Luck,
Sias

 

[KevinP]

Date/Time Message
-------------------- ----------------------------------------------------------
02/06/2023 10:40:09 ANR0840I Session 2 started for administrator KLP (Linux
x86-64) (SSL tsm1.msi.umn.edu[10.31.0.211]:37120) on
tsm1.msi.umn.edu:1505. (SESSION: 2)
02/06/2023 10:40:42 ANR2017I Administrator KLP issued command: UPDATE SERVER
tsm1 PASSWORD=?***? (SESSION: 2)
02/06/2023 10:40:42 ANR9999D_3855409967 secUpdatePassword(secpwd.c:548)
Thread<1427>: Unable to get key of type 22:256 (SESSION:
2)
02/06/2023 10:40:42 ANR9999D Thread<1427> issued message 9999 from: (SESSION:
2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x14743a0
/opt/tivoli/tsm/server/bin/dsmserv(OutDiagToCons+0xb0)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x147c8c0
/opt/tivoli/tsm/server/bin/dsmserv(outDiagfExt+0x1f0)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x10eda90
/opt/tivoli/tsm/server/bin/dsmserv(secUpdatePassword+0x4-
50) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x6d4139
/opt/tivoli/tsm/server/bin/dsmserv(AdmUpdateServer+0x272-
9) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x70c5df
/opt/tivoli/tsm/server/bin/dsmserv(AdmUseExtCmdTab+0x79f-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x584f68
/opt/tivoli/tsm/server/bin/dsmserv(AdmCommandLocal+0x488-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x58713c
/opt/tivoli/tsm/server/bin/dsmserv(admCommand+0x186c)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1101432
/opt/tivoli/tsm/server/bin/dsmserv(SmAdminCommandThread+-
0x22) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1519ae3
/opt/tivoli/tsm/server/bin/dsmserv(StartThread+0x93)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff78c4ea5
/lib64/libpthread.so.0(*UNKNOWN*) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff3d83b0d
/lib64/libc.so.6(clone+0x6d) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D_3446028870 AdmUpdateServer(adms2sc.c:5049)
Thread<1427>: secUpdateVvPassword rc=9999(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> issued message 9999 from: (SESSION:
2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x14743a0
/opt/tivoli/tsm/server/bin/dsmserv(OutDiagToCons+0xb0)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x147c8c0
/opt/tivoli/tsm/server/bin/dsmserv(outDiagfExt+0x1f0)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x6d4860
/opt/tivoli/tsm/server/bin/dsmserv(AdmUpdateServer+0x2e5-
0) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x70c5df
/opt/tivoli/tsm/server/bin/dsmserv(AdmUseExtCmdTab+0x79f-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x584f68
/opt/tivoli/tsm/server/bin/dsmserv(AdmCommandLocal+0x488-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x58713c
/opt/tivoli/tsm/server/bin/dsmserv(admCommand+0x186c)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1101432
/opt/tivoli/tsm/server/bin/dsmserv(SmAdminCommandThread+-
0x22) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1519ae3
/opt/tivoli/tsm/server/bin/dsmserv(StartThread+0x93)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff78c4ea5
/lib64/libpthread.so.0(*UNKNOWN*) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff3d83b0d
/lib64/libc.so.6(clone+0x6d) (SESSION: 2)
02/06/2023 10:40:42 ANR2032E UPDATE SERVER: Command failed - internal server
error detected. (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> issued message 2032 from: (SESSION:
2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1470efb
/opt/tivoli/tsm/server/bin/dsmserv(outTextfTrFL+0x8b)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x6d3ac2
/opt/tivoli/tsm/server/bin/dsmserv(AdmUpdateServer+0x20b-
2) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x70c5df
/opt/tivoli/tsm/server/bin/dsmserv(AdmUseExtCmdTab+0x79f-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x584f68
/opt/tivoli/tsm/server/bin/dsmserv(AdmCommandLocal+0x488-
) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x58713c
/opt/tivoli/tsm/server/bin/dsmserv(admCommand+0x186c)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1101432
/opt/tivoli/tsm/server/bin/dsmserv(SmAdminCommandThread+-
0x22) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x1519ae3
/opt/tivoli/tsm/server/bin/dsmserv(StartThread+0x93)
(SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff78c4ea5
/lib64/libpthread.so.0(*UNKNOWN*) (SESSION: 2)
02/06/2023 10:40:42 ANR9999D Thread<1427> 0x7ffff3d83b0d
/lib64/libc.so.6(clone+0x6d) (SESSION: 2)
02/06/2023 10:40:55 ANR2017I Administrator KLP issued command: ROLLBACK
(SESSION: 2)
02/06/2023 10:41:29 ANR2017I Administrator KLP issued command: QUERY ACTLOG
begint=-00:02 (SESSION: 2)

For the db2 commands, I don't see TSMLIB anywhere in the output, but Server_Name is TSM1.

 

[LED888]

If we did not see the server name, then the issue is with either the "SERVER_CONNECT" or "SERVER_CONNECT_INFO" tables. Since we see the server name for both out put the issue is not with the table.

We issue:
UPDATE SERVER tsm1 PASSWORD=?***?

In the actlog we see -
ANR9999D_3855409967 secUpdatePassword(secpwd.c:548) Thread<1427>: Unable to get key of type 22:256 ....
Starting to suspect an issue with the dsmserv.pwd file rather an issue with the DB2 database.

I do not know if we can do this or not.
- On the Library Client, since its having the issue. Halt IBM Spectrum Protect server's process.
- Rename dsmserv.pwd file in instance directory
- Restart the Library Client.

I am sure that the Library Client is a production system.
If you have a test system to rename the dsmserv.pwd file to get an idea of what to expect.

May have to set the password on the Library Manager and Library Client.

After restarting the Library Client and we get an untrusted server certificate, may want to review the following

Retrying certificate exchange between servers

If the above does not resolve the issue may need to open a case with IBM Support.

Good luck,
Sias

 

[KevinP]

Hi Sias,

Thanks for the help, I've opened a case with IBM. I did try moving the dsmserv.pwd out of the way and restarted the servers, and it was re-created, but still no joy.

Thanks again.

 

[LED888]

Thanks for the update.
Once you find out the solution, please post it here , curious what is the solution.

- Sias

 

[KevinP]

Hi,

The solution ended up being delete all the paths for TSM1 on TSMLIB,
delete server TSM1 from TSMLIB (it still errored, so I had to do this piece via DB2).

Then on TSM1

update server tsmlib sessionsecurity=trans
set serverpassword XXXXXXXX

on TSMLIB

define server tsm1 hla=10.31.0.211 lla=1500 serverpa=XXXXXXXX

then on both servers:

update server server_name forcesync=yes

I could then ping both servers from the other.

I added back the paths on TSMLIB and TSM1 was able to request drives and such.

Thanks for the help!