ANS2120W The last store operation date reported by the server...

[flex]

Since some time the client/API writes this to the dsmerror.log on DB2 DB servers:

02/20/24   00:46:09 ANS2120W The last store operation date reported by the server SERVER1 of 02/19/24   23:46:09 UTC does not match the last store operation date of 02/19/24   21:22:44 UTC stored by the client for the filespace /XDB.
02/20/24   00:46:09 ANS2120W The last store operation date reported by the server SERVER1 of 02/19/24   23:46:09 UTC does not match the last store operation date of 02/19/24   21:22:44 UTC stored by the client for the filespace /XDB.
02/20/24   08:46:30 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   07:46:30 UTC does not match the last store operation date of 02/20/24   04:05:20 UTC stored by the client for the filespace /YDB.
02/20/24   08:46:30 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   07:46:30 UTC does not match the last store operation date of 02/20/24   04:05:20 UTC stored by the client for the filespace /YDB.
02/20/24   12:53:39 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   11:53:39 UTC does not match the last store operation date of 02/20/24   07:46:29 UTC stored by the client for the filespace /ZDB.
02/20/24   12:53:39 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   11:53:39 UTC does not match the last store operation date of 02/20/24   07:46:29 UTC stored by the client for the filespace /ZDB.
02/21/24   00:46:59 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   23:46:59 UTC does not match the last store operation date of 02/20/24   21:23:40 UTC stored by the client for the filespace /ZDB.
02/21/24   00:46:59 ANS2120W The last store operation date reported by the server SERVER1 of 02/20/24   23:46:59 UTC does not match the last store operation date of 02/20/24   21:23:40 UTC stored by the client for the filespace /ZDB.
02/21/24   08:54:18 ANS2120W The last store operation date reported by the server SERVER1 of 02/21/24   07:54:18 UTC does not match the last store operation date of 02/21/24   04:01:10 UTC stored by the client for the filespace /ZDB.
02/21/24   08:54:19 ANS2120W The last store operation date reported by the server SERVER1 of 02/21/24   07:54:18 UTC does not match the last store operation date of 02/21/24   04:01:10 UTC stored by the client for the filespace /ZDB.

Do you have any idea what to check?

 

[LED888]

What is the full version of the SP Server and os that is hosting the application?
What is the version of DB2 and os that is hosting the application?

Are we able to backup the DB2 database?

Around the time frame of 02/21/24 08:54:18 and 02/21/24 08:54:19, on the SP Server what message(s) are we seeing in the SP Server actlog?
In the DB2 diag log, any error message(s) around the above time frame?

Also, are there any error message in the dsierror.log for the SP API?


Good Luck,
Sias

 

[flex]

Thank you very much for your feedback!

TSM: AIX 7200-05-06-2320, Server Version 8, Release 1, Level 14.100
DB2s: AIX 7200-05-06-2320, Client Version 7, Release 1, Level 6.5, Informational tokens are "DB2 v11.5.7.0", "s2111221000", "DYN2111221000AIX", and Fix Pack "0".

Yep. Apart from that messages, the backups seem to run right:

TSMADM [SERVER1]: q actlog begind=02/21/2024 begint=08:54:15 endd=02/21/2024 endt=08:54:20
IBM Spectrum Protect

Date/Time                Message
--------------------     ----------------------------------------------------------
02/21/2024 08:54:15      ANR0406I Session 742849 started for node DB_DB1_PR (AIX) (Tcp/Ip 10.237.97.32:34568). (SESSION: 742849)
02/21/2024 08:54:17      ANR0406I Session 742850 started for node DB_DB1_PR (DB2/AIX64) (Tcp/Ip 10.237.97.32:34571). (SESSION: 742850)
02/21/2024 08:54:17      ANR2507I Schedule DB2_LOGARCHIVE_4HRS_0845 for domain DB2 started at 02/21/24 08:45:00 for node DB_DB1_PR completed successfully at 02/21/24 08:54:17. (SESSION: 742849)
02/21/2024 08:54:17      ANR0403I Session 742849 ended for node DB_DB1_PR (AIX). (SESSION: 742849)
02/21/2024 08:54:17      ANR0406I Session 742851 started for node DB_DB1_PR (AIX) (Tcp/Ip 10.237.97.32:34572). (SESSION: 742851)
02/21/2024 08:54:17      ANR0403I Session 742851 ended for node DB_DB1_PR (AIX). (SESSION: 742851)
02/21/2024 08:54:18      ANR0406I Session 742852 started for node DB_BB1_PR (DB2/AIX64) (Tcp/Ip 10.237.97.32:34573). (SESSION: 742852)
02/21/2024 08:54:18      ANR0406I Session 742853 started for node DB_DB1_PR (DB2/AIX64) (Tcp/Ip 10.237.97.32:34574). (SESSION:742853)
02/21/2024 08:54:18      ANR0406I Session 742854 started for node DB_DB1_PR (DB2/AIX64) (Tcp/Ip 10.237.97.32:34575). (SESSION: 742854)
02/21/2024 08:54:19      ANR0403I Session 742852 ended for node DB_DB1_PR (DB2/AIX64). (SESSION: 742852)
02/21/2024 08:54:19      ANR0403I Session 742854 ended for node DB_DB1_PR (DB2/AIX64). (SESSION: 742854)

As far as I can see, there is no dsierror.logs on those machines.

I will also check the db2diaglogs.

Cheers,
_flex.

 

[LED888]

Your very welcome!

Okay ... messages in the actlog, we see the session start and end.

From the message manual regarding ANS2120W.

Explanation​

The last store operation date reported on the server does not match the date stored locally on the client. If connected to the replication server it is likely that the replication was done before the last store operation from the client to the primary server. Hence, the replication is out of date. If connected to the primary server it is likely that the previous backup was done from another machine.

System action​

Processing continues.

User response​

Contact the IBM Spectrum Protect server administrator. Restoring from an out of date replica may lead to loss of data. If the primary server cannot be recovered, a full backup to the replica may be appropriate. If connected to the primary server, the message can be ignored.

I've encounter ...

ANS2120W The last store operation date reported by the server <server> of MM/DD/YYYY HH:MM:SS UTC does not match the last store operation date of MM/DD/YYYY HH:MM:SS UTC stored by the client.

That is due to using more than one data mover to backup the VM via the TDP4VM.
I was hoping that there would be some hints in the actlog.

We are backing up DB2 not VM.

FYI:
With DB2, there is no TDP.
The DB2 hooks into the SP Server.
In other words, the TDP for DB2 is built into the SP Server code.
Which is why there is no TDP for DB2.

Since the backup is completing, I would not expect any error in the dsierror.log .
It does not hurt to look in the diserror.log .
If there was an error message in the dsierro.log, the backup should not be completing.

With the DB2 backup completing, I am not expecting any error in the db2diaglogs.

When did the issue started?
Did we just upgraded the SP Server to 8.1.14.100?
At this time the latest patch code for the SP Server is 8.1.16.100 .
The latest maintenance code for the SP Server is 8.1.21.000 .
I do not see any known issue regarding what we are encountering.

May have to open a case with IBM Support so that they can take a closer look at the code.


Good Luck,
Sias

 

[flex]

What interesting about this issue is that not all DB2s are affected, only some, but we haven't yet found a correlation between them...
The db2diaglog didn't contain any errors at that time reported by dsmerror_db2.

As we've no other ideas now, we're planning to restart the TSM server process at some point. :-(

 

[LED888]

Thanks for the update.

If the SP Server was hosted on a Windows OS, I would suggest a schedule system reboot.
You don't know how may times that I have encounter strange issue, and it was resolved by a system reboot.

With the SP Sever hosted on an AIX os, I have never rebooted just stop and restart the application.

Please keep us posted.

Good Luck,
Sias

 

[Mita201]

Have you checked if DB2 server's and ISP server's clocks are in sync?

 

[flex]

Of course, these were the first things we checked.
All machines are in sync with NTP.
In addition most of them are also using Db2 HADR.