ANS9101E Migrated files could not be found during dsmrecall, but they are listed in the volume content.

[tivolimistery]

Dear All,

starting from some weeks ago, we are experiencing a very strange problem.
Our cluster users are trying to recall some files from /archive GPFS file system (/archive file system is managed by GPFS-driven HSM migration), but an error message says that some files "could not be found".

I also tried to recall them using dsmrecall command but with no luck:

[root@tsm1]# dsmrecall -d /archive/sp1/CESM/archive/sps_199806_011/C3S/sps_199806_011.C3S.tar
IBM Tivoli Storage Manager
Command Line Space Management Client Interface
Client Version 6, Release 4, Level 0.0
Client date/time: 28/08/2018 15:39:05
(c) Copyright by IBM Corporation and other(s) 1990, 2012. All Rights Reserved.

Session established with server TSMSERVER: AIX-RS/6000
Server Version 5, Release 5, Level 6.0
Server date/time: 28/08/2018 15:39:17 Last access: 28/08/2018 15:39:17

28/08/2018 15:39:07 ANS9101E Migrated files matching '/archive/sp1/CESM/archive/sps_199806_011/C3S/sps_199806_011.C3S.tar' could not be found.

In order to understand if the file is really missing or if it is still saved on a volume, I submitted a query to TSM server. The result: the file is present on volume with label "A01030L4".

The volume is available and correctly positioned in the tape library.

Library Name Volume Name Status Owner Last Use Home Device
Element Type
------------ ----------- ---------------- ---------- --------- ------- ------
TS3500 A01030L4 Private TSMSERVER Data 2.020 LTO

tsm: TSMSERVER>q vol A01030L4 f=d

Volume Name: A01030L4
Storage Pool Name: TAPEHSM_ATHENA
Device Class Name: LTO4
Estimated Capacity: 819,2 G
Scaled Capacity Applied:
Pct Util: 23,9
Volume Status: Filling
Access: Read/Write
Pct. Reclaimable Space: 0,0
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 1
Write Pass Number: 1
Approx. Date Last Written: 28.08.2018 13:39:46
Approx. Date Last Read: 28.08.2018 14:01:50
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 26.08.2018 22:45:40
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: None


As you can see, the file is the last one in the volume content list.

tsm: TSMSERVER>q content A01030L4

Node Name Type Filespace FSID Client's Name for File
Name
--------------- ---- ---------- ---- --------------------------------------
NGPFS_ATHENA SpMg /archive 2 /medscope/medscope_OC2a_n02/medscope_-
OC2a_n02.2022-02.tar.gz
NGPFS_ATHENA SpMg /archive 2 /medscope/medscope_OC2a_n02/medscope_-
OC2a_n02.2022-03.tar.gz
NGPFS_ATHENA SpMg /archive 2 /medscope/medscope_OC2a_n02/medscope_-
OC2a_n02.2022-04.tar.gz
NGPFS_ATHENA SpMg /archive 2 /medscope/medscope_OC2a_n02/medscope_-
OC2a_n02.2022-05.tar.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h3.-
1980-04.nc.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h3.-
1979-10.nc.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h0.-
1980-05.tar.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h1.-
1980-05.nc.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h0.-
1979-11.tar.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h2.-
1980-05.nc.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h1.-
1979-11.nc.gz
NGPFS_ATHENA SpMg /archive 2 /cmip01/CESM/archive/CMCC-CM2-VHR4-HS-
T/atm/hist/CMCC-CM2-VHR4-HST.cam.h2.-
1979-11.nc.gz
NGPFS_ATHENA SpMg /archive 2 /sp1/CESM/archive/sps_199806_011/C3S/-
sps_199806_011.C3S.tar

Could you please help me to understand what's happening? What can I do to solve the issue?
Because of this issue I have to restore the file from the backup pool, but it is not the best way to operate.

Thank you in advance.
Regards,
Mauro