Delete STGpooldir with unavailable containers

[blankgap]

Hi,

a few weeks ago we had an incident where a co-worker unintentionally damaged some containers in 2 of our 31 containerpool-directories, but lets focus on just one, the last of them (Directory31)
I managed to get rid of most of the damaged containers, moved all the data to other directories and updated the stgpooldir to access=destroyed

q stgpooldir

DEDUPPOOL D:\STG\DR\Container31 Destroyed


Unfortunately, there are 3 containers remaining in this directory:

q container state=unavailable

D:\STG\DR\Container31\14\00000000000014cb.dcf DEDUPPOOL Dedup Unavailable
D:\STG\DR\Container31\24\0000000000002426.dcf DEDUPPOOL Dedup Unavailable
D:\STG\DR\Container31\39\00000000000039e6.dcf DEDUPPOOL Dedup Unavailable

Does anyone have an advice on how to get rid of these 3 remaining containers?
The Reusedelay on the stgpool is set to 0 (temporary) and if I issue "audit container <containername> action=removedamaged", the process finishes successfully, but the containers are not removed.
These containers do not even exist in the filesystem anymore (The directories 14, 24 and 39 don't exist in Container31 Mountpoint)

I really need to delete this stgpooldir to reuse the diskspace mounted on it, but even after a week with reusedelay=0, TSM just won't remove those containers and it's not possible to delete the stgpooldir with remaining containers in it.

Any help would be appreciated.

Kind regards

 

[Trident]

Hi,

You have used action=markdamaged befor removedamaged ?

If it still fails, open a pmr.

 

[MvtLoo]

> I really need to delete this stgpooldir to reuse the diskspace mounted on it

If you are really sure that there is no data anymore on this mount-point, you can remove it already at filesystem level and reuse the space.
Of course you still will have some dirt within SP, but at least you can continue with your storage needs.

From my experience, sometimes it takes some hours (or even until the next day) before background processes within SP are finished. So also just wait another day and see if the volumes disappear by itself.

 

[marclant]

You can see with the Query EXTENTUPDates how many extents are left to delete from the database.

When deleting containers, the server has a lot of work to do. The server needs to identify all the extents on that container, then identify which files they belong to, then delete that file also from the database, then check all the extents that belong to that file and determine if they are shared with other files. If shared with other files, they are kept, if not, also discarded. If the database layout or disks are subpar, that process will also take longer to do.

 

[blankgap]

> I really need to delete this stgpooldir to reuse the diskspace mounted on it

If you are really sure that there is no data anymore on this mount-point, you can remove it already at filesystem level and reuse the space.
Of course you still will have some dirt within SP, but at least you can continue with your storage needs.

Unfortunately I can't really reuse the space on the TSM-Server, as I don't want to create a new stgpooldir #32 when stgpooldir #31 is still defined and can't be deleted (because there are containers in it)

When deleting containers, the server has a lot of work to do. The server needs to identify all the extents on that container, then identify which files they belong to, then delete that file also from the database, then check all the extents that belong to that file and determine if they are shared with other files. If shared with other files, they are kept, if not, also discarded. If the database layout or disks are subpar, that process will also take longer to do.

I understand that there is a lot of workload on the database. But those containers are stuck at "unavailable" for multiple weeks now, so I think there should have been enough time for the Server to remove them.

I will post an update, if there is one.

 

[marclant]

I understand that there is a lot of workload on the database. But those containers are stuck at "unavailable" for multiple weeks now, so I think there should have been enough time for the Server to remove them.

What does the Query EXTENTUPDates command report?

 

[marclant]

Did you do this https://adsm.org/forum/index.php?th...ith-unavailable-containers.33249/#post-138127 ?

 

[blankgap]

What does the Query EXTENTUPDates command report?

Output of Query extentupdate:



Also, I tried to use action=markdamaged before action=removedamaged
But it doesn't delete any extents:
ANR3729I The AUDIT CONTAINER command process 2232 completed:
0 damaged objects are deleted.
0 damaged objects are skipped.
0 dependent objects are deleted.
0 orphan objects are deleted in cloud-container storage.
0 orphan objects are skipped in cloud-container storage pools