Removing Empty Storage Pool Volumes

[ILCattivo]

While doing some housekeeping I discovered a lot of EMPTY volumes (Some unused for quite some time), within a Primary DEDUP Storage Pool [FILE Dev Class] and would like to reclaim the disk space back and let the stg pool gradually recreate new vols if it ever needed more in the future.

Deleting the empty volumes from the stg pool within TSM is not a problem as I can see them being deleted from the stg pool within the actlog, but the physical volumes on the disk [e.g P:\DDSTG\DDSTGVOL1423] don't actually disappear to free up the space? I assumed that the 'del vol <volume path&name> cmd covered this?

TSM Server = Windows
TSM Server Ver 7.1.5.2

Is there another process that removes these later or something?
I'm a bit concerned that there now won't be enough space on the disk to create new volumes should it need them for backups if the older 'deleted' ones still reside on the disk?

Cheers

 

[marclant]

Have you run reclamation after deleting the volumes? I'm not sure if reclamation will clear them, but if you have not ran reclamation yet on that pool, it's worth a shot.
If they are still there after reclamation, it's safe to delete them as long as they are still not listed in Q VOL.

 

[erwanns]

Hi,

Pre-allocated volumes, whether they are DISK or FILE device type are not deleted at the FS level when they are deleted at TSM level.

As long as they do not exist in TSM anymore, you can safely delete them from the FS.

 

[ILCattivo]

Many thanks for the info guys... These are/were in-deed pre-allocated volumes (All of which no longer list using a Q Vol cmd)
I have blown these away using a ps script at the OS level and I now have the disk space reclaimed as expected.

One thing I have noticed this morning though is that I still have a hell of a lot of empty volumes present totalling nearly 1 TB of disk space (These do list in Q Vol), yet a few of our larger db backups fail because they report that there is not enough storage space available to fulfil their backups stored on the stg pool DEDUPPOOL. These continue to occur pre & post expiration & reclamation?

Here's an example of one of these volumes...


Volume Name: D:\DEDUP\DEDUPVOL014
Storage Pool Name: DEDUPPOOL
Device Class Name: FILE
Estimated Capacity: 5.0 G
Scaled Capacity Applied:
Pct Util: 0.0
Volume Status: Empty
Access: Read/Write
Pct. Reclaimable Space: 0.0
Scratch Volume?: No
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 5,658
Write Pass Number: 64
Approx. Date Last Written: 01/22/2017 02:30:28
Approx. Date Last Read: 01/23/2017 06:58:52
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): ADMIN
Last Update Date/Time: 05/10/2016 17:18:53
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager:
Logical Block Protected: No

REUSE DELAY is '0' on the DEDUPPOOL stg pool.

So why is TSM not using these EMPTY vols to accommodate backups?

Cheers guys.

 

[ILCattivo]

Right.. Bit of a discovery here... (This is a TSM server I have just recently inherited btw)

I have noticed that the 'directory' of the devclass associated to the storage pool has absolutely no correlation to where file volumes are actually being stored/accessed?

q devcl

Device Class Name: FILE
Device Access Strategy: Sequential
Storage Pool Count: 2
Device Type: FILE
Format: DRIVE
Est/Max Capacity (MB): 512,000.0
Mount Limit: 1,200
Mount Wait (min):
Mount Retention (min):
Label Prefix:
Drive Letter:
Library:
Directory: E:\TSM VOLUMES\DEDUP-VOLUMES
Server Name:

q vol

Volume Name: D:\DEDUP\DEDUPVOL014
Storage Pool Name: DEDUPPOOL
Device Class Name: FILE
Estimated Capacity: 5.0 G

???

Could this be the reason why it's not accessing the empty vols ??
Will simply adding 'D:\DEDUP' as a directory on the FILE dev class make it start using them?

 

[marclant]

I have noticed that the 'directory' of the devclass associated to the storage pool has absolutely no correlation to where file volumes are actually being stored/accessed?

It does, however...

You can define a devclass today pointing to directory D:\filepool and write to it for a certain period. Then you can update the devclass to point to E:\filepool instead. All the volumes in D:\filepool will still be available for reads, but all new data will now be written to E:\filepool.

Could this be the reason why it's not accessing the empty vols ??

Yes

Will simply adding 'D:\DEDUP' as a directory on the FILE dev class make it start using them?

Yes. Be careful, you can't add a directory or append to the list of directories, it replaces the list. The value you specify in the directory option has to include all the directories you want to use.

 

[ILCattivo]

Brilliant thanks marclant,

E:\TSM VOLUMES\DEDUP-VOLUMES doesn't actually exist on this particular TSM server anymore??, and TSM nicely verifies as such when appending more volume directories to the devclass.
In this case It has been removed to accommodate the proper volume directories for the 2 storage pools it hosts.


Device Class Name: FILE
Device Access Strategy: Sequential
Storage Pool Count: 2
Device Type: FILE
Format: DRIVE
Est/Max Capacity (MB): 512,000.0
Mount Limit: 1,200
Mount Wait (min):
Mount Retention (min):
Label Prefix:
Drive Letter:
Library:
Directory: D:\DEDUP,Q:\DEDUP
Server Name:

Will keep an eye on the hourly db backups that are running on this tsm server over the DEDUPPOOL to see if it starts using the shed load of EMPTY vols available to it.

Cheers fella.

 

[ILCattivo]

Hmmm.. Still not playing ball..!!

Have restarted the TSM server too...? Bit worrying this!

-------------------- ----------------------------------------------------------
01/23/2017 14:00:17 ANR0406I Session 756600 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(18564)). (SESSION: 756600)
01/23/2017 14:00:18 ANR0403I Session 756600 ended for node STANDBY4 (WinNT).
(SESSION: 756600)
01/23/2017 14:00:24 ANR0406I Session 756601 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(58737)). (SESSION: 756601)
01/23/2017 14:00:29 ANR0406I Session 756602 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(18319)). (SESSION: 756602)
01/23/2017 14:00:29 ANR0403I Session 756602 ended for node STANDBY4 (WinNT).
(SESSION: 756602)
01/23/2017 14:00:29 ANE4952I (Session: 756601, Node: STANDBY4) Total number
of objects inspected: 217 (SESSION: 756601)
01/23/2017 14:00:29 ANE4954I (Session: 756601, Node: STANDBY4) Total number
of objects backed up: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4958I (Session: 756601, Node: STANDBY4) Total number
of objects updated: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4960I (Session: 756601, Node: STANDBY4) Total number
of objects rebound: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4957I (Session: 756601, Node: STANDBY4) Total number
of objects deleted: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4970I (Session: 756601, Node: STANDBY4) Total number
of objects expired: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4959I (Session: 756601, Node: STANDBY4) Total number
of objects failed: 149 (SESSION: 756601)
01/23/2017 14:00:29 ANE4965I (Session: 756601, Node: STANDBY4) Total number
of subfile objects: 0 (SESSION: 756601)
01/23/2017 14:00:29 ANE4977I (Session: 756601, Node: STANDBY4) Total number
of bytes inspected: 156.46 MB (SESSION: 756601)
01/23/2017 14:00:29 ANE4961I (Session: 756601, Node: STANDBY4) Total number
of bytes transferred: 19.53 KB (SESSION: 756601)
01/23/2017 14:00:29 ANE4963I (Session: 756601, Node: STANDBY4) Data transfer
time: 0.00 sec (SESSION: 756601)
01/23/2017 14:00:29 ANE4966I (Session: 756601, Node: STANDBY4) Network data
transfer rate: 0.00 KB/sec (SESSION:
756601)
01/23/2017 14:00:29 ANE4967I (Session: 756601, Node: STANDBY4) Aggregate
data transfer rate: 3.52 KB/sec (SESSION:
756601)
01/23/2017 14:00:29 ANE4968I (Session: 756601, Node: STANDBY4) Objects
more... (<ENTER> to continue, 'C' to cancel)

compressed by: 0% (SESSION:
756601)
01/23/2017 14:00:29 ANE4976I (Session: 756601, Node: STANDBY4) Total data
reduction ratio: 99.99% (SESSION: 756601)
01/23/2017 14:00:29 ANE4969I (Session: 756601, Node: STANDBY4) Subfile
objects reduced by: 0% (SESSION:
756601)
01/23/2017 14:00:29 ANE4964I (Session: 756601, Node: STANDBY4) Elapsed
processing time: 00:00:05 (SESSION:
756601)
01/23/2017 14:00:29 ANR2579E Schedule DB_INC_HOURLY in domain DB for
node STANDBY4 failed (return code 12). (SESSION: 756601)
01/23/2017 14:00:30 ANR0403I Session 756601 ended for node STANDBY4 (WinNT).
(SESSION: 756601)
01/23/2017 14:00:34 ANR0406I Session 756603 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(59119)). (SESSION: 756603)
01/23/2017 14:00:34 ANR0403I Session 756603 ended for node STANDBY4 (WinNT).
(SESSION: 756603)
01/23/2017 14:04:12 ANR0406I Session 756618 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(61297)). (SESSION: 756618)
01/23/2017 14:04:17 ANR0406I Session 756619 started for node STANDBY4 (WinNT)
(Tcp/Ip 213.235.55.114(54159)). (SESSION: 756619)
01/23/2017 14:04:17 ANR0525W Transaction failed for session 756619 for node
STANDBY4 (WinNT) - storage media inaccessible. (SESSION:
756619)
01/23/2017 14:04:17 ANR0403I Session 756619 ended for node STANDBY4 (WinNT).
(SESSION: 756619)

 

[marclant]

Looks like you did a search for STANDBY4 because there are no messages without that. Try unfiltered from 01/23/2017 14:00 to 14:05 and look for volume or media errors.

 

[marclant]

Also at all the messages when the server starts to see if there are problems with volumes.

 

[ILCattivo]

Yep, my bad.. Not the most helpful that was it.. Maybe this is instead!!

01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756602)
01/23/2017 14:00:29 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756602)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL1831 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2586 - mount failed. (SESSION: 756619)
01/23/2017 14:04:17 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 756619)

These 4 volumes are in a 'Filling' state so it appears that this node is only wanting to use these specific volumes.

i.e
Volume Name: D:\DEDUP\DEDUPVOL2666
Storage Pool Name: DEDUPPOOL
Device Class Name: FILE
Estimated Capacity: 5.0 G
Scaled Capacity Applied:
Pct Util: 12.2
Volume Status: Filling
Access: Read/Write
Pct. Reclaimable Space: 0.0
Scratch Volume?: No
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 2,285
Write Pass Number: 53
Approx. Date Last Written: 01/20/2017 12:25:41
Approx. Date Last Read: 01/20/2017 12:25:43
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): ADMIN
Last Update Date/Time: 05/10/2016 18:06:35
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager:
Logical Block Protected: No

The node STANDBY4 has a mount limit of 99

 

[marclant]

Collocation enabled?
Let's pick that volume: DEDUPVOL2666 Are there errors about this volume since the last server start?

q ac search=DEDUPVOL2666 begintime=hh:mm

Also media errors:
q ac search=ANR8...E begintime=hh:mm

Set hh:mm to the time you restarted the server earlier today.

 

[ILCattivo]

Yep, Collocation is enabled : 'Group' , hwever there are no Groups defined so it reverts to 'Node' as default I believe...

OK, here's where things look like they've gone wrong for these particular volumes..

They came up as 'EMPTY' when I did my cleanup of empty volumes on Friday.. however a stg pool backup cmd looks like it needed them..

-------------------- ---------------------------------------------------------
-------------------- ---------------------------------------------------------
01/20/2017 01:24:49 ANR1044I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for space reclamation. (PROCESS: 3351)
01/20/2017 01:24:49 ANR8340I FILE volume D:\DEDUP\DEDUPVOL2666 mounted.
(PROCESS: 3351)
01/20/2017 01:24:49 ANR0512I Process 3351 opened input volume
D:\DEDUP\DEDUPVOL2666. (PROCESS: 3351)
01/20/2017 07:03:33 ANR1228I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for storage pool backup. (SESSION: 713961,
PROCESS: 3362)
01/20/2017 07:04:20 ANR8340I FILE volume D:\DEDUP\DEDUPVOL2666 mounted.
(SESSION: 713961, PROCESS: 3362)
01/20/2017 07:04:20 ANR0512I Process 3362 opened input volume
D:\DEDUP\DEDUPVOL2666. (SESSION: 713961, PROCESS: 3362)
01/20/2017 07:31:37 ANR1228I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for storage pool backup. (SESSION: 713961,
PROCESS: 3361)
01/20/2017 07:31:37 ANR8340I FILE volume D:\DEDUP\DEDUPVOL2666 mounted.
(SESSION: 713961, PROCESS: 3361)
01/20/2017 07:31:37 ANR0512I Process 3361 opened input volume
D:\DEDUP\DEDUPVOL2666. (SESSION: 713961, PROCESS: 3361)
01/20/2017 08:23:27 ANR1040I Space reclamation started for volume
D:\DEDUP\DEDUPVOL2666, storage pool DEDUPPOOL (process
number 3381). (PROCESS: 3381)
01/20/2017 08:23:27 ANR1044I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for space reclamation. (PROCESS: 3381)
01/20/2017 08:23:27 ANR1176I Moving data for collocation set 1 of 1 on volume
D:\DEDUP\DEDUPVOL2666. (PROCESS: 3381)
01/20/2017 08:23:50 ANR8340I FILE volume D:\DEDUP\DEDUPVOL2666 mounted.
(PROCESS: 3381)
01/20/2017 08:23:50 ANR0512I Process 3381 opened input volume
D:\DEDUP\DEDUPVOL2666. (PROCESS: 3381)
01/20/2017 08:24:14 ANR0515I Process 3381 closed volume
D:\DEDUP\DEDUPVOL2666. (PROCESS: 3381)
01/20/2017 08:24:14 ANR1041I Space reclamation ended for volume
D:\DEDUP\DEDUPVOL2666. (PROCESS: 3381)
01/20/2017 12:24:23 ANR8340I FILE volume D:\DEDUP\DEDUPVOL2666 mounted.
(SESSION: 718053)
01/20/2017 12:24:23 ANR0511I Session 718053 opened output volume
D:\DEDUP\DEDUPVOL2666. (SESSION: 718053)
01/20/2017 12:25:41 ANR0514I Session 718053 closed volume
D:\DEDUP\DEDUPVOL2666. (SESSION: 718053)
01/20/2017 12:49:12 ANR2017I Administrator ADMIN issued command: DELETE
VOLUME D:\DEDUP\DEDUPVOL2666 (SESSION: 717442)
01/20/2017 12:49:13 ANR2406E DELETE VOLUME: Volume D:\DEDUP\DEDUPVOL2666
still contains data. (SESSION: 717442)
01/21/2017 06:49:48 ANR1228I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for storage pool backup. (SESSION: 726532,
PROCESS: 3414)
01/21/2017 06:50:16 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 726532,
PROCESS: 3414)
01/21/2017 06:50:16 ANR1229W Volume D:\DEDUP\DEDUPVOL2666 is inaccessible
therefore it cannot be backed up to a copy storage pool
or copied to an active-data storage pool. (SESSION:
726532, PROCESS: 3414)
01/21/2017 06:50:18 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 726532,
PROCESS: 3414)
01/21/2017 06:50:18 ANR1229W Volume D:\DEDUP\DEDUPVOL2666 is inaccessible
therefore it cannot be backed up to a copy storage pool
or copied to an active-data storage pool. (SESSION:
726532, PROCESS: 3414)
01/21/2017 08:51:11 ANR1228I Removable volume D:\DEDUP\DEDUPVOL2666 is
required for storage pool backup. (SESSION: 726532,
PROCESS: 3414)
01/21/2017 08:51:11 ANR1401W Mount request denied for volume
more... (<ENTER> to continue, 'C' to cancel)

D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 726532,
PROCESS: 3414)
01/21/2017 08:51:11 ANR1229W Volume D:\DEDUP\DEDUPVOL2666 is inaccessible
therefore it cannot be backed up to a copy storage pool
or copied to an active-data storage pool. (SESSION:
726532, PROCESS: 3414)
01/21/2017 16:01:28 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 731974)
01/21/2017 16:01:28 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 731974)
01/21/2017 16:01:28 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 731974)
01/21/2017 16:01:28 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 731974)
01/21/2017 16:01:28 ANR1401W Mount request denied for volume
D:\DEDUP\DEDUPVOL2666 - mount failed. (SESSION: 731974)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
What I don't understand is that if you look at the cmd for the deletion of the volume it should not have touched it as it returned output to say it still contained data..?
Hence the reason I did not use the 'discarddata=y' afterwards for this very reason...

Can we get these FILE volumes back to their normal working state?

 

[ILCattivo]

Bad news... The 4 volume files are no longer present in the D:\DEDUP dir, so my ps script has blown them out of the water, hence the mount errors..

Ok, this looks like it was a case of bad timing and those vols (which were once EMPTY) were reused and filling again at the time I manually deleted them via my ps script. (Lesson learnt the hard way)

I'm lucky that this looks like it has only affected one Client node 'STANDBY4' which is the only node that runs hourly each day and is reporting as failing.. The rest are nightly scheduled jobs and are backing up successfully.

So my question now is.. how do I get this node STANDBY4 to start using new volumes for backups?

Cheers

 

[marclant]

Bad news... The 4 volume files are no longer present in the D:\DEDUP dir, so my ps script has blown them out of the water, hence the mount errors..

Should look at this to recover from this and minimize the data loss: http://www-01.ibm.com/support/docview.wss?uid=swg21883611

 

[ILCattivo]

A great many thanks marclant for your assistance with identifying this issue and the recommendations to resolve it..
Sorted now on all fronts.

Regards

NB:// For anyone reading this who is intending to clean-up empty file volumes via TSM with a script. I would strongly advise verifying that each volume was actually removed from TSM via the actlog or q vol before proceeding to purge the physical volume from disk.

Use either...

Query actlog begindate=today Begintime=<time you initiated your volume delete script> search='ANR2406E'

Query actlog begindate=today Begintime=<time you initiated your volume delete script> search='still contains data'

or

Q Vol Stg =<storage pool you are removing the volumes from>