Too Much Data Being Preserved on Tape

[saivan]

Greetings,

Fairly new to Spectrum Protect. We have a TS4300 running tape backups and one issue I'm running into is running out of tape. Our primary pool is reporting over 35 TB of tape being filled, but we have only about max 5 TB of data to back up. We've been running this system for almost a year, and about 6 months ago bought 10 scratch tapes, which are now almost run out.

I'm guessing that it's old data that's not being deleted and is being allowed to remain on tape, but I don't know if that's the case.

We'd really appreciate any help you could give.

 

[Trident]

Hi,

Are you using colocation for your tape pool? What does q vol (only tapevols) say?. I am looking for state full/filling/pending.

 

[saivan]

Thanks, Trident.

Not using co-location that I know of.

000001M8    LTOCOPYPOOL    LTOTAPE    0 M    0    Empty
000002M8    LTOPOOL    LTOTAPE    20.464 T    4.393    Filling
000004M8    LTOPOOL    LTOTAPE    11.661 T    69.724    Full
000005M8    LTOCOPYPOOL    LTOTAPE    10.72 T    63.043    Full
000006M8    LTOPOOL    LTOTAPE    11.244 T    65.344    Full
000007M8    LTOCOPYPOOL    LTOTAPE    11.085 T    71.771    Full
000008M8    LTOPOOL    LTOTAPE    10.792 T    63.52    Full
000009M8    LTOCOPYPOOL    LTOTAPE    20.464 T    16.382    Filling
000013M8    LTOCOPYPOOL    LTOTAPE    8.935 T    70.047    Full
000014M8    LTOPOOL    LTOTAPE    11.377 T    68.222    Full
000015M8    LTOPOOL    LTOTAPE    20.464 T    17.863    Filling
000017M8    LTOCOPYPOOL    LTOTAPE    20.464 T    54.517    Filling
000018M8    LTOPOOL    LTOTAPE    20.464 T    4.166    Filling

 

[Trident]

Hi,
Try this:

select cast(status as char(12)) as Status,count(*) as No_of_tapes,sum(EST_CAPACITY_MB)/1024/1024 as Total_cap,sum(EST_CAPACITY_MB*PCT_UTILIZED/100)/1024/1024 as Used_cap from volumes where STGPOOL_NAME='LTOPOOL' group by status

Just trying to see if you have a lot of filling tapes with low usage

 

[marclant]

Is expiration and reclamation of LTOPOOL and LTOCOPYPOOL run successfully on the server daily?

If yes, what's the reclamation threshold that you use?

When you do query volume f=d, do you have tapes with Pct. Reclaimable Space greater than the reclamation threshold that you use?

Not using co-location that I know of.

Please verify by looking at Q STG F=D

 

[saivan]

Hi,
Try this:

select cast(status as char(12)) as Status,count(*) as No_of_tapes,sum(EST_CAPACITY_MB)/1024/1024 as Total_cap,sum(EST_CAPACITY_MB*PCT_UTILIZED/100)/1024/1024 as Used_cap from volumes where STGPOOL_NAME='LTOPOOL' group by status

Just trying to see if you have a lot of filling tapes with low usage

SP1> select cast(status as char(12)) as Status,count(*) as No_of_tapes,sum(EST_CAPACITY_MB)/1024/1024 as Total_cap,sum(EST_CAPACITY_MB*PCT_UTILIZED/100)/1024/1024 as Used_cap from volumes where STGPOOL_NAME='LTOPOOL' group by status
STATUS    NO_OF_TAPES    TOTAL_CAP    USED_CAP
FILLING    3    61.3    5.423
FULL    4    45    30.083

 

[saivan]

Is expiration and reclamation of LTOPOOL and LTOCOPYPOOL run successfully on the server daily?

If yes, what's the reclamation threshold that you use?

When you do query volume f=d, do you have tapes with Pct. Reclaimable Space greater than the reclamation threshold that you use?

Please verify by looking at Q STG F=D

Expiration and reclamation are set daily.

Right now we have out reclamation threshold set at 50%, and none are currently over the threshold.

I have attached a screenshot of the Maintenance window. If you see below, you'll see in the reclaim command rows, there is a green arrow instead of a checkmark, is this weird?



Ouput of Q STG F=D:

Storage Pool Name: LTOCOPYPOOL
                     Storage Pool Type: Copy
                     Device Class Name: LTOTAPE
                          Storage Type: DEVCLASS
                            Cloud Type:
                             Cloud URL:
                        Cloud Identity:
                        Cloud Location:
                    Estimated Capacity: 102,743.3 G
                    Space Trigger Util:
                              Pct Util: 35.363
                              Pct Migr:
                           Pct Logical: 99.94
                          High Mig Pct:
                           Low Mig Pct:
                       Migration Delay:
                    Migration Continue:
                   Migration Processes:
                 Reclamation Processes: 1
                     Next Storage Pool:
                  Reclaim Storage Pool:
                Maximum Size Threshold:
                                Access: Read/Write
                           Description:
                     Overflow Location:
                 Cache Migrated Files?:
                            Collocate?: No
                 Reclamation Threshold: 50
             Offsite Reclamation Limit: No Limit
       Maximum Scratch Volumes Allowed: 7
        Number of Scratch Volumes Used: 6
         Delay Period for Volume Reuse: 0 Day(s)
                Migration in Progress?:
                  Amount Migrated (MB):
      Elapsed Migration Time (seconds):
              Reclamation in Progress?: No
        Last Update by (administrator): DB2ADMIN
                 Last Update Date/Time: 6/2/20, 16:50:29
              Storage Pool Data Format: Native
                  Copy Storage Pool(s):
                   Active Data Pool(s):
               Continue Copy on Error?:
                              CRC Data: No
                      Reclamation Type: Threshold
           Overwrite Data when Deleted:
                     Deduplicate Data?: No
  Processes For Identifying Duplicates:
                            Compressed:
         Space Used for Protected Data:
                   Total Pending Space:
                 Deduplication Savings:
                   Compression Savings:
                     Total Space Saved:
                        Auto-copy Mode:
 Contains Data Deduplicated by Client?: No
          Maximum Simultaneous Writers:
                     Protect Processes:
               Protection Storage Pool:
         Protect Local Storage Pool(s):
              Reclamation Volume Limit:
Date of Last Protection to Remote Pool:
 Date of Last Protection to Local Pool:
          Deduplicate Requires Backup?:
                             Encrypted:
                         Pct Encrypted:
            Cloud Space Allocated (MB):
             Cloud Space Utilized (MB):
                           Bucket Name:
              Local Estimated Capacity:
                        Local Pct Util:
                     Local Pct Logical:
                   Cloud Storage Class:
Remove Restored Cpy Before End of Life:

                     Storage Pool Name: LTOPOOL
                     Storage Pool Type: Primary
                     Device Class Name: LTOTAPE
                          Storage Type: DEVCLASS
                            Cloud Type:
                             Cloud URL:
                        Cloud Identity:
                        Cloud Location:
                    Estimated Capacity: 109,020.9 G
                    Space Trigger Util:
                              Pct Util: 33.346
                              Pct Migr: 100
                           Pct Logical: 99.883
                          High Mig Pct: 90
                           Low Mig Pct: 70
                       Migration Delay: 0
                    Migration Continue: Yes
                   Migration Processes: 1
                 Reclamation Processes: 1
                     Next Storage Pool:
                  Reclaim Storage Pool:
                Maximum Size Threshold: No Limit
                                Access: Read/Write
                           Description:
                     Overflow Location:
                 Cache Migrated Files?:
                            Collocate?: Group
                 Reclamation Threshold: 50
             Offsite Reclamation Limit:
       Maximum Scratch Volumes Allowed: 7
        Number of Scratch Volumes Used: 7
         Delay Period for Volume Reuse: 0 Day(s)
                Migration in Progress?: No
                  Amount Migrated (MB): 0
      Elapsed Migration Time (seconds): 0
              Reclamation in Progress?: No
        Last Update by (administrator): DB2ADMIN
                 Last Update Date/Time: 6/2/20, 16:50:40
              Storage Pool Data Format: Native
                  Copy Storage Pool(s):
                   Active Data Pool(s):
               Continue Copy on Error?: Yes
                              CRC Data: No
                      Reclamation Type: Threshold
           Overwrite Data when Deleted:
                     Deduplicate Data?: No
  Processes For Identifying Duplicates:
                            Compressed:
         Space Used for Protected Data:
                   Total Pending Space:
                 Deduplication Savings:
                   Compression Savings:
                     Total Space Saved:
                        Auto-copy Mode: Client
 Contains Data Deduplicated by Client?: No
          Maximum Simultaneous Writers:
                     Protect Processes:
               Protection Storage Pool:
         Protect Local Storage Pool(s):
              Reclamation Volume Limit:
Date of Last Protection to Remote Pool:
 Date of Last Protection to Local Pool:
          Deduplicate Requires Backup?:
                             Encrypted:
                         Pct Encrypted:
            Cloud Space Allocated (MB):
             Cloud Space Utilized (MB):
                           Bucket Name:
              Local Estimated Capacity:
                        Local Pct Util:
                     Local Pct Logical:
                   Cloud Storage Class:
Remove Restored Cpy Before End of Life:

 

[marclant]

Ok, so if your expiration and reclamation run successfully daily, and you have no tapes with more than 50% reclaimable, looks like things are working as they should. If you want to free up more tapes, you could run reclamation more aggressively by using a lower threshold. For example, if you have a lot of tapes with 40% reclaimable, setting your threshold to 40% would free up some tapes.

Check your retention policies to see if you are keeping data longer than desired.
q co * active
q co * active t=a

To see which node has data in which management class:
select node_name,class_name,count(*) as NUM_FILES, sum(actual_size) as BYTES from backups group by node_name,class_name
Note that this query while not intense, can run for several minutes to a few hours.

 

[saivan]

Thanks, marclant.

The results of q co * active are below. Is it saying that it's keeping copies for 365 days? If so, that's way too long. How do I change it?

SP1> q co * active f=d
                 Policy Domain Name: DIRECT_TO_TAPE
                    Policy Set Name: ACTIVE
                    Mgmt Class Name: DATA
                    Copy Group Name: STANDARD
                    Copy Group Type: Backup
               Versions Data Exists: 3
              Versions Data Deleted: 1
              Retain Extra Versions: No Limit
                Retain Only Version: 365
                          Copy Mode: Modified
                 Copy Serialization: Shared Static
                     Copy Frequency: 1
                   Copy Destination: LTOPOOL
Table of Contents (TOC) Destination:
     Last Update by (administrator): DB2ADMIN
              Last Update Date/Time: 8/6/19, 13:11:04
                   Managing profile:
                    Changes Pending: No

                 Policy Domain Name: DIRECT_TO_TAPE
                    Policy Set Name: ACTIVE
                    Mgmt Class Name: IMAGE
                    Copy Group Name: STANDARD
                    Copy Group Type: Backup
               Versions Data Exists: 3
              Versions Data Deleted: 1
              Retain Extra Versions: No Limit
                Retain Only Version: 365
                          Copy Mode: Modified
                 Copy Serialization: Shared Static
                     Copy Frequency: 1
                   Copy Destination: LTOPOOL
Table of Contents (TOC) Destination:
     Last Update by (administrator): DB2ADMIN
              Last Update Date/Time: 8/6/19, 13:11:04
                   Managing profile:
                    Changes Pending: No

 

[marclant]

You use the UPDATE COPYGROUP command. HELP UPDATE COPYGROUP. Make sure you look at the steps for archive copygroup.

You can do q occ t=a to see if you have any nodes with archived data.

Or my earlier query, but for the archive table
select node_name,class_name,count(*) as NUM_FILES, sum(actual_size) as BYTES from archives group by node_name,class_name

 

[saivan]

OK, so it gives me this failure message, can you point me in the right direction to get around this?

SP1> update copygroup direct_to_tape active data standard type=backup destination=LTO_POOL retonly=90
ANR1585E UPDATE COPYGROUP: Policy set ACTIVE cannot be modified.

 

[RecoveryOne]

You can't update the active policyset, you need to find the policy set that is being referenced, which should be easily found if you just issue a q co
Once you have updated the policyset, then you will need to activate it. Check out HELP ACTIVATE POLICYSET

Some good docs worth reading:
https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/srv.reference/r_cmd_policyset_update.html

https://www.ibm.com/support/knowled...icyset_activate.html#r_cmd_policyset_activate

And perhaps my favorite which goes over retention settings so well:
https://www.ibm.com/support/knowled...bm.itsm.srv.doc/r_mplmntpol_retbackpvers.html


Also, if you are using the Operations Center, you can goto Services > Polices as here:

Clicking on one of the policies will show something like this:


To the right hand side of that window not shown is a configure button, click that and you can change settings there. Also, there is an activate button that is greyed out that will become active when you save the change.

Before making any changes to your retention, I'd really look at the retaining backup versions link above to better understand what your changes will be doing and how it will translate to your environment.

Hope this helps!

 

[saivan]

Actually, I just copied the policy set and made the change. So, just to confirm, changing the retonly from 365 to 90 will clear up tape space from copies that are over 90 days old?

 

[saivan]

OK, thanks for the link. I read up on that. To clarify:
Right now we have

• Retain Extra Versions: No Limit
• Retain Only Version: 365

This means that we're indefinitely keeping inactive backup versions? I guess I don't fully understand their definitions of Retain Extra and Retain Only... but anyhow, we don't want to keep anything more than 90 days. So if I set both of those to 90 days will that help us to reclaim all that tape?

 

[saivan]

OK, I went ahead and made that change from 365 to 90. EXPIRE INVENTORY just deleted a bunch of old stuff. So, I think mission accomplished. Thanks!

 

[marclant]

I guess I don't fully understand their definitions of Retain Extra and Retain Only

If you check the help, it explains what they mean, just need to scroll down a bit. They all work together.
https://www.ibm.com/support/knowled....reference/r_cmd_copygroup_backup_update.html

But yes, if you don't want to keep anything more than 90 days, nolimit is definitely too long. Keep it mind that it works along with VEREXIST. So don't necessarily have data for 90 days if it exceeds VEREXIST first. If you need to restore up to 90 days, you need RETEXTRA=90 and VEREXISTS=NOLIMIT. So you will have an unlimited number of versions, but never older than 90 days.