1. Community Tip: Please Give Thanks to Those Sharing Their Knowledge.
    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.
  2. Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)
    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Can I change backup versions value in order to reduce tapes usage?

Discussion in 'Backup / Archive Discussion' started by tivolimistery, Oct 6, 2017.

  1. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Hi All,

    I would like to reduce the number of tapes used by the TSM backup service.
    Do you think that this task could be done changing the current number of backup versions from 2 to only 1 version?
    Do you think that this kind of change could be done even though the involved backup storage pool has been already used?

    Thank you in advance,
    Mauro
     
  2.  
  3. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    Yes. However the number of versions should be based on business requirements, not tape usage. If your business is satisfied with 1 version, then yes that would reduce the number of tapes once expiration and reclamation runs.
    Yes. It's common to change retention policies as requirements change.
     
    tivolimistery likes this.
  4. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Thank you very much for your support.
    It was very useful
     
  5. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Hi Marclant,

    I'm sorry but I think that I should disturb you again.
    I reduced the number of backup versions from 2 to 1; I reduced the retention time as you can see from output below:

    tsm: TSMSERVER>query copygroup cluster_athena active archivefs_bck type=backup f=d

    Policy Domain Name: CLUSTER_ATHENA
    Policy Set Name: ACTIVE
    Mgmt Class Name: ARCHIVEFS_BCK
    Copy Group Name: STANDARD
    Copy Group Type: Backup
    Versions Data Exists: 1
    Versions Data Deleted: 1
    Retain Extra Versions: 0
    Retain Only Version: 1
    Copy Mode: Modified
    Copy Serialization: Shared Static
    Copy Frequency: 0
    Copy Destination: TAPEBCK_HSM_ATHENA
    Table of Contents (TOC) Destination:
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 09/10/2017 00:37:57
    Managing profile:
    Changes Pending: No

    I also save some statistics about the storage pool TAPEBCK_HSM_ATHENA:

    NODE_NAME TYPE NUM_FILES PHYSICAL_MB
    ------------------ ------------------ ----------- ----------------
    NGPFS_ATHENA Bkup 908534 1481827234.31

    After some days, I realized that nothing is changed. I see the same values.
    From "q act" command output, I noticed that "expire inventory" command doesn't scan all management class:

    ANR4391I Expiration processing node NGPFS_ATHENA, filespace /users/home, fsId 1, domain CLUSTER_ATHENA, and management class DEFAULT - for BACKUP type files.
    ANR4391I Expiration processing node NGPFS_ATHENA, filespace /archive, fsId 2, domain CLUSTER_ATHENA, and management class STANDARD - for BACKUP type files.
    ANR4391I Expiration processing node NGPFS_ATHENA, filespace /, fsId 4, domain CLUSTER_ATHENA, and management class DEFAULT - for BACKUP type files.


    ARCHIVEFS_BCK management class is not checked by "expire inventory". I think that it happen because it is not a default one.

    Policy Policy Mgmt Default
    Domain Set Name Class Mgmt
    Name Name Class ?
    --------- --------- --------- ---------
    AIX ACTIVE STANDARD Yes
    AIX STANDARD STANDARD Yes
    CLUSTER ACTIVE STANDARD Yes
    CLUSTER STANDARD STANDARD Yes
    CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK - No
    CLUSTER_ATHENA - ACTIVE STANDARD Yes
    CLUSTER_ATHENA - STANDARD ARCHIVEFS_BCK - No
    CLUSTER_ATHENA - STANDARD STANDARD Yes
    STANDARD ACTIVE STANDARD Yes
    STANDARD STANDARD STANDARD Yes

    How does can I start the expiration for this storage pool also? What's the difference between the following lines?

    CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK - No
    CLUSTER_ATHENA - STANDARD ARCHIVEFS_BCK - No

    If possible, is there a way to start the expiration without changing the current default management class?

    Thank you so much,
    Mauro Tridici
     
  6. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    P.S.: usually I execute a first and important backup procedure using CLUSTER_ATHENA STANDARD mgnt class (it writes to "TAPEBCK" storga pool). A second procedure (less important than the first one) is based on CLUSTER_ATHENA ARCHIVEFS_BCK mgmt class (it writes to "TAPEBCK_HSM_ATHENA" storage pool). So, another question is: do you think that I could temporary set CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK as default mgmt and then execute "expire inventory" without involve all backed-up data related to the first backup procedure?
    After "expire inventory" command I should restore the previous mgmt as the default one.
     
  7. DazRaz

    DazRaz ADSM.ORG Senior Member

    Joined:
    Jun 15, 2008
    Messages:
    148
    Likes Received:
    14
    Did you activate the new policy? It doesn't change anything until it is activated.
     
  8. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Hi DazRaz,

    from the 'q mgmt' output (above) it seems that the archivefs_bck is active,but it is not the default one. The default mgmt class is Standard and when I execute expire inventory it doesnt check other mgmt. I would like expire the data related to the bacukp managed by archivefs_bck mgmt without touch the data related to the backup managed by standard mgmt.

    Thank you
    Mauro
     
  9. DazRaz

    DazRaz ADSM.ORG Senior Member

    Joined:
    Jun 15, 2008
    Messages:
    148
    Likes Received:
    14
    Looking you your management class settings, I can see that there is no extra backups to be deleted.

    Versions Data Exists: 1
    Versions Data Deleted: 1
    Retain Extra Versions: 0
    Retain Only Version: 1


    RETExtra
    Specifies the number of days that the server retains a backup version
    after that version becomes inactive. A version of a file becomes
    inactive when the client stores a more recent backup version, or when
    the client deletes the file from the workstation and then runs a full
    incremental backup. The server deletes inactive versions based on
    retention time even if the number of inactive versions does not
    exceed the number allowed by the VEREXISTS or VERDELETED parameters.
    This parameter is optional. Possible values are:

    days
    Specifies the number of days to retain inactive backup versions.
    You can specify an integer from 0 to 9999.

    NOLimit
    Specifies that you want to retain inactive backup versions
    indefinitely.

    If you specify NOLIMIT, the server deletes extra backup versions
    based on the VEREXISTS parameter (when the file still exists on
    the client file system) or the VERDELETED parameter (when the file
    no longer exists on the client file system).

    Looks like there was never 2 version of the data in the first place.
     
    tivolimistery likes this.
  10. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Hi DazRaz,

    I really appreciate your analysis, thank you.
    But, mea culpa, I didn't give you some missing and important information, I'm sorry.
    I will try to explain why I need to expire the backup storage pool B and what kind of problem I'm experiencing.

    Introduction:
    In the past, on our HPC cluster, we created 3 GPFS file systems:
    /users/home, the users home fs;
    /work, the scratch fs;
    /archive, the HSM managed fs.

    /users/home file system is protected by incremental backup and this backup is based on the STANDARD management class that writes data in backup storage poool A, so:

    STANDARD MGMT CLASS (DEFAULT) -> Backup Policy A (2 versions, 1 version data deleted, 30 days extra version, 60 days only version) -> backup storage pool A

    /archive file system is an HSM managed file system, but, before migrate the files, we are used to backup them. In TSM client dsm.sys config file there is the following instruction

    include.backup /archive/.../* archivefs_bck

    This backup was based on the ARCHIVEFS_BCK management class that writes data in backup storage pool B, so:

    CUSTOM MGMT CLASS -> Backup Policy B (2 version data exists, 01version data deleted, 30 days extra version, 60 days only version) -> backup storage pool B

    Anyway, in order to reduce the number of tapes used by this backup, I deleted a lot of (not needed) files from /archive fs and I changed the Backup Policy B as follows:

    CUSTOM MGMT CLASS -> Backup Policy B (1 version data exists, 0 version data deleted, 0 days extra version, 0 days only version) -> backup storage pool B

    Then I executed "expire inventory" but nothing is happening.
    I think that this issue is due to the ARCHIVEFS_BCK that is not the default one.
    From the "q act" output, it seems that expire inventory doesn't check the backup storage pool related to ARCHIVEFS_BCK mgmt class.
    I could try to set this mgmt class as default one, launch expire inventory and set again the old mgmt as default.
    Can I do that? Or do you think that this policy could "expire" the backup storage pool A data also?
     
  11. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    No. Default MC is just that, the default. So if you don't specify a MC, it uses the default, if you specify one, it uses the one you specify. Expiration doesn't run against management classes, it runs against inactive objects, and then compares them to the retention of the management class they are bound to in order to determine if they should expire or not.

    If you retain extra versions 0 days, that means you don't keep extra versions. So you were keeping 2 versions for 0 days, that's pretty much the same as keeping 1 version because the net result is you only end up with the active version.
     
    tivolimistery likes this.
  12. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Thank you, Marclant. Everything is more clear now.
    Anyway, I need to understand why, after I deleted files from /archive file system mentioned above and after I changed the backup policy in the ARCHIVEFS_BCK mgmt, the number of files in the backup storage pool B is still the same. Expiration works well on HSM storage pool, but not in backup storage pool B.

    NUMBER OF FILE ON /ARCHIVE FS BEFORE DELETION:

    HSM POOL STATISTICS:

    tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEHSM_ATHENA' GROUP BY stgpool_name

    NUMBER_OF_FILES
    ---------------
    482444

    BACKUP POOL STATISTICS:

    tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

    NUMBER_OF_FILES
    ---------------
    908534

    NUMBER OF FILE ON /ARCHIVE FS AFTER DELETION:

    HSM POOL STATISTICS:

    tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEHSM_ATHENA' GROUP BY stgpool_name

    NUMBER_OF_FILES
    ---------------
    190779

    BACKUP POOL STATISTICS:

    tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

    NUMBER_OF_FILES
    ---------------
    908534

    Thank you for the time you spent for me,
    Mauro
     
  13. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    LAST UPDATE:

    I moved into a folder locate din /archive file system and I just checked the list of files that have been backed up in the past and that are still restorable.

    dsmc restore "/archive/sysm02/test_files_hsm/*" -pick -inactive

    I see only active version of files, BUT I see also files that have been deleted 1 year ago. They no longer exist on /archive file system.
    So, I tried to "refresh" TSM server database (and files status) running the following command:

    dsmc incremental "/archive/sysm02/test_files_hsm/*" -subdir=yes

    Magically, this command only updated the already backed up files and expired a great part of them!
    It seems to be the way I have to follow to expire all deleted files.

    But, now, I can't execute the "dsmc incremental /archive/* -subdir=yes" command to "refresh all files status.
    I would like update only the already backed up files in order to expire the deleted ones.

    Is there a way to do it easy?

    Thank you,
    Mauro
     
  14. DazRaz

    DazRaz ADSM.ORG Senior Member

    Joined:
    Jun 15, 2008
    Messages:
    148
    Likes Received:
    14
    Schedule your backup to run every day. TSM doesn't know when you have deleted a file on the client, it relies on the backup to run to update it.

    Why can't you execute it?
     
    tivolimistery likes this.
  15. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Backup service running on /archive file system saves only the files candidated to be migrated using HSM.
    In other words, GPFS produces the files lists (GPFS driven migration), TSM backups them and then HSM migrates them.
    So, if an HSM storage pool tape will be damaged, I can restore the files from backup storage pool tape.
    But I can't backup all files saved on /archive file system because the number of files is really huge.

    This is the reason why I need to backup only the files that have been already backed up (the update procedure should be very fast).
     
  16. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    I'm trying to obtain the backed up files using:

    dsmc query backup "/archive/*" -subdir=yes -filesonly > archive_backed_up_files.list

    If everything will be ok, I will try to backup the archive_backed_up_files.list filelist using:

    dsmc incremental -filelist=archive_backed_up_files.list

    At the end I will execute "expire inventory" command and I will pray for me :)
     
  17. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Hi All,

    following the procedure described above, I noticed that:

    1) "dsmc incremental -filelist=archive_backed_up_files.list" works well if I need to "update" only the already backed up files that are still on /archive file system;
    2) "dsmc incremental -filelist=archive_backed_up_files.list" doesn't work if I need to "expire" only the already backed up files that are not on /archive file system: TSM says that it can't backup a missing file :)

    At the end, I executed manually "dsmc incremental /archive/* -subdir=yes" and I stopped it immediately after the "expiration" phase avoiding, in this way, the backup of the resident files that have not been already backed up.

    It's not an elegant solution, but it works :)
    I reduced the number of files on backup storage pool B from 908.854 to 288900.

    tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

    NUMBER_OF_FILES
    ---------------
    288900


    Please, let me know if you have some other idea.
    Thank you again,
    Mauro
     
  18. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    Check that node for active vs inactive versions:
    Code:
    select state,count(*) as count from backups where node_name='NGPFS_ATHENA' and class_name='ARCHIVEFS_BCK' group by state
    If the count of inactive versions is 0, there is nothing that can qualify for expiration.
     
    tivolimistery likes this.
  19. tivolimistery

    tivolimistery ADSM.ORG Member

    Joined:
    Oct 10, 2009
    Messages:
    45
    Likes Received:
    0
    Thanks, select is still running :)
    I will write here the result as soon as it will be completed.
     
  20. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    I don't need the results, it's for your benefit only.
     

Share This Page