Can I change backup versions value in order to reduce tapes usage?

tivolimistery

ADSM.ORG Member
Joined
Oct 10, 2009
Messages
58
Reaction score
0
Points
0
PREDATAR Control23

Hi All,

I would like to reduce the number of tapes used by the TSM backup service.
Do you think that this task could be done changing the current number of backup versions from 2 to only 1 version?
Do you think that this kind of change could be done even though the involved backup storage pool has been already used?

Thank you in advance,
Mauro
 
PREDATAR Control23

Do you think that this task could be done changing the current number of backup versions from 2 to only 1 version?
Yes. However the number of versions should be based on business requirements, not tape usage. If your business is satisfied with 1 version, then yes that would reduce the number of tapes once expiration and reclamation runs.
Do you think that this kind of change could be done even though the involved backup storage pool has been already used?
Yes. It's common to change retention policies as requirements change.
 
PREDATAR Control23

Hi Marclant,

I'm sorry but I think that I should disturb you again.
I reduced the number of backup versions from 2 to 1; I reduced the retention time as you can see from output below:

tsm: TSMSERVER>query copygroup cluster_athena active archivefs_bck type=backup f=d

Policy Domain Name: CLUSTER_ATHENA
Policy Set Name: ACTIVE
Mgmt Class Name: ARCHIVEFS_BCK
Copy Group Name: STANDARD
Copy Group Type: Backup
Versions Data Exists: 1
Versions Data Deleted: 1
Retain Extra Versions: 0
Retain Only Version: 1
Copy Mode: Modified
Copy Serialization: Shared Static
Copy Frequency: 0
Copy Destination: TAPEBCK_HSM_ATHENA
Table of Contents (TOC) Destination:
Last Update by (administrator): ADMIN
Last Update Date/Time: 09/10/2017 00:37:57
Managing profile:
Changes Pending: No

I also save some statistics about the storage pool TAPEBCK_HSM_ATHENA:

NODE_NAME TYPE NUM_FILES PHYSICAL_MB
------------------ ------------------ ----------- ----------------
NGPFS_ATHENA Bkup 908534 1481827234.31

After some days, I realized that nothing is changed. I see the same values.
From "q act" command output, I noticed that "expire inventory" command doesn't scan all management class:

ANR4391I Expiration processing node NGPFS_ATHENA, filespace /users/home, fsId 1, domain CLUSTER_ATHENA, and management class DEFAULT - for BACKUP type files.
ANR4391I Expiration processing node NGPFS_ATHENA, filespace /archive, fsId 2, domain CLUSTER_ATHENA, and management class STANDARD - for BACKUP type files.
ANR4391I Expiration processing node NGPFS_ATHENA, filespace /, fsId 4, domain CLUSTER_ATHENA, and management class DEFAULT - for BACKUP type files.


ARCHIVEFS_BCK management class is not checked by "expire inventory". I think that it happen because it is not a default one.

Policy Policy Mgmt Default
Domain Set Name Class Mgmt
Name Name Class ?
--------- --------- --------- ---------
AIX ACTIVE STANDARD Yes
AIX STANDARD STANDARD Yes
CLUSTER ACTIVE STANDARD Yes
CLUSTER STANDARD STANDARD Yes
CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK - No
CLUSTER_ATHENA - ACTIVE STANDARD Yes
CLUSTER_ATHENA - STANDARD ARCHIVEFS_BCK - No
CLUSTER_ATHENA - STANDARD STANDARD Yes
STANDARD ACTIVE STANDARD Yes
STANDARD STANDARD STANDARD Yes

How does can I start the expiration for this storage pool also? What's the difference between the following lines?

CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK - No
CLUSTER_ATHENA - STANDARD ARCHIVEFS_BCK - No

If possible, is there a way to start the expiration without changing the current default management class?

Thank you so much,
Mauro Tridici
 
PREDATAR Control23

P.S.: usually I execute a first and important backup procedure using CLUSTER_ATHENA STANDARD mgnt class (it writes to "TAPEBCK" storga pool). A second procedure (less important than the first one) is based on CLUSTER_ATHENA ARCHIVEFS_BCK mgmt class (it writes to "TAPEBCK_HSM_ATHENA" storage pool). So, another question is: do you think that I could temporary set CLUSTER_ATHENA - ACTIVE ARCHIVEFS_BCK as default mgmt and then execute "expire inventory" without involve all backed-up data related to the first backup procedure?
After "expire inventory" command I should restore the previous mgmt as the default one.
 
PREDATAR Control23

Did you activate the new policy? It doesn't change anything until it is activated.
 
PREDATAR Control23

Hi DazRaz,

from the 'q mgmt' output (above) it seems that the archivefs_bck is active,but it is not the default one. The default mgmt class is Standard and when I execute expire inventory it doesnt check other mgmt. I would like expire the data related to the bacukp managed by archivefs_bck mgmt without touch the data related to the backup managed by standard mgmt.

Thank you
Mauro
 
PREDATAR Control23

Looking you your management class settings, I can see that there is no extra backups to be deleted.

Versions Data Exists: 1
Versions Data Deleted: 1
Retain Extra Versions: 0
Retain Only Version: 1


RETExtra
Specifies the number of days that the server retains a backup version
after that version becomes inactive. A version of a file becomes
inactive when the client stores a more recent backup version, or when
the client deletes the file from the workstation and then runs a full
incremental backup. The server deletes inactive versions based on
retention time even if the number of inactive versions does not
exceed the number allowed by the VEREXISTS or VERDELETED parameters.
This parameter is optional. Possible values are:

days
Specifies the number of days to retain inactive backup versions.
You can specify an integer from 0 to 9999.

NOLimit
Specifies that you want to retain inactive backup versions
indefinitely.

If you specify NOLIMIT, the server deletes extra backup versions
based on the VEREXISTS parameter (when the file still exists on
the client file system) or the VERDELETED parameter (when the file
no longer exists on the client file system).

Looks like there was never 2 version of the data in the first place.
 
PREDATAR Control23

Hi DazRaz,

I really appreciate your analysis, thank you.
But, mea culpa, I didn't give you some missing and important information, I'm sorry.
I will try to explain why I need to expire the backup storage pool B and what kind of problem I'm experiencing.

Introduction:
In the past, on our HPC cluster, we created 3 GPFS file systems:
/users/home, the users home fs;
/work, the scratch fs;
/archive, the HSM managed fs.

/users/home file system is protected by incremental backup and this backup is based on the STANDARD management class that writes data in backup storage poool A, so:

STANDARD MGMT CLASS (DEFAULT) -> Backup Policy A (2 versions, 1 version data deleted, 30 days extra version, 60 days only version) -> backup storage pool A

/archive file system is an HSM managed file system, but, before migrate the files, we are used to backup them. In TSM client dsm.sys config file there is the following instruction

include.backup /archive/.../* archivefs_bck

This backup was based on the ARCHIVEFS_BCK management class that writes data in backup storage pool B, so:

CUSTOM MGMT CLASS -> Backup Policy B (2 version data exists, 01version data deleted, 30 days extra version, 60 days only version) -> backup storage pool B

Anyway, in order to reduce the number of tapes used by this backup, I deleted a lot of (not needed) files from /archive fs and I changed the Backup Policy B as follows:

CUSTOM MGMT CLASS -> Backup Policy B (1 version data exists, 0 version data deleted, 0 days extra version, 0 days only version) -> backup storage pool B

Then I executed "expire inventory" but nothing is happening.
I think that this issue is due to the ARCHIVEFS_BCK that is not the default one.
From the "q act" output, it seems that expire inventory doesn't check the backup storage pool related to ARCHIVEFS_BCK mgmt class.
I could try to set this mgmt class as default one, launch expire inventory and set again the old mgmt as default.
Can I do that? Or do you think that this policy could "expire" the backup storage pool A data also?
 
PREDATAR Control23

I think that this issue is due to the ARCHIVEFS_BCK that is not the default one.
No. Default MC is just that, the default. So if you don't specify a MC, it uses the default, if you specify one, it uses the one you specify. Expiration doesn't run against management classes, it runs against inactive objects, and then compares them to the retention of the management class they are bound to in order to determine if they should expire or not.

Retain Extra Versions: 0
If you retain extra versions 0 days, that means you don't keep extra versions. So you were keeping 2 versions for 0 days, that's pretty much the same as keeping 1 version because the net result is you only end up with the active version.
 
PREDATAR Control23

Thank you, Marclant. Everything is more clear now.
Anyway, I need to understand why, after I deleted files from /archive file system mentioned above and after I changed the backup policy in the ARCHIVEFS_BCK mgmt, the number of files in the backup storage pool B is still the same. Expiration works well on HSM storage pool, but not in backup storage pool B.

NUMBER OF FILE ON /ARCHIVE FS BEFORE DELETION:

HSM POOL STATISTICS:

tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEHSM_ATHENA' GROUP BY stgpool_name

NUMBER_OF_FILES
---------------
482444

BACKUP POOL STATISTICS:

tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

NUMBER_OF_FILES
---------------
908534

NUMBER OF FILE ON /ARCHIVE FS AFTER DELETION:

HSM POOL STATISTICS:

tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEHSM_ATHENA' GROUP BY stgpool_name

NUMBER_OF_FILES
---------------
190779

BACKUP POOL STATISTICS:

tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

NUMBER_OF_FILES
---------------
908534

Thank you for the time you spent for me,
Mauro
 
PREDATAR Control23

LAST UPDATE:

I moved into a folder locate din /archive file system and I just checked the list of files that have been backed up in the past and that are still restorable.

dsmc restore "/archive/sysm02/test_files_hsm/*" -pick -inactive

I see only active version of files, BUT I see also files that have been deleted 1 year ago. They no longer exist on /archive file system.
So, I tried to "refresh" TSM server database (and files status) running the following command:

dsmc incremental "/archive/sysm02/test_files_hsm/*" -subdir=yes

Magically, this command only updated the already backed up files and expired a great part of them!
It seems to be the way I have to follow to expire all deleted files.

But, now, I can't execute the "dsmc incremental /archive/* -subdir=yes" command to "refresh all files status.
I would like update only the already backed up files in order to expire the deleted ones.

Is there a way to do it easy?

Thank you,
Mauro
 
PREDATAR Control23

Schedule your backup to run every day. TSM doesn't know when you have deleted a file on the client, it relies on the backup to run to update it.

But, now, I can't execute the "dsmc incremental /archive/* -subdir=yes" command to "refresh all files status.

Why can't you execute it?
 
PREDATAR Control23

Backup service running on /archive file system saves only the files candidated to be migrated using HSM.
In other words, GPFS produces the files lists (GPFS driven migration), TSM backups them and then HSM migrates them.
So, if an HSM storage pool tape will be damaged, I can restore the files from backup storage pool tape.
But I can't backup all files saved on /archive file system because the number of files is really huge.

This is the reason why I need to backup only the files that have been already backed up (the update procedure should be very fast).
 
PREDATAR Control23

I'm trying to obtain the backed up files using:

dsmc query backup "/archive/*" -subdir=yes -filesonly > archive_backed_up_files.list

If everything will be ok, I will try to backup the archive_backed_up_files.list filelist using:

dsmc incremental -filelist=archive_backed_up_files.list

At the end I will execute "expire inventory" command and I will pray for me :)
 
PREDATAR Control23

Hi All,

following the procedure described above, I noticed that:

1) "dsmc incremental -filelist=archive_backed_up_files.list" works well if I need to "update" only the already backed up files that are still on /archive file system;
2) "dsmc incremental -filelist=archive_backed_up_files.list" doesn't work if I need to "expire" only the already backed up files that are not on /archive file system: TSM says that it can't backup a missing file :)

At the end, I executed manually "dsmc incremental /archive/* -subdir=yes" and I stopped it immediately after the "expiration" phase avoiding, in this way, the backup of the resident files that have not been already backed up.

It's not an elegant solution, but it works :)
I reduced the number of files on backup storage pool B from 908.854 to 288900.

tsm: TSMSERVER>select SUM(num_files) as Number_of_Files FROM occupancy WHERE stgpool_name LIKE 'TAPEBCK_HSM_ATHENA' GROUP BY stgpool_name

NUMBER_OF_FILES
---------------
288900


Please, let me know if you have some other idea.
Thank you again,
Mauro
 
PREDATAR Control23

Check that node for active vs inactive versions:
Code:
select state,count(*) as count from backups where node_name='NGPFS_ATHENA' and class_name='ARCHIVEFS_BCK' group by state
If the count of inactive versions is 0, there is nothing that can qualify for expiration.
 
PREDATAR Control23

Thanks, select is still running :)
I will write here the result as soon as it will be completed.
 
PREDATAR Control23

I don't need the results, it's for your benefit only.
 
Top