Deduplication Savings Issue (0% Savings) in Directory Container Storage Pool - TSM 7.1.3

[xyzegg]

Hi,

We need to understand why we got this result (0% savings) in Directory Container Storage Pool.

Environment description:

• TSM Server: Storage Management Server for Linux/x86_64 - Version 7, Release 1, Level 3.0

• Storage pool description:

tsm: BKPSERVER>q stg FOO_POOL_DC f=d

Storage Pool Name: foo_POOL_DC
Storage Pool Type: Primary
Device Class Name:
Storage Type: DIRECTORY
Cloud Type:
Cloud URL:
Cloud Identity:
Cloud Location:
Estimated Capacity: 12,110 G
Space Trigger Util:
Pct Util: 99.1
Pct Migr:
Pct Logical: 100.0
High Mig Pct:
Low Mig Pct:
Migration Delay:
Migration Continue:
Migration Processes:
Reclamation Processes:
Next Storage Pool: STD_POOL
Reclaim Storage Pool:
Maximum Size Threshold: No Limit
Access: Read/Write
Description: Storage Pool (Directory Container)
Overflow Location:
Cache Migrated Files?:
Collocate?:
Reclamation Threshold:
Offsite Reclamation Limit:
Maximum Scratch Volumes Allowed:
Number of Scratch Volumes Used:
Delay Period for Container Reuse: 1
Migration in Progress?:
Amount Migrated (MB):
Elapsed Migration Time (seconds):
Reclamation in Progress?:
Last Update by (administrator): foo.bar
Last Update Date/Time: 04/06/2016 12:00:55
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?:
CRC Data: No
Reclamation Type:
Overwrite Data when Deleted:
Deduplicate Data?: Yes
Processes For Identifying Duplicates:
Duplicate Data Not Stored: 0 (0%)
Auto-copy Mode:
Contains Data Deduplicated by Client?:
Maximum Simultaneous Writers: No Limit
Protection Storage Pool:
Date of Last Protection:
Deduplicate Requires Backup?:
Encrypted:
Space Utilized(MB):


tsm: BKPSERVER>

​

• Client version:
  Command Line Backup-Archive Client Interface
  Client Version 7, Release 1, Level 2.0

• dsm.sys:

servername BKPSERVER
commmethod tcpip
tcpport 1500
tcpclientport 1506
httpport 1586
tcpserveraddress 10.95.240.156
nodename XPTO-P1G-DEDUP-TEST
passwordaccess generate
tcpwin 64
tcpbuff 256
txnbyte 2097152
tcpnodelay yes
LARGECOMmbuffers no
compression yes
schedmode prompted
schedlogname /opt/tivoli/tsm/client/ba/bin/dsmsched_dedup.log
schedlogretention 30
errorlogname /opt/tivoli/tsm/client/ba/bin/dsmerror_dedup.log
errorlogretention 30
resourceutilization 10

* Domains
domain /xxx/yyy/zzzzz/wwww/data
domain /xxx/yyy/zzzzz/wwww/tablespace/qwebin1tsd01
domain /xxx/yyy/zzzzz/wwww/tablespace/qwe1tsi01
domain /xxx/yyy/zzzzz/wwww/tablespace/qwe1tsd01
domain /xxx/yyy/zzzzz/wwww/tablespace/qwebin1tsi01

******************************
**** EXCLUDE List ************
******************************
*** EXCLUDE /xxx/yyy/zzzzz/wwww/archive/.../*

**********************************
**** EXCLUDE from Compression ****
**********************************
EXCLUDE.COMPRESSION /xxx/yyy/zzzzz/wwww/tablespace/qwebin1tsd01/.../*
​

• Some Info

Reduction percentage: 0%
Bytes saved: 0 MB
Reduction ratio: No capacity
​

• Firt TEST backup RESULT:

Total number of objects inspected: 17,807
Total number of objects backed up: 15,616
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 0
Total number of objects failed: 0
Total number of objects encrypted: 0
Total number of objects grew: 0
Total number of retries: 0
Total number of bytes inspected: 10.05 TB
Total number of bytes transferred: 11.63 TB
Data transfer time: 553,272.83 sec
Network data transfer rate: 22,573.70 KB/sec
Aggregate data transfer rate: 90,098.39 KB/sec
Objects compressed by: -16%
Total data reduction ratio: 0.00%
Elapsed processing time: 38:30:19​

• df -h output:

Filesystem
Size | Used | Avail | Use% | Mounted on
/dev/mapper/vg_foo_bin1tsd01-lv_foo_bin1tsd01
15T | 9.3T | 4.8T | 67% | /xxx/yyy/zzzzz/wwww/tablespace/foobin1tsd01

/dev/mapper/vg_foo1_tsi01-lv_foo1_tsi01
394G | 108G | 266G | 29% | /xxx/yyy/zzzzz/wwww/tablespace/foo1tsi01

/dev/mapper/vg_foo1_tsd01-lv_foo1_tsd01
4.0T | 1.1T | 2.8T | 28% | /xxx/yyy/zzzzz/wwww/tablespace/foo1tsd01

/dev/mapper/vg_foo1_data-lv_foo1_data
1.2T | 6.6G | 1.1T | 1% | /xxx/yyy/zzzzz/wwww/data

/dev/mapper/vg_foo1_bin1tsi01-lv_foo1_bin1tsi01
98G | 641M | 93G | 1% | /xxx/yyy/zzzzz/wwww/tablespace/foobin1tsi01

###

/xxx/yyy/zzzzz/wwww/tablespace/foobin1tsd01 contains binary (PostgreSQL database) data. We expect to have some savings here, but, due to nature of data, some of we believes that TSM will not dedup the data here.

Now, we're going to try to delete this filespace and send it again without COMPRESSION.

Did I miss something?

​

 

[marclant]

How many nodes do you have backing up to it?
Have you done multiple backups, ie: multiple versions?

If you only backed up one node with only one backup, it's possible there is no duplicates.

When you say compression, are you talking client-side-compression? Is so, that's the issue. Compressed data, once broken down in chunks end up with a unique signature, therefore won't find duplicates. Should use server side compression starting at 7.1.5 if you want to compress the data for greater data reduction.

 

[xyzegg]

How many nodes do you have backing up to it?
Have you done multiple backups, ie: multiple versions?

Just a single node and one backup, but I expected to see some reduction savings in the same way.

If you only backed up one node with only one backup, it's possible there is no duplicates.

Ok. I'll consider it. Thanks.

When you say compression, are you talking client-side-compression? Is so, that's the issue. Compressed data, once broken down in chunks end up with a unique signature, therefore won't find duplicates. Should use server side compression starting at 7.1.5 if you want to compress the data for greater data reduction.

Thanks again for the information.

Now, I need to recover all the previously allocated space to do another backup with compression disabled. So I deleted the larger filespace, but It's deduplicated extents were not removed from storage pool. Maybe tweaking the "Delay period for container reuse" to solve this.

 

[marclant]

Maybe tweaking the "Delay period for container reuse" to solve this.

Typically, you would to keep that delay to the same as your DB backup retention, but for testing like you are doing, you could drop it to free it sooner.

 

[xyzegg]

Typically, you would to keep that delay to the same as your DB backup retention, but for testing like you are doing, you could drop it to free it sooner.

marclant,

What kind of action do I need to perform to discard that data? Until now, It seems like TSM does not delete the deduplicated extents from that storage pool.

 

[xyzegg]

Just for your information: TSM frees up the unused space from that storage pool some time ago.

 

[hogmaster]

You should look into the new 7.1.5 release of the server. where you can enable compression on th econtainerpool.This may help-

/hogmaster

 

[miejas]

Don't know if this is still applicable, but there's a bug in container storage pools with TSM versions older than 7.1.6. The update itself, doesn't fix the semi-corrupted containers you have now, but new data will be saved properly. The old data will never be deleted or reclaimed. The quickest way to get it fixed, is to create new stgpools with new containers and destroy the old ones. There are ways to get (most) of the data copied, but it will take a lot of time and effort. IBM support or your TSM partner can help you with your specific case.

 

[Trident]

Hi,

Look into 'generate dedupstats' command.