Deduplication not working in 6.1.2.1?

[Monkeh]

Is anyone running TSM 6.1.2.1 and using Deduplication? I built a test server running 6.1.2.0 and had a dedupe ration of 60%

Today I upgraded it to 6.1.2.1 and configured a file dev class copypool. I configured it to dedupe and left it running for a couple of hours. When I returned it had completed but has only found 4% of duplicate data? it has the exact same data as the primary storage pool!


I then created a new 6.1.2.1 server and backed up the same data again and it only found 3% of duplicate data. To me it looks like dedupe is broken in 6.1.2.1. I am going to try downgrading the second server to 6.1.2.0 and see what happens to dedup % after a few backups.

edit: Im running the servers on Windows 2003 sp2
Dell 2950 4GB ram 8 cores.

The test data is file system backups of 8 non-live servers and a static copy of files from a file server.

 

[Monkeh]

I have downgraded to TSM 6.1.2.0. After running reclaim and move data's the deduplication figure has increased again, but I still have a large difference between primary and copy dedup levels.



tsm: TSM031A>q stg copy* f=d

Storage Pool Name: COPYPOOL
Storage Pool Type: Copy
Device Class Name: COPYFILE
Estimated Capacity: 244 G
Space Trigger Util: 91.9
Pct Util: 24.5
Pct Migr:
Pct Logical: 96.8
High Mig Pct:
Low Mig Pct:
Migration Delay:
Migration Continue:
Migration Processes:
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold:
Access: Read/Write
Description:
Overflow Location:
Cache Migrated Files?:
Collocate?: No
Reclamation Threshold: 100
Offsite Reclamation Limit: No Limit
Maximum Scratch Volumes Allowed: 999
Number of Scratch Volumes Used: 32
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?:
Amount Migrated (MB):
Elapsed Migration Time (seconds):
Reclamation in Progress?: No
Last Update by (administrator): ADMIN
Last Update Date/Time: 12/03/2009 09:46:59
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?:
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: Yes
Processes For Identifying Duplicates: 1
Duplicate Data Not Stored: 12,508 M (17%)


tsm: TSM031A>q stg file* f=d

Storage Pool Name: FILEPOOL
Storage Pool Type: Primary
Device Class Name: FILE
Estimated Capacity: 204 G
Space Trigger Util: 85.6
Pct Util: 23.0
Pct Migr: 23.0
Pct Logical: 65.9
High Mig Pct: 90
Low Mig Pct: 70
Migration Delay: 0
Migration Continue: Yes
Migration Processes: 1
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold: No Limit
Access: Read/Write
Description:
Overflow Location:
Cache Migrated Files?:
Collocate?: No
Reclamation Threshold: 100
Offsite Reclamation Limit:
Maximum Scratch Volumes Allowed: 100
Number of Scratch Volumes Used: 27
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?: No
Amount Migrated (MB): 0.00
Elapsed Migration Time (seconds): 0
Reclamation in Progress?: No
Last Update by (administrator): ADMIN
Last Update Date/Time: 12/02/2009 14:45:20
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?: Yes
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: Yes
Processes For Identifying Duplicates: 1
Duplicate Data Not Stored: 26,862 M (37%)


I have the same data in both pools. I have even moved the data off of all the volumes onto to new volumes to ensure that it has been processed by TSM.

Looking at the Logical_MB reported by the occupancy, I can see that it still has not deduplicated to the same level. Cannot report on Physical_MB as TSM looses this once you enable deduplication

tsm: TSM031A>run q_storage6

TYPE GB Files
----- ---------------------------------- ------------
Arch 6.88 1870
Bkup 89.35 467666

Total GB Total Files
---------------------------------- ------------
96.24 469536

Primary_Pool GB Files
-------------------------------- ---------------------------------- ------------
FILEPOOL 39.07 234768

Copy_Pool GB Files
-------------------------------- ---------------------------------- ------------
COPYPOOL 57.16 234768

Total Primary GB Total Primary Files
---------------------------------- --------------------
39.07 234768

Total Copy GB Total Copy Files
---------------------------------- -----------------
57.16 234768
ANR1462I RUN: Command script Q_STORAGE6 completed successfully.

 

[BBB]

I thought that when you did a "ba stg" from a deduped pool, to a deduped copypool, tsm would just copy the data as is (not require another deduplication).

When you did the move data commands, did you add the "reconstruct=yes" flag?

Can you post a q devc f=d for the two filepools?

What are you storing in this pool? Is it lots of small files or directories?

 

[Monkeh]

It certainly seems like it works that way as the identify processes do not report any new deduplicates for the copy pool.

I am storing filesystem backups from several 2003 servers, and db2 backups from the TCR server.

No I did not use the reconstruct option


Device Class Name: FILE
Device Access Strategy: Sequential
Storage Pool Count: 1
Device Type: FILE
Format: DRIVE
Est/Max Capacity (MB): 2,048.0
Mount Limit: 20
Mount Wait (min):
Mount Retention (min):
Label Prefix:
Drive Letter:
Library:
Directory: P:\STGPOOL,L:\STGPOOL
Server Name:
Retry Period:
Retry Interval:
Twosided:
Shared:
High-level Address:
Minimum Capacity:
WORM: No
Drive Encryption:
Scaled Capacity:
Last Update by (administrator): ADMIN
Last Update Date/Time: 19.10.2009 11:28:12

tsm: TSM031A>q devc copy* f=d

Device Class Name: COPYFILE
Device Access Strategy: Sequential
Storage Pool Count: 1
Device Type: FILE
Format: DRIVE
Est/Max Capacity (MB): 2,048.0
Mount Limit: 20
Mount Wait (min):
Mount Retention (min):
Label Prefix:
Drive Letter:
Library:
Directory: S:\COPYPOOL
Server Name:
Retry Period:
Retry Interval:
Twosided:
Shared:
High-level Address:
Minimum Capacity:
WORM: No
Drive Encryption:
Scaled Capacity:
Last Update by (administrator): ADMIN
Last Update Date/Time: 02.12.2009 14:41:41

 

[BBB]

Try doing the move datas again, but use recons=yes to reclaim space inside aggregates, this could possible make a big difference to the amount of space reclaimed.

 

[Monkeh]

moving data between volumes was starting to increase the dedupe value, but was taking too long, so I created a new copy storage pool and ran a backup db. It reached a 53% duplicate data, the primary pool is 49%. it's not exact but I guess that the volume util has an effect on this ratio.

I may run an update to 6.1.2.1 again just to check what happens. If the ratio drops I will have to log it with IBM

 

[Monkeh]

I fear that all the move data processes had an adverse effect on the integrity of my storage pools.

Today I deleted some inactive version of file from a node, when expiration ran it produced the following errors.

The size of the file storage pool has not decreased, the node now thinks that the data is still on the TSM server, but fails when I try to delete it again or restore it.

http://www-01.ibm.com/support/docvi...00&uid=swg21408547&loc=en_US&cs=utf-8&lang=en

I'm going to scrap this server and leave TSM6 for a few more versions, until most of these bugs have been resolved.


ANR0159E dbieval.c(860): Database deadlock detected on 20:27.
ANR0162W Supplemental database diagnostic information: -1:40001:-911
([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
).
ANR0157W Database operation INSERT for table BF.Dereferenced.Chunks failed with
result code 19 and tracking ID: 0785E620.
ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
operation code 19 and tracking id 0785E620. The data for column 0 is:
(int32)0.
ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
operation code 19 and tracking id 0785E620. The data for column 1 is:
(int64)2016394.
ANR0102E bfdedup.c(7572): Error 19 inserting row in table
"BF.Dereferenced.Chunks".
ANR1181E bftxn.c(485): Data storage transaction 0:3456914 was aborted.
ANR2183W bfdedup.c(8564): Transaction 0:3456914 was aborted.
ANR0136I Table updating statistics performed successfully for 19 of 19.
ANR0159E dbieval.c(860): Database deadlock detected on 17:15.
ANR0162W Supplemental database diagnostic information: -1:40001:-911
([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
).

 

[Masonit]

I fear that all the move data processes had an adverse effect on the integrity of my storage pools.

Today I deleted some inactive version of file from a node, when expiration ran it produced the following errors.

The size of the file storage pool has not decreased, the node now thinks that the data is still on the TSM server, but fails when I try to delete it again or restore it.

http://www-01.ibm.com/support/docvi...00&uid=swg21408547&loc=en_US&cs=utf-8&lang=en

I'm going to scrap this server and leave TSM6 for a few more versions, until most of these bugs have been resolved.


ANR0159E dbieval.c(860): Database deadlock detected on 20:27.
ANR0162W Supplemental database diagnostic information: -1:40001:-911
([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
).
ANR0157W Database operation INSERT for table BF.Dereferenced.Chunks failed with
result code 19 and tracking ID: 0785E620.
ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
operation code 19 and tracking id 0785E620. The data for column 0 is:
(int32)0.
ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
operation code 19 and tracking id 0785E620. The data for column 1 is:
(int64)2016394.
ANR0102E bfdedup.c(7572): Error 19 inserting row in table
"BF.Dereferenced.Chunks".
ANR1181E bftxn.c(485): Data storage transaction 0:3456914 was aborted.
ANR2183W bfdedup.c(8564): Transaction 0:3456914 was aborted.
ANR0136I Table updating statistics performed successfully for 19 of 19.
ANR0159E dbieval.c(860): Database deadlock detected on 17:15.
ANR0162W Supplemental database diagnostic information: -1:40001:-911
([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
).

http://www-01.ibm.com/support/docview.wss?rs=663&tcss=Newsletter&uid=swg21408547

\Masonit