Results 1 to 8 of 8
  1. #1
    Member
    Join Date
    May 2004
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Deduplication not working in 6.1.2.1?

    Is anyone running TSM 6.1.2.1 and using Deduplication? I built a test server running 6.1.2.0 and had a dedupe ration of 60%

    Today I upgraded it to 6.1.2.1 and configured a file dev class copypool. I configured it to dedupe and left it running for a couple of hours. When I returned it had completed but has only found 4% of duplicate data? it has the exact same data as the primary storage pool!


    I then created a new 6.1.2.1 server and backed up the same data again and it only found 3% of duplicate data. To me it looks like dedupe is broken in 6.1.2.1. I am going to try downgrading the second server to 6.1.2.0 and see what happens to dedup % after a few backups.

    edit: Im running the servers on Windows 2003 sp2
    Dell 2950 4GB ram 8 cores.

    The test data is file system backups of 8 non-live servers and a static copy of files from a file server.

  2. #2
    Member
    Join Date
    May 2004
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    I have downgraded to TSM 6.1.2.0. After running reclaim and move data's the deduplication figure has increased again, but I still have a large difference between primary and copy dedup levels.



    tsm: TSM031A>q stg copy* f=d

    Storage Pool Name: COPYPOOL
    Storage Pool Type: Copy
    Device Class Name: COPYFILE
    Estimated Capacity: 244 G
    Space Trigger Util: 91.9
    Pct Util: 24.5
    Pct Migr:
    Pct Logical: 96.8
    High Mig Pct:
    Low Mig Pct:
    Migration Delay:
    Migration Continue:
    Migration Processes:
    Reclamation Processes: 1
    Next Storage Pool:
    Reclaim Storage Pool:
    Maximum Size Threshold:
    Access: Read/Write
    Description:
    Overflow Location:
    Cache Migrated Files?:
    Collocate?: No
    Reclamation Threshold: 100
    Offsite Reclamation Limit: No Limit
    Maximum Scratch Volumes Allowed: 999
    Number of Scratch Volumes Used: 32
    Delay Period for Volume Reuse: 0 Day(s)
    Migration in Progress?:
    Amount Migrated (MB):
    Elapsed Migration Time (seconds):
    Reclamation in Progress?: No
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 12/03/2009 09:46:59
    Storage Pool Data Format: Native
    Copy Storage Pool(s):
    Active Data Pool(s):
    Continue Copy on Error?:
    CRC Data: No
    Reclamation Type: Threshold
    Overwrite Data when Deleted:
    Deduplicate Data?: Yes
    Processes For Identifying Duplicates: 1
    Duplicate Data Not Stored: 12,508 M (17%)


    tsm: TSM031A>q stg file* f=d

    Storage Pool Name: FILEPOOL
    Storage Pool Type: Primary
    Device Class Name: FILE
    Estimated Capacity: 204 G
    Space Trigger Util: 85.6
    Pct Util: 23.0
    Pct Migr: 23.0
    Pct Logical: 65.9
    High Mig Pct: 90
    Low Mig Pct: 70
    Migration Delay: 0
    Migration Continue: Yes
    Migration Processes: 1
    Reclamation Processes: 1
    Next Storage Pool:
    Reclaim Storage Pool:
    Maximum Size Threshold: No Limit
    Access: Read/Write
    Description:
    Overflow Location:
    Cache Migrated Files?:
    Collocate?: No
    Reclamation Threshold: 100
    Offsite Reclamation Limit:
    Maximum Scratch Volumes Allowed: 100
    Number of Scratch Volumes Used: 27
    Delay Period for Volume Reuse: 0 Day(s)
    Migration in Progress?: No
    Amount Migrated (MB): 0.00
    Elapsed Migration Time (seconds): 0
    Reclamation in Progress?: No
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 12/02/2009 14:45:20
    Storage Pool Data Format: Native
    Copy Storage Pool(s):
    Active Data Pool(s):
    Continue Copy on Error?: Yes
    CRC Data: No
    Reclamation Type: Threshold
    Overwrite Data when Deleted:
    Deduplicate Data?: Yes
    Processes For Identifying Duplicates: 1
    Duplicate Data Not Stored: 26,862 M (37%)


    I have the same data in both pools. I have even moved the data off of all the volumes onto to new volumes to ensure that it has been processed by TSM.

    Looking at the Logical_MB reported by the occupancy, I can see that it still has not deduplicated to the same level. Cannot report on Physical_MB as TSM looses this once you enable deduplication

    tsm: TSM031A>run q_storage6

    TYPE GB Files
    ----- ---------------------------------- ------------
    Arch 6.88 1870
    Bkup 89.35 467666

    Total GB Total Files
    ---------------------------------- ------------
    96.24 469536

    Primary_Pool GB Files
    -------------------------------- ---------------------------------- ------------
    FILEPOOL 39.07 234768

    Copy_Pool GB Files
    -------------------------------- ---------------------------------- ------------
    COPYPOOL 57.16 234768

    Total Primary GB Total Primary Files
    ---------------------------------- --------------------
    39.07 234768

    Total Copy GB Total Copy Files
    ---------------------------------- -----------------
    57.16 234768
    ANR1462I RUN: Command script Q_STORAGE6 completed successfully.

  3. #3
    Moderator BBB's Avatar
    Join Date
    Feb 2007
    Location
    Brisbane, Australia
    Posts
    2,075
    Thanks
    0
    Thanked 5 Times in 5 Posts

    Default

    I thought that when you did a "ba stg" from a deduped pool, to a deduped copypool, tsm would just copy the data as is (not require another deduplication).

    When you did the move data commands, did you add the "reconstruct=yes" flag?

    Can you post a q devc f=d for the two filepools?

    What are you storing in this pool? Is it lots of small files or directories?

  4. #4
    Member
    Join Date
    May 2004
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    It certainly seems like it works that way as the identify processes do not report any new deduplicates for the copy pool.

    I am storing filesystem backups from several 2003 servers, and db2 backups from the TCR server.

    No I did not use the reconstruct option


    Device Class Name: FILE
    Device Access Strategy: Sequential
    Storage Pool Count: 1
    Device Type: FILE
    Format: DRIVE
    Est/Max Capacity (MB): 2,048.0
    Mount Limit: 20
    Mount Wait (min):
    Mount Retention (min):
    Label Prefix:
    Drive Letter:
    Library:
    Directory: P:\STGPOOL,L:\STGPOOL
    Server Name:
    Retry Period:
    Retry Interval:
    Twosided:
    Shared:
    High-level Address:
    Minimum Capacity:
    WORM: No
    Drive Encryption:
    Scaled Capacity:
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 19.10.2009 11:28:12

    tsm: TSM031A>q devc copy* f=d

    Device Class Name: COPYFILE
    Device Access Strategy: Sequential
    Storage Pool Count: 1
    Device Type: FILE
    Format: DRIVE
    Est/Max Capacity (MB): 2,048.0
    Mount Limit: 20
    Mount Wait (min):
    Mount Retention (min):
    Label Prefix:
    Drive Letter:
    Library:
    Directory: S:\COPYPOOL
    Server Name:
    Retry Period:
    Retry Interval:
    Twosided:
    Shared:
    High-level Address:
    Minimum Capacity:
    WORM: No
    Drive Encryption:
    Scaled Capacity:
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 02.12.2009 14:41:41

  5. #5
    Moderator BBB's Avatar
    Join Date
    Feb 2007
    Location
    Brisbane, Australia
    Posts
    2,075
    Thanks
    0
    Thanked 5 Times in 5 Posts

    Default

    Try doing the move datas again, but use recons=yes to reclaim space inside aggregates, this could possible make a big difference to the amount of space reclaimed.

  6. #6
    Member
    Join Date
    May 2004
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    moving data between volumes was starting to increase the dedupe value, but was taking too long, so I created a new copy storage pool and ran a backup db. It reached a 53% duplicate data, the primary pool is 49%. it's not exact but I guess that the volume util has an effect on this ratio.

    I may run an update to 6.1.2.1 again just to check what happens. If the ratio drops I will have to log it with IBM

  7. #7
    Member
    Join Date
    May 2004
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    I fear that all the move data processes had an adverse effect on the integrity of my storage pools.

    Today I deleted some inactive version of file from a node, when expiration ran it produced the following errors.

    The size of the file storage pool has not decreased, the node now thinks that the data is still on the TSM server, but fails when I try to delete it again or restore it.

    http://www-01.ibm.com/support/docvie...=utf-8&lang=en

    I'm going to scrap this server and leave TSM6 for a few more versions, until most of these bugs have been resolved.


    ANR0159E dbieval.c(860): Database deadlock detected on 20:27.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).
    ANR0157W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    result code 19 and tracking ID: 0785E620.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 0 is:
    (int32)0.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 1 is:
    (int64)2016394.
    ANR0102E bfdedup.c(7572): Error 19 inserting row in table
    "BF.Dereferenced.Chunks".
    ANR1181E bftxn.c(485): Data storage transaction 0:3456914 was aborted.
    ANR2183W bfdedup.c(8564): Transaction 0:3456914 was aborted.
    ANR0136I Table updating statistics performed successfully for 19 of 19.
    ANR0159E dbieval.c(860): Database deadlock detected on 17:15.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).

  8. #8
    Member
    Join Date
    Sep 2007
    Posts
    131
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Quote Originally Posted by Monkeh View Post
    I fear that all the move data processes had an adverse effect on the integrity of my storage pools.

    Today I deleted some inactive version of file from a node, when expiration ran it produced the following errors.

    The size of the file storage pool has not decreased, the node now thinks that the data is still on the TSM server, but fails when I try to delete it again or restore it.

    http://www-01.ibm.com/support/docvie...=utf-8&lang=en

    I'm going to scrap this server and leave TSM6 for a few more versions, until most of these bugs have been resolved.


    ANR0159E dbieval.c(860): Database deadlock detected on 20:27.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).
    ANR0157W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    result code 19 and tracking ID: 0785E620.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 0 is:
    (int32)0.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 1 is:
    (int64)2016394.
    ANR0102E bfdedup.c(7572): Error 19 inserting row in table
    "BF.Dereferenced.Chunks".
    ANR1181E bftxn.c(485): Data storage transaction 0:3456914 was aborted.
    ANR2183W bfdedup.c(8564): Transaction 0:3456914 was aborted.
    ANR0136I Table updating statistics performed successfully for 19 of 19.
    ANR0159E dbieval.c(860): Database deadlock detected on 17:15.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).
    http://www-01.ibm.com/support/docvie...id=swg21408547

    \Masonit

Similar Threads

  1. TSM Data Deduplication
    By pfsubaru in forum TSM Operation
    Replies: 7
    Last Post: 04-01-2010, 03:25 AM
  2. Data Deduplication
    By Donna in forum TSM Operation
    Replies: 2
    Last Post: 11-03-2009, 05:17 PM
  3. Dilligent deduplication
    By denisl in forum Tape / Media Library
    Replies: 2
    Last Post: 07-30-2008, 10:32 PM
  4. Deduplication ratios
    By denisl in forum Tape / Media Library
    Replies: 0
    Last Post: 05-13-2008, 09:14 PM
  5. VTL and Data Deduplication
    By rowl in forum VTL - Virtual Tape Library
    Replies: 0
    Last Post: 06-28-2007, 04:38 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •