1. Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING) Click the link to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This message will disappear after you have made at least 12 posts. Thank you for your cooperation.

Deduplication not working in 6.1.2.1?

Discussion in 'TSM Server' started by Monkeh, Dec 2, 2009.

  1. Monkeh

    Monkeh New Member

    Joined:
    May 21, 2004
    Messages:
    55
    Likes Received:
    0
    Occupation:
    www.tectrade.co.uk
    Is anyone running TSM 6.1.2.1 and using Deduplication? I built a test server running 6.1.2.0 and had a dedupe ration of 60%

    Today I upgraded it to 6.1.2.1 and configured a file dev class copypool. I configured it to dedupe and left it running for a couple of hours. When I returned it had completed but has only found 4% of duplicate data? it has the exact same data as the primary storage pool!


    I then created a new 6.1.2.1 server and backed up the same data again and it only found 3% of duplicate data. To me it looks like dedupe is broken in 6.1.2.1. I am going to try downgrading the second server to 6.1.2.0 and see what happens to dedup % after a few backups.

    edit: Im running the servers on Windows 2003 sp2
    Dell 2950 4GB ram 8 cores.

    The test data is file system backups of 8 non-live servers and a static copy of files from a file server.
     
  2.  
  3. Monkeh

    Monkeh New Member

    Joined:
    May 21, 2004
    Messages:
    55
    Likes Received:
    0
    Occupation:
    www.tectrade.co.uk
    I have downgraded to TSM 6.1.2.0. After running reclaim and move data's the deduplication figure has increased again, but I still have a large difference between primary and copy dedup levels.



    tsm: TSM031A>q stg copy* f=d

    Storage Pool Name: COPYPOOL
    Storage Pool Type: Copy
    Device Class Name: COPYFILE
    Estimated Capacity: 244 G
    Space Trigger Util: 91.9
    Pct Util: 24.5
    Pct Migr:
    Pct Logical: 96.8
    High Mig Pct:
    Low Mig Pct:
    Migration Delay:
    Migration Continue:
    Migration Processes:
    Reclamation Processes: 1
    Next Storage Pool:
    Reclaim Storage Pool:
    Maximum Size Threshold:
    Access: Read/Write
    Description:
    Overflow Location:
    Cache Migrated Files?:
    Collocate?: No
    Reclamation Threshold: 100
    Offsite Reclamation Limit: No Limit
    Maximum Scratch Volumes Allowed: 999
    Number of Scratch Volumes Used: 32
    Delay Period for Volume Reuse: 0 Day(s)
    Migration in Progress?:
    Amount Migrated (MB):
    Elapsed Migration Time (seconds):
    Reclamation in Progress?: No
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 12/03/2009 09:46:59
    Storage Pool Data Format: Native
    Copy Storage Pool(s):
    Active Data Pool(s):
    Continue Copy on Error?:
    CRC Data: No
    Reclamation Type: Threshold
    Overwrite Data when Deleted:
    Deduplicate Data?: Yes
    Processes For Identifying Duplicates: 1
    Duplicate Data Not Stored: 12,508 M (17%)


    tsm: TSM031A>q stg file* f=d

    Storage Pool Name: FILEPOOL
    Storage Pool Type: Primary
    Device Class Name: FILE
    Estimated Capacity: 204 G
    Space Trigger Util: 85.6
    Pct Util: 23.0
    Pct Migr: 23.0
    Pct Logical: 65.9
    High Mig Pct: 90
    Low Mig Pct: 70
    Migration Delay: 0
    Migration Continue: Yes
    Migration Processes: 1
    Reclamation Processes: 1
    Next Storage Pool:
    Reclaim Storage Pool:
    Maximum Size Threshold: No Limit
    Access: Read/Write
    Description:
    Overflow Location:
    Cache Migrated Files?:
    Collocate?: No
    Reclamation Threshold: 100
    Offsite Reclamation Limit:
    Maximum Scratch Volumes Allowed: 100
    Number of Scratch Volumes Used: 27
    Delay Period for Volume Reuse: 0 Day(s)
    Migration in Progress?: No
    Amount Migrated (MB): 0.00
    Elapsed Migration Time (seconds): 0
    Reclamation in Progress?: No
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 12/02/2009 14:45:20
    Storage Pool Data Format: Native
    Copy Storage Pool(s):
    Active Data Pool(s):
    Continue Copy on Error?: Yes
    CRC Data: No
    Reclamation Type: Threshold
    Overwrite Data when Deleted:
    Deduplicate Data?: Yes
    Processes For Identifying Duplicates: 1
    Duplicate Data Not Stored: 26,862 M (37%)


    I have the same data in both pools. I have even moved the data off of all the volumes onto to new volumes to ensure that it has been processed by TSM.

    Looking at the Logical_MB reported by the occupancy, I can see that it still has not deduplicated to the same level. Cannot report on Physical_MB as TSM looses this once you enable deduplication

    tsm: TSM031A>run q_storage6

    TYPE GB Files
    ----- ---------------------------------- ------------
    Arch 6.88 1870
    Bkup 89.35 467666

    Total GB Total Files
    ---------------------------------- ------------
    96.24 469536

    Primary_Pool GB Files
    -------------------------------- ---------------------------------- ------------
    FILEPOOL 39.07 234768

    Copy_Pool GB Files
    -------------------------------- ---------------------------------- ------------
    COPYPOOL 57.16 234768

    Total Primary GB Total Primary Files
    ---------------------------------- --------------------
    39.07 234768

    Total Copy GB Total Copy Files
    ---------------------------------- -----------------
    57.16 234768
    ANR1462I RUN: Command script Q_STORAGE6 completed successfully.
     
  4. BBB

    BBB Moderator

    Joined:
    Feb 13, 2007
    Messages:
    2,076
    Likes Received:
    20
    Location:
    Brisbane, Australia
    I thought that when you did a "ba stg" from a deduped pool, to a deduped copypool, tsm would just copy the data as is (not require another deduplication).

    When you did the move data commands, did you add the "reconstruct=yes" flag?

    Can you post a q devc f=d for the two filepools?

    What are you storing in this pool? Is it lots of small files or directories?
     
  5. Monkeh

    Monkeh New Member

    Joined:
    May 21, 2004
    Messages:
    55
    Likes Received:
    0
    Occupation:
    www.tectrade.co.uk
    It certainly seems like it works that way as the identify processes do not report any new deduplicates for the copy pool.

    I am storing filesystem backups from several 2003 servers, and db2 backups from the TCR server.

    No I did not use the reconstruct option


    Device Class Name: FILE
    Device Access Strategy: Sequential
    Storage Pool Count: 1
    Device Type: FILE
    Format: DRIVE
    Est/Max Capacity (MB): 2,048.0
    Mount Limit: 20
    Mount Wait (min):
    Mount Retention (min):
    Label Prefix:
    Drive Letter:
    Library:
    Directory: P:\STGPOOL,L:\STGPOOL
    Server Name:
    Retry Period:
    Retry Interval:
    Twosided:
    Shared:
    High-level Address:
    Minimum Capacity:
    WORM: No
    Drive Encryption:
    Scaled Capacity:
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 19.10.2009 11:28:12

    tsm: TSM031A>q devc copy* f=d

    Device Class Name: COPYFILE
    Device Access Strategy: Sequential
    Storage Pool Count: 1
    Device Type: FILE
    Format: DRIVE
    Est/Max Capacity (MB): 2,048.0
    Mount Limit: 20
    Mount Wait (min):
    Mount Retention (min):
    Label Prefix:
    Drive Letter:
    Library:
    Directory: S:\COPYPOOL
    Server Name:
    Retry Period:
    Retry Interval:
    Twosided:
    Shared:
    High-level Address:
    Minimum Capacity:
    WORM: No
    Drive Encryption:
    Scaled Capacity:
    Last Update by (administrator): ADMIN
    Last Update Date/Time: 02.12.2009 14:41:41
     
  6. BBB

    BBB Moderator

    Joined:
    Feb 13, 2007
    Messages:
    2,076
    Likes Received:
    20
    Location:
    Brisbane, Australia
    Try doing the move datas again, but use recons=yes to reclaim space inside aggregates, this could possible make a big difference to the amount of space reclaimed.
     
  7. Monkeh

    Monkeh New Member

    Joined:
    May 21, 2004
    Messages:
    55
    Likes Received:
    0
    Occupation:
    www.tectrade.co.uk
    moving data between volumes was starting to increase the dedupe value, but was taking too long, so I created a new copy storage pool and ran a backup db. It reached a 53% duplicate data, the primary pool is 49%. it's not exact but I guess that the volume util has an effect on this ratio.

    I may run an update to 6.1.2.1 again just to check what happens. If the ratio drops I will have to log it with IBM
     
  8. Monkeh

    Monkeh New Member

    Joined:
    May 21, 2004
    Messages:
    55
    Likes Received:
    0
    Occupation:
    www.tectrade.co.uk
    I fear that all the move data processes had an adverse effect on the integrity of my storage pools.

    Today I deleted some inactive version of file from a node, when expiration ran it produced the following errors.

    The size of the file storage pool has not decreased, the node now thinks that the data is still on the TSM server, but fails when I try to delete it again or restore it.

    http://www-01.ibm.com/support/docvi...00&uid=swg21408547&loc=en_US&cs=utf-8&lang=en

    I'm going to scrap this server and leave TSM6 for a few more versions, until most of these bugs have been resolved.


    ANR0159E dbieval.c(860): Database deadlock detected on 20:27.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).
    ANR0157W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    result code 19 and tracking ID: 0785E620.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 0 is:
    (int32)0.
    ANR0158W Database operation INSERT for table BF.Dereferenced.Chunks failed with
    operation code 19 and tracking id 0785E620. The data for column 1 is:
    (int64)2016394.
    ANR0102E bfdedup.c(7572): Error 19 inserting row in table
    "BF.Dereferenced.Chunks".
    ANR1181E bftxn.c(485): Data storage transaction 0:3456914 was aborted.
    ANR2183W bfdedup.c(8564): Transaction 0:3456914 was aborted.
    ANR0136I Table updating statistics performed successfully for 19 of 19.
    ANR0159E dbieval.c(860): Database deadlock detected on 17:15.
    ANR0162W Supplemental database diagnostic information: -1:40001:-911
    ([IBM][CLI Driver][DB2/NT] SQL0911N The current transaction has been rolled
    back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
    ).
     
  9. Masonit

    Masonit New Member

    Joined:
    Sep 17, 2007
    Messages:
    131
    Likes Received:
    1
    http://www-01.ibm.com/support/docview.wss?rs=663&tcss=Newsletter&uid=swg21408547

    \Masonit
     

Share This Page