1. Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING) Click the link to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This message will disappear after you have made at least 12 posts. Thank you for your cooperation.

Win2K8R2 TSM 6.3 DB performance issue.

Discussion in 'Performance Tuning' started by bimux, Feb 10, 2012.

  1. bimux

    bimux New Member

    Joined:
    Mar 3, 2005
    Messages:
    8
    Likes Received:
    0
    I have a weird problem and I am out of ideas.

    Windows 2008 R2 SP1, Xeon E5645 @2.40GHz (2 Processors), 96GB of RAM, 64 bits
    TSM server 6.3.0 64 bits
    TSM client 6.2.4.0

    When I do a selective backup of 5.58MB_568Files_129Folders I have a slow behavior which points me to a database problem.
    Total number of objects inspected: 699
    Total number of objects backed up: 699
    Total number of objects updated: 0
    Total number of objects rebound: 0
    Total number of objects deleted: 0
    Total number of objects expired: 0
    Total number of objects failed: 0
    Total number of subfile objects: 0
    Total number of bytes inspected: 4.25 MB
    Total number of bytes transferred: 4.25 MB
    Data transfer time: 0.37 sec
    Network data transfer rate: 11,615.57 KB/sec
    Aggregate data transfer rate: 1.67 KB/sec
    Objects compressed by: 0%
    Total data reduction ratio: 0.00%
    Subfile objects reduced by: 0%
    Elapsed processing time: 00:43:27
    The database seems to have a fair amount of disk activity, then all of a sudden it stops. Almost as if a threshold has been reached.

    I did some client and server traces with IBM support, but unless I put this a sev1 there is little change of them looking at this "minor" performance problem. TSM support points me to the Performance Tuning Guide, but I will escalate if I can't find any way out.

    Here is some info that came back from TSM support:
    Most of the time for this transaction was spent in Tm Lock Wait.
    The final statistics for the whole backup show
    DB2 Reg Exec 9859 11785.606 1.195 17.066
    Tm Lock Wait 78 17054.885 218.652 3240.208
    These are the 2 factors that impact the performance .
    The meaning of the parameters is:
    Tm Lock Wait Acquiring transaction manager lock
    DB2 Reg Exec Execute the registered SQL statement
    This indicates that the problem is with the db2 database.
    And this was with a test of 14.4MB_952Files_188Folders selective backup.

    I think there might be an OS issue but I am not sure where to begin.
    A call will probably be opened with MS on Monday.

    Seems also that the problem is present when I move the database to new disks ... (but that may be just slow DB2 restore process). ... I am trying to eliminate any disk related issues by moving the database and logs to different disks ... now in the process to move DB and log to C: ...

    Any ideas are welcome. Please.
     
    Last edited: Feb 10, 2012
  2.  
  3. rore

    rore Senior Member

    Joined:
    Nov 27, 2005
    Messages:
    637
    Likes Received:
    15
    Occupation:
    SysAdmin
    Location:
    Montreal, CA
    Hi,
    Some questions to start:
    How big is your database? What is the storage layout? (FC, SAS?, RAID?, # spindles, etc)
    Are you doing source and/or target deduplication?
    Is your database configured for automatic reorg? what is the schedule?

    Rudy
     
  4. chad_small

    chad_small Moderator

    Joined:
    Dec 17, 2002
    Messages:
    2,197
    Likes Received:
    43
    Occupation:
    AIX/SAN/TSM
    Location:
    Gilbert, AZ
    Here's an idea...can you copy or rename the directory? See if TSM has issues when you run an incremental of the directory (should be a full since the name changed due to the rename or copy).
     
  5. bimux

    bimux New Member

    Joined:
    Mar 3, 2005
    Messages:
    8
    Likes Received:
    0
    DB was on FC drives, moved to SATA, then moved to local SAS drive. So it's not related.
    No Dedup active, anywhere.
    DB size=less than 2 GB ... (new TSM server)
    Automatic reorg by default.
    Problem is not with TSM client. It seems to be on the TSM DB itself.
    Yes the test implies a full backup.
    ******************************
    New evolution of the issue:
    - While having Level2 TSM support online, we had to install ActivePerl to be able to run a data collection script.
    - The perl script is not supposed to do anything other than collect stats. https://www-304.ibm.com/support/docview.wss?uid=swg21432937
    - After the Perl install a selective backup was ran and a 3.5MB 300 files took 15 minutes.
    - At first attempt of collecting data, the problem "disappeared". and the 3.5MB 300 files took 30 seconds. ... WOW ... this is magic.
    - L2 support has no clue what could have caused the change. They suggested something in the DB reorg to be done at a less busy time, but I have no such timeslot. Backups are comming in all the time, and if DB reorg is causing issues, it better get fixed soon.
    - L2 indicated a problem exist with DB reorg and client sessions getting locked and crashing, but I do not have this behavior. (I did not have this behavior)

    For the time being backups have increased dramatically, and the problem is there no more.

    I am lost. (but somewhat happy to have a decent server) ... remains to be seen how much time before the problem reoccurs.
    Thanks to all for they ideas and tips.
     
  6. Stalef

    Stalef New Member

    Joined:
    Mar 12, 2009
    Messages:
    6
    Likes Received:
    0
    Occupation:
    IT-Consultant
    Location:
    Norway
    I had some slow experiences on TSM 6.2.3/6.3 and Win2k8 R2.
    I suspect it got to do something with selftuning parameters in TSM.
    I was playing with different parameters between server and client to achieve the highest possible transfer speeds. Ended up at 94.7MB for a 1Gbps link on the client. A week later it had dropped down to 17-24MB. If I restarted the server it achieved the high speed for a while before it dropped down again. I don't think it's related to the OS Network tuning parameters since CIFS is running on almost full speed every time, while TSM fluctates.
     

Share This Page