Win2K8R2 TSM 6.3 DB performance issue.

[bimux]

I have a weird problem and I am out of ideas.

Windows 2008 R2 SP1, Xeon E5645 @2.40GHz (2 Processors), 96GB of RAM, 64 bits
TSM server 6.3.0 64 bits
TSM client 6.2.4.0

When I do a selective backup of 5.58MB_568Files_129Folders I have a slow behavior which points me to a database problem.
Total number of objects inspected: 699
Total number of objects backed up: 699
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 0
Total number of objects failed: 0
Total number of subfile objects: 0
Total number of bytes inspected: 4.25 MB
Total number of bytes transferred: 4.25 MB
Data transfer time: 0.37 sec
Network data transfer rate: 11,615.57 KB/sec
Aggregate data transfer rate: 1.67 KB/sec
Objects compressed by: 0%
Total data reduction ratio: 0.00%
Subfile objects reduced by: 0%
Elapsed processing time: 00:43:27
The database seems to have a fair amount of disk activity, then all of a sudden it stops. Almost as if a threshold has been reached.

I did some client and server traces with IBM support, but unless I put this a sev1 there is little change of them looking at this "minor" performance problem. TSM support points me to the Performance Tuning Guide, but I will escalate if I can't find any way out.

Here is some info that came back from TSM support:
Most of the time for this transaction was spent in Tm Lock Wait.
The final statistics for the whole backup show
DB2 Reg Exec 9859 11785.606 1.195 17.066
Tm Lock Wait 78 17054.885 218.652 3240.208
These are the 2 factors that impact the performance .
The meaning of the parameters is:
Tm Lock Wait Acquiring transaction manager lock
DB2 Reg Exec Execute the registered SQL statement
This indicates that the problem is with the db2 database.
And this was with a test of 14.4MB_952Files_188Folders selective backup.

I think there might be an OS issue but I am not sure where to begin.
A call will probably be opened with MS on Monday.

Seems also that the problem is present when I move the database to new disks ... (but that may be just slow DB2 restore process). ... I am trying to eliminate any disk related issues by moving the database and logs to different disks ... now in the process to move DB and log to C: ...

Any ideas are welcome. Please.

 

[rore]

Hi,
Some questions to start:
How big is your database? What is the storage layout? (FC, SAS?, RAID?, # spindles, etc)
Are you doing source and/or target deduplication?
Is your database configured for automatic reorg? what is the schedule?

Rudy

 

[chad_small]

Here's an idea...can you copy or rename the directory? See if TSM has issues when you run an incremental of the directory (should be a full since the name changed due to the rename or copy).

 

[bimux]

DB was on FC drives, moved to SATA, then moved to local SAS drive. So it's not related.
No Dedup active, anywhere.
DB size=less than 2 GB ... (new TSM server)
Automatic reorg by default.
Problem is not with TSM client. It seems to be on the TSM DB itself.
Yes the test implies a full backup.
******************************
New evolution of the issue:
- While having Level2 TSM support online, we had to install ActivePerl to be able to run a data collection script.
- The perl script is not supposed to do anything other than collect stats. https://www-304.ibm.com/support/docview.wss?uid=swg21432937
- After the Perl install a selective backup was ran and a 3.5MB 300 files took 15 minutes.
- At first attempt of collecting data, the problem "disappeared". and the 3.5MB 300 files took 30 seconds. ... WOW ... this is magic.
- L2 support has no clue what could have caused the change. They suggested something in the DB reorg to be done at a less busy time, but I have no such timeslot. Backups are comming in all the time, and if DB reorg is causing issues, it better get fixed soon.
- L2 indicated a problem exist with DB reorg and client sessions getting locked and crashing, but I do not have this behavior. (I did not have this behavior)

For the time being backups have increased dramatically, and the problem is there no more.

I am lost. (but somewhat happy to have a decent server) ... remains to be seen how much time before the problem reoccurs.
Thanks to all for they ideas and tips.

 

[Stalef]

I had some slow experiences on TSM 6.2.3/6.3 and Win2k8 R2.
I suspect it got to do something with selftuning parameters in TSM.
I was playing with different parameters between server and client to achieve the highest possible transfer speeds. Ended up at 94.7MB for a 1Gbps link on the client. A week later it had dropped down to 17-24MB. If I restarted the server it achieved the high speed for a while before it dropped down again. I don't think it's related to the OS Network tuning parameters since CIFS is running on almost full speed every time, while TSM fluctates.