New Directory storage pools and high CPU usage

[Teet]

Hello.

I'm running TSM 7.1.7 with TSM VE 8.1 and lately I've been trying out the new Directory storage pools but which seem to keep my TSM server CPU at 100% when backing up VM's, this didn't occur with DISK or FILE stgpools. I'm not seeing any high iops/disk, ram or network utilization problems at all, only extremely high CPU usage on the TSM server which seems to be dragging down the overall backup performance which was much better with prementioned older stgpool types. No high resource usage on the TSM VE server either. I even used deduplication with 4 processes (=4 cores on the CPU) when backing up to FILE stgpool.

TSM 7.1.7 server- Dell PE1950 server with 4 core E5410, 20GB RAM

I've already tried to turn off compression on the directory stgpool without any noticeable help but is the new Directory stgpool inline deduplication really that CPU resource heavy that prementioned E5410 will be a bottleneck?

Any thoughts/advice/recommendations etc are very welcome.

Thanks In Advance.

Teet Saar.

 

[marclant]

How big is your daily ingest? That is before dedup and compression.

Trying to figure out in which blueprint you fit in: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli Storage Manager/page/IBM Spectrum Protect Blueprints






The small blueprint has 12 cores.
The medium blueprint has 16 cores.
The large blueprint has 44 cores.

 

[marclant]

I heard there's a "mini" in the works, but I have not heard what daily ingest it will be able to handle and what the specs are.

 

[Teet]

First ingest would be about 3-4TB (just started to backup VM's to Directory storage pool), next ingests would be approximately <500GB, so the amounts aren't that large. But if I understand correctly, the TSM inline dedup is very CPU heavy and I just need more hertz n' cores?

 

[marclant]

the TSM inline dedup is very CPU heavy and I just need more hertz n' cores?

I wouldn't say CPU specific heavy, but rather resource intensive. CPU, memory, database disks, active log disk, etc. You save on storage and post processing, that comes at the cost of more resources.

You could use client-side dedup where possible, if the client sends the data already deduped, the server doesn't need to that work. It just stores the data, it doesn't have to do anything to it.

 

[Teet]

I'm currently using client side dedup when backing up a bunch of VM's but still the TSM server's CPU is at 100% by db2syscs.exe. I haven't really kept my eye on when FILE stgpool were deduped when data was already written to it but it seems that the TSM deduplication is too much for the 4 core E5410.

 

[marclant]

Sometimes, the database layout can cause high CPU util.
- how big is the DB?
- how many database directories?
- is that DB on SSD or 15K disks?

 

[Teet]

Sometimes, the database layout can cause high CPU util.
- how big is the DB?
- how many database directories?
- is that DB on SSD or 15K disks?

Database size and other info:
Database Name: TSMDB1
Total Space of File System (MB): 51,200
Space Used on File System(MB): 45,777
Space Used by Database(MB): 35,232
Free Space Available (MB): 5,423
Total Pages: 3,810,308
Usable Pages: 3,809,532
Used Pages: 3,761,552
Free Pages: 47,980

Currently there are ~30 database directories, and the database is regrettably on SATA, but as I previously mentioned, there arent any high usage on the disks- disk queue length stays well below 0,5 and usage is about 400-500KB/sec by the dv2syscs.exe

Also after starting using the client (TSM VE) side deduplication, the performance seems even worse- TSM CPU usage is still at 100% while the TSM VE resource usage stays pretty much non-existent.

 

[marclant]

Currently there are ~30 database directories, and the database is regrettably on SATA, but as I previously mentioned, there arent any high usage on the disks- disk queue length stays well below 0,5 and usage is about 400-500KB/sec by the dv2syscs.exe

What's the response time in milliseconds?

Also after starting using the client (TSM VE) side deduplication, the performance seems even worse- TSM CPU usage is still at 100% while the TSM VE resource usage stays pretty much non-existent.

There must be something else going on as well, moving to client-side dedup is supposed to move the workload away from the server, so it should help, not make it worse or at the very least not have an impact if it wasn't the cause of the high CPU.

 

[Teet]

Update: the high CPU usage resolved itself, I currently don't know exactly why was db2syscs.exe using all the CPU for (maybe db2 reorg, which was supposed to be CPU heavy) but currently everything seems to be quiet and deduplication also seems to be working, case currently closed.

 

[marclant]

(maybe db2 reorg

If it was inded reorg, you may want to double-check your REORGBEGINTIME and REORGDURATION. You want to run it outside the backup window and outside the database backup. If it overlaps the database backup, it won't cause problems, other than the db backup will pause the reorg, so it's kind of pointless to run it during that time.

The other reorg options have changed with time, might be worth reviewing this: http://www-01.ibm.com/support/docview.wss?uid=swg21967852