Container pool performance

[dcabro]

Hello gents, I've been a long-term TSM user and I'm actively maintaining one for the last two years. My company has recently started a project of backup consolidation and the goal was to move backup data out of production storage (V7K) onto a dedicated backup storage (V5K - just purchased) and re-check and tweak all of the backup policies. Our old production server was TSM 7.1.3 running on AIX in active/passive cluster. My idea was to configure new TSM (8.1.5), or ISP if you wish, and slowly move nodes, one by one. However, first ERP (dev system) node to be moved showed significant performance decrease. Full system backup of 400GB, with old TSM (destination was storage pool with file device class) would take 47 minutes to complete. However, with 8.1.5 it takes 2 hours and 15 minutes to do the same job. After about a month of testing and tweaking, (one PMR open, no progress), I have decided to start from scratch and install a fresh TSM 7.1.3 onto a new AIX LPAR, with V5k in the back-end, with file storage pool. Full backup takes 35 minutes. Then I have upgraded that server to 8.1.0 and converted file storage pool to directory/container pool and again, backup takes 2 hours 38 minutes. I'm going to try to upgrade to 8.1.x versions in order, but I don't expect any improvement. What is your experience with container pools when compared to old file device class? Any tweaks that I can do? Thanks!

 

[Trident]

Hi,
I have seen the the trends as you have for a large SQL DB. Backup is much faster using devcl FILE compared to dedup storage. Not sure why though. I have not opened a PMR for the same issue.

Layout seems to be about the same. AIX (7.1/7.2 on S822) and FC based storage on NL drives.

If you have SSD available, it would be interessting to see if the performance is better for containerpool on SSD drives rather than NL.

When you write data to container pools, is the disks (hdiskXX) using 100% of its IO capacity, or is the hdiskXX in use just idling along?

-= Trident =-

 

[dcabro]

I do have SSD available (4x100GB) for the TSM database. I will try with a new hdisk for the cointainer pool but I don't think the problem lies in the storage or disk drives.

I've just run a new full backup and this hdisk is almost idling. It is busy between 1-4%.

I'm seeing this behavior (slower writing to container pool) also for the file system backup, not just DB backup.

 

[dcabro]

It seems that container storage pool has a limit per stream of about 60-80Mb/sec (thank you Uros Gosar from IBM) ! So the solution is to enable parallelism when moving from sequential file storage pool to a container storage pool. With 8 parallel sessions to a compressed container stg, full backup now completes in 22 minutes.
However, it is such a shame that official PMR support still has absolutely no clue on what the root cause of the problem is. Even after so much time since containers have been introduced.

 

[RecoveryOne]

It seems that container storage pool has a limit per stream of about 60-80Mb/sec

Oh wow. I'm using container pools for my VE environment, haven't switched over any other systems yet to use this storage type. Good to know and thank you for the info.

 

[droach]

Can you explain or show exactly what settings you made to improve the container performance? Was it a tweak to the stgp, or was it done on the client?

 

[dcabro]

As this was TDP for ERP (SAP on Oracle backup) the following was done:
On the server, increased client sessions to 10 (maxnummp)
On the client:
- in dsm.sys resourceutilization was increased to 10
- in initSID.utl file, maxsessions, max_back_sessions, max_arch_sessions and sessions was also increased, we used 8 for all (although node was allowed to establish 10 on the server)
- in initSID.sap file, rman_channels = 8

Nothing was done on the stgpool (container-directory).

We see about 2x or more backup speed increase. We haven't reached full backup cycle yet (we keep prod backup 28 days) but at the moment, total data reduction is 77% (compression and deduplication).

Can you explain or show exactly what settings you made to improve the container performance? Was it a tweak to the stgp, or was it done on the client?

 

[Jackal]

Do you have your TCPWINDOWSIZE value in the dsmserv.opt file set to 0?