8.1.12 Dedup Node Tier (Block Size )

[dietmar]

Hi,

Just because i think it is a very important update to let all know .

Container pools do Dedup Tier Levels and the minium Tier Level is 50kb see the new update node command for 8.1.12 :

MINIMUMExtentsize :

Specifies the extent size that is used during data deduplication operations for cloud-container storage pools and directory-container storage pools on this node. In most system environments, the default value of 50 KB is appropriate. However, if you plan to deduplicate data from an Oracle or SAP database, and the average extent size is less than 100 KB, you can help optimize performance by specifying a larger extent size. Data in Oracle and SAP databases is typically deduplicated with extent sizes that are much smaller than the default average size of 256 KB. Small extent sizes can negatively affect the performance of backup and expiration operations and can result in unnecessary growth of the IBM Spectrum Protect server database.

We found out this is a big problem if you are going for TDP for VE. Because all TDP Data is on 50KB ( Because the Server see's only Blocks in 1MB Sizes, and the Client Developer do not talk to the Server Developer) .

So if you store like 100TB of Data on VM Backup's it is getting very ugly to copy/restore them from Tape. Even the Replication is slow in that case. We found that because we use multible Containers and wonder why the performance is so different .

This is of course if you do Blue Print configs with big SATA Drives which gives u around 70 IO/s . So in this case we had about 12000 IOPS in total ( if all Disks are uses in parallel on all LUN's ) which does not work because the SW it is optimized on filling used Containers first ( space vs performance ) .

We have changed the TDP DC Nodes to 250kb and see big improvements in protect and replication performance . We have more VMWARE STG Pool Usage now (+12% ) and the DB Usage did decrease a lot which is also cool. Keep in mind all chunks will be "new" because of the size. So it depends a lot on the copygroup and in case of VMWare i think the last FULL. I expect that the STGPOOL will lower down a bit more, but it went up to +25% strait away as we have done bulk full backups .

Also take in mind if you have copy container pool and want to restore them in case of a disaster. I have seen the restore of a container spreads good to all LUN's attached, but it does help very little if it restores them in 50kb chunks . I have done some real and test disaster restores of Containers and it was always slow if we compared what the Tape Drive could deliver and we thought just stream that sh.t to Disk. No it does not work that way ... I have seen this also on File Level (File Server ) nodes and increased the extend size there as well.

There is a SQL which gives the average size of the junks for a container pool. With this Info + your Backend IOPS it could give a little guess now long it will take to restore . Of course with all other optimium's like DB2 on NVMe , a lot of Cores with high frequency .

db2 "select avg(cast(length as bigint)) from sd_chunk_locations where poolid=XXXXX for read only with ur"
( takes some time to finish, output is in bytes )

poolid = XXX ( show sdpool )

The bottom is of course dedup is not as good anymore and therefore more pool space . But the more Data is much better to process / handle.

In our case having different Pools it was good to check . I have got no Info from IBM Support how to query the average Extend Size used per Node .

This is also a good point if you have an extra large DB2 . Just changing TDP DC Nodes which holds appr 30% of all Data stored reduced the Size from around 275.000.000 to 244.906.466 Pages. ( No problem to handle 3TB+ , just this example ) . The DB2 did also increase to 282.000.000 Pages temp.

Just image 5 time less pointers in the DB2 for TDP Data ( 250 vs 50 ) , and bigger Backup Data chunks to process. I think this is a must have to get a good performing ISP Installation .

Hope this help others.

Br, Dietmar

D&C IT Consulting

 

[Rigido]

Hi Dietmar,
I'm going through a "VTL replacement with DC", SP 8.1.12 (maybe 100 soon) on Power LPAR and AIX 7.2.
The first SP server I'm going to work with is dedicated to SAP HANA backup and some databases are bigger than 1TB. Should I increase ExtentSize? It is not clear if I should set it to 256K.

 

[dietmar]

Hi,

This is only the "minimum Extend size" . So if the Source Data is "bigger" it would not change anything.

I asked IBM Support to have the minimum as a global for all Nodes, because i find it not useful at ALL to save Data in 50kb in any case . But they did not recommend to do in an existing Install .

Currently i see only 1 "negativ" point in changing. Less Dedup and more Space on the Containers. But we see only 12% increase where ALL Data where 50kb in size . So if there are mixxed Tier Level saves in different block sizes i assume it will be less .

As described there are a lot of positive things :

* smaller DB Size
* much faster processing time for protect network/tape ( maybe even for classic backup/restores )

Maybe "someone" have a better idea how sap hana (backint/prole ) saves their files ( size ) and therefore how it ends in the container .

If i would do a new installation, i would go for the 250kb settings for the node .

I also find it a good idea to have multible Containers for different "sources" . Special if you do a protect local to Tape. So at least u could decide what to restore first back from Tape . So all my install's @ Customers do have a dedicated Pool for VMWARE , or Databases .

Also Databases itself do compression and do very little dedup on the ISP Container. So this pool might also be a candidate to disable dedup/compression which also helps ....

If you have a Container Pool already u could check with the mentioned sql above how your average chunk size is ... U even might to start the new ISP Server with standard and do some backup tests - check with the db2 sql and u will see the block size used .....

good luck and let us know .

br, Dietmar

 

[Rigido]

Thank you for your quick response,
we just started with activities and we had first backup on DC last night, I think I can change it with no issues.
Should I run the select from the ISP prompt or from a DB2 command line?
(ISP prompt tells me "System" admin is not authorized...).

Ciao!

 

[dietmar]

This is why we decided to work on an IBM SR . The Protect to was also notable different for this Pool , and therefore a much better candidate to work on. We cannot just repair/restore a Pool for testing again just for the Case . But it will be done again this year . ( with increased extend sizes on the nodes )

ANR4982I The repair storage pool process for XXXXXX on
server XXXXXXX from XXXXXX on server XXXXXXXX is
complete. Extents repaired: 119427858 of 119427858.
Extents failed: 0. Extents skipped: 0. Amount repaired:
6,983 GB of 6,983 GB. Amount failed: 0 bytes. Amount
skipped: 0 bytes. Elapsed time: 0 Days, 8 Hours, 3
Minutes. (SESSION: 30)
ANR0986I Process 1119 for Repair Stgpool running in the
BACKGROUND processed 119,427,858 items for a total of
7,498,253,774,403 bytes with a completion state of
SUCCESS at 01:08:42. (SESSION: 30)

LTO7 - 6 Drives . DB2 on NVMe . Intel Gold Xenon 48 Cores ...

Nobody understands without knowing that it needs 8 hours for 7 TB Restores ...

I have done a restore of a another pool with bigger chunks and better IOPS on the Backend which did 48 TB in 20 hours. ( other Customer ) . ( old 5030 which supported RAID 10 on 6tb sata ) . Which was good i think.

You see the bytes and the items/chunks . So you could just devide bytes by items to get the average chunk size used ... ( check your current protect stgpool, it should also give an idea about the size of the currently protected data as it shows also items/chunks and GB )

During this Desaster Test we saw IOPS up to 11-13K which maxed out the 192x6 TB SATA Total IO .

The Repair/Restore of the Container does not stream . It seems that it takes an amount of Data to process from the DB2 and then works on them until finished and then to start again until all is done ( Audit Table ) .

So also a clean an reorg'ed DB2 helps here as well .....

As i have gone thru hell, i could just tell u .

* Do DR Tests ( having new Hardware is a good candidate to DR Test the existing )
* Use Tape

Br, Dietmar

 

[dietmar]

db2cmd ( or i guess su - db2instance owner on linux/aix )

db2 connect to TSMDB1
db2 set schema TSMDB1
db2 "select ...."

...

i just checked the sap hana pool where i have access to . ( not big )

Deduplication Savings: 7,738 G (26,56%)
Compression Savings: 10,754 G (50,27%)
Total Space Saved: 18,492 G (63,48%)

The current running protect today

ANR4980I The protect storage pool process for STG_XXX on server XXXXXX to STG_XXX on server XXXXX is complete. Extents protected: 645816 of 645816. Extents failed to protect: 0. Extents deleted: 441030 of 441030. Amount protected: 193 GB of 193 GB. Amount failed: 0 bytes. Amount transferred: 188 GB. Elapsed time: 0 Days, 0 Hours, 19 Minutes. (SESSION: 2633535, PROCESS: 4987)

So the average size is appr 320kb here ....
(193 Gigabytes = 207232172032 Bytes / 645816 Protected )

Therefore i guess there will be no or very little change if setting is set to 250kb ....

br, Dietmar

 

[dietmar]

just a small update about Dedup Tier Level update on Container Pool for TDP VE and Restore Container from Tape .

. DB2 still decreasing , avg extent size is currently 164KB and increasing by time , container still hold old Data ... So still on the way but a very good change.

Repair/Restore from Tape has a lot of issues restoring .ncf files . Those non dedup containers could hold up to 1 MIO Extents in a single 1 GB Container ( check with audit container ... ). And those are restored one by one extent as it seems. During restore i saw IO times of 15MB/s only for some time.

So do not store mini files, if you ever want the important data back from tape fast . ( or use a different container ) . This could be 100% different which means 20 or 40 TB Restore/Repair per Day .

To see the differences :
( select count ... containers where container_name like .ncf or .dcf ... )

Pool 1(150kb) : dedup 1344 / 93 non dedup ( this is very ugly dedup non dedup ratio !! -> baclients with high occ and low GB , + system state + 180D retention ... ) -> I restored this one with an avg of 1 TB / hour .
Pool 2(280kb) : dedup 20973 / 30 non dedup ( it is good -> DB Exports )
Pool 3(165kb) : dedup 10040 / 131 non dedup ( this is TDP VE )
( = avg extent size )

At the end MIO of small files are still bad even on containers . Those should be some sort of "auto group'ed on write and transfered in one piece during protect and repair process . Those little more Space would speed up the hole process a lot . ( i think )

As a conclution for this test we will check what Data is really needed and how long it is needed to be kept in the Container Pool . I mean no need to have system state for 180D as an example. Maybe look into the high occ nodes and use some different MC for directories with a lot of files which might not be kept that long ...

Special valid in case of a DR Restore on Containers : Do only Store what u really need ( of course ) . Because u need to get back all first ...

Br, Dietmar

ps. and if u have tape use the DR prepare plan and send it to an external email