data domain slow re-hydration (undedup) of exchange datasets with TSM

[donlab]

We've had our 890 in production for over a year now as our onsite VTL pool with TSM and overall its stellar appliance for deduplication. Due to an ongoing legal hold requirement, we need to keep about 700TB (yes, terabytes) of old full exchange backups.

The challenge:

The datadomain is getting full so we are bleeding off TSM's onsite exchange pool to tape in order to free up DataDomain space. However, the tapeout process from DataDomain onsite exchange VTL pool to 3592 FC tape drives is going at a rate of 40 GB / Hr which ties up our drives and is too slow. We usually see these drives get a rate of 250 to 300 GB/Hr during our regular daily offsites.

We stuck a bandaid on the slow tape out process by creating two large 5TB SATA diskpools for TSM. We then crank up as many as 6 tape move processes to each diskpool, moving VTL tape from exchange onsite pool to diskpool. When migration threshold hits 80% in the diskpools TSM will grab a physical tape and spool it off at 250 GB/ Hr. (much better).

So here's the issue, we aren't bleeding this old data off fast enough and DataDomain is choking on re-hydrating this exchange data. They are uncompressed full exchange TDP backups over a year old. Even with 12 VTL to Diskpool tape move processes cranking on TSM, we are avg about 1TB a day to the diskpools. We have tried draining other pools such as our DFS file server domain also under legal hold from VTL to diskpool to PTL and the VTL to diskpool rate was 80 GB/Hr (better). Our daily offsite tape out of last night's backups from DD to PTL flies at 250 / 300GB hour. So we think we've narrowed down the issue to DataDomain not liking old exchange backups.

Has anyone else experienced slow re-hydration of older backups from data domain?
Does anyone have additional insights / scenarios we haven't considered?


Hardware:
Our TSM lpar is running on a power7 frame with
4 dedicated procs / 32GB RAM
TSM 6.3.2
AIX 6.1
4 x 8GB HBA zoned to DataDomain
4 x 4GB HBA zoned to 8 x 3592 tape drives

DataDomain 890 with OS v5.0.x.x
one logical library DDVTL
16,000 slots
100 drives

no replication, old school iron mountain DR

 

[rowl]

This sounds like a problem I had to deal with a few years back. Same thing, legal hold for Exchange backups for the court case that never ended. I had ProtecTIER VTL and not Data Domain, but it was very much the same problem. I could not extract data from the VTL fast enough. What I ended up doing was to redirect new Exchange backups that were covered by this hold to the disk pool, and sending that to physical tape. While in the background left a couple streams from VTL to Disk to Tape spooling along for the old data. Eventually all the legal hold data was on disk pool or tape, and normal data was in my VTL.

The sad thing was, as far as I could tell this data was not recoverable in any practical sense. I asked Exchange admin how long it would take to recover a server, they said 4 hours on a good day. So extend that to, what if the court wants to see all email from June 1 through September 1 of 2007 and it quickly became obvious that they couldn't restore the data fast enough, not to mention they were not 100% certain which version of Exchange was running 5 years ago on that server.

Good Luck!
-Rowl