We have just build a setup at a large university that replicates (on
average) 6.9TB per hour but that data is deduplicated on the source server.
If you have 10Gb/s+ bandwith and no limitation on the performance on the
source you should but able to handle 7TB per day (mixed workload, not only
tiny files) on a target server if it's an M blueprint or faster model
without any issue.
The specifications of the compute hardware at this customer are less then
what you specify memory wise, we are spot on M blueprint but with NVME's
for the database.
If your source server can deliver the data fast enough it's all about super
fast database and activelog performance on the target, it's needs to do an
insane amount of iop/s to chunk, hash and check all the chunks you are
throwing at it. And yes, memory helps but 256GB isn't as important as
database speed and raw CPU power to plow thru that data.

Did you run benchmarks on the database and activelog volumes on your target
server?
We reach about 110.000 IOP/s on the database volumes using NVME and we have
found that to be the key to unlocking Spectrum Protect ludicrous-mode.

https://imgur.com/a/SAA7OAZ


On Thu, Apr 26, 2018 at 10:37 PM, Skylar Thompson <skylar2 AT uw DOT edu> wrote:

> Are you CPU or disk-bound on the source or target servers? Even if you have
> lots of CPUs, replication might be running on a single thread and just
> using
> one CPU.
>
> On Thu, Apr 26, 2018 at 02:46:24PM -0400, Zoltan Forray wrote:
> > As we get deeper into Replication and my boss wants to use it more and
> more
> > as an offsite recovery platform.
> >
> > As we try to reach "best practices" of replicating everything, we are
> > finding this desire to be difficult if not impossible to achieve due to
> the
> > resource demands.
> >
> > Total we want to eventually replicate is around 700TB from 5-source
> servers
> > to 1-target server which is dedicated to replication.
> >
> > So the big question is, can this be done?
> >
> > We recently rebuilt the offsite target server to as big as we could
> afford
> > ($38K).  It has 256GB of RAM.  64-threads of CPU. Storage is primarily
> > 500TB of ISILON/NFS. Connectivity is via quad 10G (2-for IP traffic from
> > source servers and 2-for ISILON/NFS).
> >
> > Yet we can only replicate around 3TB daily when we backup around 7TB.
> >
> > Looking for suggestions/thoughts/experiences?
> >
> > All boxes are RHEL Linux and 7.1.7.300
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray AT vcu DOT edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://phishing.vcu.edu/
>
> --
> -- Skylar Thompson (skylar2 AT u.washington DOT edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>