ADSM-L

Re: [ADSM-L] Improving Replication performance

2018-04-30 12:38:43
Subject: Re: [ADSM-L] Improving Replication performance
From: Zoltan Forray <zforray AT VCU DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Mon, 30 Apr 2018 12:36:53 -0400
We were lucky to afford the pair of 3TB SSD on the target server, which is
dedicated to the DB.

Since our primary disk these days is ISILON/NFS, I am trying to stuff as
much rotating disk into the servers as they are replaced.  The one we are
about to switch to has 12-10TB internal disk (for LZ/primary storage - not
including OS, actlog, archlog), 72-threads, 256GB RAM and quad 10G.  I
figure the 100TB internal storage will hold 1/3 of this servers onsite
occupancy, allowing me to dedup and start getting away from tape for onsite
storage.

On Fri, Apr 27, 2018 at 3:58 AM, Stefan Folkerts <stefan.folkerts AT gmail DOT 
com>
wrote:

> We have just build a setup at a large university that replicates (on
> average) 6.9TB per hour but that data is deduplicated on the source server.
> If you have 10Gb/s+ bandwith and no limitation on the performance on the
> source you should but able to handle 7TB per day (mixed workload, not only
> tiny files) on a target server if it's an M blueprint or faster model
> without any issue.
> The specifications of the compute hardware at this customer are less then
> what you specify memory wise, we are spot on M blueprint but with NVME's
> for the database.
> If your source server can deliver the data fast enough it's all about super
> fast database and activelog performance on the target, it's needs to do an
> insane amount of iop/s to chunk, hash and check all the chunks you are
> throwing at it. And yes, memory helps but 256GB isn't as important as
> database speed and raw CPU power to plow thru that data.
>
> Did you run benchmarks on the database and activelog volumes on your target
> server?
> We reach about 110.000 IOP/s on the database volumes using NVME and we have
> found that to be the key to unlocking Spectrum Protect ludicrous-mode.
>
> https://imgur.com/a/SAA7OAZ
>
>
> On Thu, Apr 26, 2018 at 10:37 PM, Skylar Thompson <skylar2 AT uw DOT edu> 
> wrote:
>
> > Are you CPU or disk-bound on the source or target servers? Even if you
> have
> > lots of CPUs, replication might be running on a single thread and just
> > using
> > one CPU.
> >
> > On Thu, Apr 26, 2018 at 02:46:24PM -0400, Zoltan Forray wrote:
> > > As we get deeper into Replication and my boss wants to use it more and
> > more
> > > as an offsite recovery platform.
> > >
> > > As we try to reach "best practices" of replicating everything, we are
> > > finding this desire to be difficult if not impossible to achieve due to
> > the
> > > resource demands.
> > >
> > > Total we want to eventually replicate is around 700TB from 5-source
> > servers
> > > to 1-target server which is dedicated to replication.
> > >
> > > So the big question is, can this be done?
> > >
> > > We recently rebuilt the offsite target server to as big as we could
> > afford
> > > ($38K).  It has 256GB of RAM.  64-threads of CPU. Storage is primarily
> > > 500TB of ISILON/NFS. Connectivity is via quad 10G (2-for IP traffic
> from
> > > source servers and 2-for ISILON/NFS).
> > >
> > > Yet we can only replicate around 3TB daily when we backup around 7TB.
> > >
> > > Looking for suggestions/thoughts/experiences?
> > >
> > > All boxes are RHEL Linux and 7.1.7.300
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zforray AT vcu DOT edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more details
> > > visit http://phishing.vcu.edu/
> >
> > --
> > -- Skylar Thompson (skylar2 AT u.washington DOT edu)
> > -- Genome Sciences Department, System Administrator
> > -- Foege Building S046, (206)-685-7354
> > -- University of Washington School of Medicine
> >
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/


ADSM.ORG Privacy and Data Security by KimLaw, PLLC