Re: [ADSM-L] Exchange 2010 backup performance

Hi,

I have set send/recv-space at 16MB globally with "no" and use_isno=0 to
override the NICs.  Most important is to set TCPWINODWSIZE=0 on the client
and TSM server and then just let the OS handle it. I had to experiment with
these settings for weeks as we have clients 500km away with high latency(9
ms ping) but with very good throughput(10 Gb fibre.)

As always YMMV, network tuning is a mess and I may have just been lucky.

Yes, client compression was off.

Regards,

Hans Chr.


On Wed, Feb 12, 2014 at 8:13 PM, Schaub, Steve <Steve_Schaub AT bcbst DOT 
com>wrote:

> Hans,
> What tcp-sizes are you using in AIX and on the nic?  When you ran that
> test on the file of zeros, you had client compression turned off, right?  I
> would love to get 200MB single-thread throughput from my clients.
> Thanks,
> -steve
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Hans Christian Riksheim
> Sent: Monday, February 10, 2014 9:04 AM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
> In my experience there is nothing wrong with the TCP stack in Windows.
> Especially Windows2008R2 performs very well. For a single stream from a
> 2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km
> away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup.
> Bottleneck being the drive.
>
> After too much experimenting I have found the critical factor to be to set
> TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes
> in AIX(and override the tcp-settings on the NIC). Windows OS can be left
> alone as its default is quite OK. YMMV of course.
>
> Regards,
>
> Hans Chr.
>
>
> On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <Steve_Schaub AT bcbst DOT com
> >wrote:
>
> > Wanda,
> >
> > I have fought with this problem myself, and here is what I concluded
> > (at least in our environment, YMMV):
> >
> > 1. Running single-stream backups (one db at a time) you will never see
> > the performance you expect, due to the Windows O/S tcpip stack.  I
> > haven't had a chance to stress-test Win2012-R2 yet, but at least
> > through 2008-R2, there seems to be a single-thread constraint that
> > prevents any backup from getting much more than about 20% of the
> bandwidth.
> >
> > 2. The only way to get around this is to do as Del suggests and
> > parallelize your backups.  If you can get 4-6 concurrent jobs running,
> > you can push the network card pretty close to 100%.  The catch, as
> > Dell also pointed out, is that you can't run concurrent backups on
> > databases that live on the same disk (since the vss snap is at the disk
> level).
> >
> > Bottom line is that you would need to divide up your Exchange
> > databases so they are on different disks (or at least, create as many
> > disks as you want to have concurrent backups, then create separate jobs
> to backup each group).
> >
> > Good luck,
> >
> > Steve Schaub
> > System Engineer II, Backup/Recovery
> > Blue Cross Blue Shield of Tennessee
> >
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> > Of Prather, Wanda
> > Sent: Sunday, February 09, 2014 1:08 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Del, you are a national treasure!
> > You are very kind to take time to respond.
> >
> > My backups are already very well balanced, I have 2 servers, the DBA's
> > have the DBs split between them so well that they backup almost the
> > same amount of data, and finish within 30 minutes of each other.
> >  (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape,
> > with /SKIPINTEGRITYCHECK specified.  Exchange DBs coming from V7000
> > disk so should be spiffy speed there.).
> >
> > I tried setting resourceutilization 10 once before, was an impressive
> > failure.  The backup appeared to be looping doing VSS snaps (or rather
> > failing to); I think it was doing as you mentioned in 2 below, trying
> > to snap the same LUN multiple times.
> >
> > Will go through the references you included, then open a performance
> > PMR if no improvement.
> >
> > Thank you so much!
> >
> > W
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> > Of Del Hoobler
> > Sent: Friday, February 07, 2014 6:48 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Hi Wanda,
> >
> > I have a few ideas for you...
> >
> > --------------------------
> >
> > Are you running in a DAG environment? If so, you could do some load
> > balancing between DAG Servers:
> >
> > Most of this in the Exchange book under "Managing Exchange Database
> > Availability Group members by using a single policy":
> >
> >
> >
> > http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.
> > ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
> >
> > The key to "load balance" when setting up the scheduled backup script
> > is to have a separate invocation of each database. For example:
> >
> > TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> >
> > Then, run this command from each of the Exchange servers at or about
> > the same time.
> >
> > --------------------------
> >
> > Here are a few more things to look at:
> >
> > To help with some performance issues, some customers have split their
> > backups into multiple "threads" or "processes" in two ways:
> >
> > 1. Increase the value of the RESOURCEUTILIZATION parameter in the
> >    DSM.OPT file for the DSMAGENT. Trying setting this to "10".
> >     Important: This needs to the DSM.OPT file for the DSMAGENT
> >                not the DP/Exchange options file.
> >
> > 2. Split the backups into multiple parallel instances of the
> >    TDPEXCC backup execution.
> >      i.e. the create separate invocations of DP/Exchange that back
> >      up a different set of databases. For example:
> >                  TDPEXCC BACKUP db1,db2,db3,db4 FULL
> >                  TDPEXCC BACKUP db5,db6,db7,db8 FULL
> >                  TDPEXCC BACKUP db9,db10,db11,db12 FULL
> >       Put these in separate command files and stagger the
> >       launching of them by 10 minutes or so.
> >       The key here is that you need to make sure that you don't
> >       have any LUNs that appears in more than one invocation.
> >       In other words, you don't want to snapshot the
> >       same LUN in separate invocations.
> >
> > Note: The integrity check is a Microsoft tool. IBM has no control over
> > the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL
> > program to perform the integrity check. It's a very I/O intensive
> > program that must examine every page of the database file (.EDB) and all
> log files.
> >
> > --------------------------
> >
> > If none of these help, you should open a PMR to get the performance
> > team to look at your environment.
> >
> >
> >
> > Thank you,
> >
> > Del
> >
> > ----------------------------------------------------
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 02/07/2014
> > 06:04:01 PM:
> >
> > > From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
> > > To: ADSM-L AT vm.marist DOT edu,
> > > Date: 02/07/2014 06:06 PM
> > > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor
> > > Manager" <ADSM-L AT vm.marist DOT edu>
> > >
> > > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in
> > > the baclient dsm.opt?
> > > Or is there anything else I can tweak to improve TSM throughput of a
> > > 2010 full backup?
> > > Got a 10G network, but Exchange full backup performance not impressive.
> > >
> > > Thanks for any ideas  - links to relevant doc also appreciated!
> > >
> > > Wanda
> > >
> > >
> > > **Please note new office phone:
> > > Wanda Prather  |  Senior Technical Specialist  |
> > > Wanda.Prather AT icfi DOT com
> > |
> > > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF
> > > International
> > > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md
> > > |443-718-4900 (o)
> > >
> > -----------------------------------------------------
> > Please see the following link for the BlueCross BlueShield of
> > Tennessee E-mail disclaimer:
> > http://www.bcbst.com/email_disclaimer.shtm
> >
> -----------------------------------------------------
> Please see the following link for the BlueCross BlueShield of Tennessee
> E-mail disclaimer:  http://www.bcbst.com/email_disclaimer.shtm
>