Wanda, one other thing I would like to point out. With the TSM Server being
on Windows, one other thing to consider is RSS (Receive Side Scaling). All
the processing of IP packets happens on 1 core of the server, and this can
be a potential bottleneck. One way to determine this is to watch the TSM
server when the Exchange Backup occurs. Is there a CPU spike on 1 core? Not
only can you enable RSS to spread the packet processing out amongst more
cores, but you can specify how many cores and which core numbers to start
with. If a perfmon run is done against the Windows Server, it is easy to
see if one particular core is doing more worth than the others, and this
can be a symptom that RSS may provide a benefit.


Dave Canan
TSM Performance
ddcanan AT us.ibm DOT com



On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda <Wanda.Prather AT icfi DOT 
com>wrote:

> Thank you - forgot to mention this is a Windows TSM server.
> I am curious that the drive is the bottleneck - a big file of zeros should
> compress, and give you > 200MB/sec on LTO5, yes?
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Hans Christian Riksheim
> Sent: Monday, February 10, 2014 9:04 AM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
> In my experience there is nothing wrong with the TCP stack in Windows.
> Especially Windows2008R2 performs very well. For a single stream from a
> 2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km
> away over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup.
> Bottleneck being the drive.
>
> After too much experimenting I have found the critical factor to be to set
> TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes
> in AIX(and override the tcp-settings on the NIC). Windows OS can be left
> alone as its default is quite OK. YMMV of course.
>
> Regards,
>
> Hans Chr.
>
>
> On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <Steve_Schaub AT bcbst DOT com
> >wrote:
>
> > Wanda,
> >
> > I have fought with this problem myself, and here is what I concluded
> > (at least in our environment, YMMV):
> >
> > 1. Running single-stream backups (one db at a time) you will never see
> > the performance you expect, due to the Windows O/S tcpip stack.  I
> > haven't had a chance to stress-test Win2012-R2 yet, but at least
> > through 2008-R2, there seems to be a single-thread constraint that
> > prevents any backup from getting much more than about 20% of the
> bandwidth.
> >
> > 2. The only way to get around this is to do as Del suggests and
> > parallelize your backups.  If you can get 4-6 concurrent jobs running,
> > you can push the network card pretty close to 100%.  The catch, as
> > Dell also pointed out, is that you can't run concurrent backups on
> > databases that live on the same disk (since the vss snap is at the disk
> level).
> >
> > Bottom line is that you would need to divide up your Exchange
> > databases so they are on different disks (or at least, create as many
> > disks as you want to have concurrent backups, then create separate jobs
> to backup each group).
> >
> > Good luck,
> >
> > Steve Schaub
> > System Engineer II, Backup/Recovery
> > Blue Cross Blue Shield of Tennessee
> >
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> > Of Prather, Wanda
> > Sent: Sunday, February 09, 2014 1:08 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Del, you are a national treasure!
> > You are very kind to take time to respond.
> >
> > My backups are already very well balanced, I have 2 servers, the DBA's
> > have the DBs split between them so well that they backup almost the
> > same amount of data, and finish within 30 minutes of each other.
> >  (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape,
> > with /SKIPINTEGRITYCHECK specified.  Exchange DBs coming from V7000
> > disk so should be spiffy speed there.).
> >
> > I tried setting resourceutilization 10 once before, was an impressive
> > failure.  The backup appeared to be looping doing VSS snaps (or rather
> > failing to); I think it was doing as you mentioned in 2 below, trying
> > to snap the same LUN multiple times.
> >
> > Will go through the references you included, then open a performance
> > PMR if no improvement.
> >
> > Thank you so much!
> >
> > W
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> > Of Del Hoobler
> > Sent: Friday, February 07, 2014 6:48 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Hi Wanda,
> >
> > I have a few ideas for you...
> >
> > --------------------------
> >
> > Are you running in a DAG environment? If so, you could do some load
> > balancing between DAG Servers:
> >
> > Most of this in the Exchange book under "Managing Exchange Database
> > Availability Group members by using a single policy":
> >
> >
> >
> > http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.
> > ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
> >
> > The key to "load balance" when setting up the scheduled backup script
> > is to have a separate invocation of each database. For example:
> >
> > TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> > TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> >
> > Then, run this command from each of the Exchange servers at or about
> > the same time.
> >
> > --------------------------
> >
> > Here are a few more things to look at:
> >
> > To help with some performance issues, some customers have split their
> > backups into multiple "threads" or "processes" in two ways:
> >
> > 1. Increase the value of the RESOURCEUTILIZATION parameter in the
> >    DSM.OPT file for the DSMAGENT. Trying setting this to "10".
> >     Important: This needs to the DSM.OPT file for the DSMAGENT
> >                not the DP/Exchange options file.
> >
> > 2. Split the backups into multiple parallel instances of the
> >    TDPEXCC backup execution.
> >      i.e. the create separate invocations of DP/Exchange that back
> >      up a different set of databases. For example:
> >                  TDPEXCC BACKUP db1,db2,db3,db4 FULL
> >                  TDPEXCC BACKUP db5,db6,db7,db8 FULL
> >                  TDPEXCC BACKUP db9,db10,db11,db12 FULL
> >       Put these in separate command files and stagger the
> >       launching of them by 10 minutes or so.
> >       The key here is that you need to make sure that you don't
> >       have any LUNs that appears in more than one invocation.
> >       In other words, you don't want to snapshot the
> >       same LUN in separate invocations.
> >
> > Note: The integrity check is a Microsoft tool. IBM has no control over
> > the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL
> > program to perform the integrity check. It's a very I/O intensive
> > program that must examine every page of the database file (.EDB) and all
> log files.
> >
> > --------------------------
> >
> > If none of these help, you should open a PMR to get the performance
> > team to look at your environment.
> >
> >
> >
> > Thank you,
> >
> > Del
> >
> > ----------------------------------------------------
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 02/07/2014
> > 06:04:01 PM:
> >
> > > From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
> > > To: ADSM-L AT vm.marist DOT edu,
> > > Date: 02/07/2014 06:06 PM
> > > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor
> > > Manager" <ADSM-L AT vm.marist DOT edu>
> > >
> > > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in
> > > the baclient dsm.opt?
> > > Or is there anything else I can tweak to improve TSM throughput of a
> > > 2010 full backup?
> > > Got a 10G network, but Exchange full backup performance not impressive.
> > >
> > > Thanks for any ideas  - links to relevant doc also appreciated!
> > >
> > > Wanda
> > >
> > >
> > > **Please note new office phone:
> > > Wanda Prather  |  Senior Technical Specialist  |
> > > Wanda.Prather AT icfi DOT com
> > |
> > > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF
> > > International
> > > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md
> > > |443-718-4900 (o)
> > >
> > -----------------------------------------------------
> > Please see the following link for the BlueCross BlueShield of
> > Tennessee E-mail disclaimer:
> > http://www.bcbst.com/email_disclaimer.shtm
> >
>