ADSM-L

Re: [ADSM-L] Exchange 2010 backup performance

2014-02-12 13:27:39
Subject: Re: [ADSM-L] Exchange 2010 backup performance
From: Dave Canan <ddcanan AT GMAIL DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 12 Feb 2014 10:22:51 -0800
Wanda, did your test show anything? Let me know if you have any questions.

Dave Canan
TSM Performance 
ddcanan AT us.ibm DOT com

-----Original Message-----
From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
Sent: ‎2/‎10/‎2014 10:19 AM
To: "ADSM-L AT VM.MARIST DOT EDU" <ADSM-L AT VM.MARIST DOT EDU>
Subject: Re: [ADSM-L] Exchange 2010 backup performance

Woot, that's cool to know!
Thanks Dave.
I will run a test tonight and look for a spike.

Is RSS something that is enabled in the NIC or in Windows? 
 I know the network guys already had to do some tweaking of the 10G TOE card.

Wanda


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Dave Canan
Sent: Monday, February 10, 2014 1:07 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Exchange 2010 backup performance

Wanda, one other thing I would like to point out. With the TSM Server being on 
Windows, one other thing to consider is RSS (Receive Side Scaling). All the 
processing of IP packets happens on 1 core of the server, and this can be a 
potential bottleneck. One way to determine this is to watch the TSM server when 
the Exchange Backup occurs. Is there a CPU spike on 1 core? Not only can you 
enable RSS to spread the packet processing out amongst more cores, but you can 
specify how many cores and which core numbers to start with. If a perfmon run 
is done against the Windows Server, it is easy to see if one particular core is 
doing more worth than the others, and this can be a symptom that RSS may 
provide a benefit.


Dave Canan
TSM Performance
ddcanan AT us.ibm DOT com



On Mon, Feb 10, 2014 at 8:16 AM, Prather, Wanda <Wanda.Prather AT icfi DOT 
com>wrote:

> Thank you - forgot to mention this is a Windows TSM server.
> I am curious that the drive is the bottleneck - a big file of zeros 
> should compress, and give you > 200MB/sec on LTO5, yes?
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of Hans Christian Riksheim
> Sent: Monday, February 10, 2014 9:04 AM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
> In my experience there is nothing wrong with the TCP stack in Windows.
> Especially Windows2008R2 performs very well. For a single stream from 
> a
> 2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 
> 500km away over 10Gig directly to LTO5 has a speed of around 200MB/ at our 
> setup.
> Bottleneck being the drive.
>
> After too much experimenting I have found the critical factor to be to 
> set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the 
> tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS 
> can be left alone as its default is quite OK. YMMV of course.
>
> Regards,
>
> Hans Chr.
>
>
> On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <Steve_Schaub AT bcbst DOT com
> >wrote:
>
> > Wanda,
> >
> > I have fought with this problem myself, and here is what I concluded 
> > (at least in our environment, YMMV):
> >
> > 1. Running single-stream backups (one db at a time) you will never 
> > see the performance you expect, due to the Windows O/S tcpip stack.  
> > I haven't had a chance to stress-test Win2012-R2 yet, but at least 
> > through 2008-R2, there seems to be a single-thread constraint that 
> > prevents any backup from getting much more than about 20% of the
> bandwidth.
> >
> > 2. The only way to get around this is to do as Del suggests and 
> > parallelize your backups.  If you can get 4-6 concurrent jobs 
> > running, you can push the network card pretty close to 100%.  The 
> > catch, as Dell also pointed out, is that you can't run concurrent 
> > backups on databases that live on the same disk (since the vss snap 
> > is at the disk
> level).
> >
> > Bottom line is that you would need to divide up your Exchange 
> > databases so they are on different disks (or at least, create as 
> > many disks as you want to have concurrent backups, then create 
> > separate jobs
> to backup each group).
> >
> > Good luck,
> >
> > Steve Schaub
> > System Engineer II, Backup/Recovery
> > Blue Cross Blue Shield of Tennessee
> >
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On 
> > Behalf Of Prather, Wanda
> > Sent: Sunday, February 09, 2014 1:08 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Del, you are a national treasure!
> > You are very kind to take time to respond.
> >
> > My backups are already very well balanced, I have 2 servers, the 
> > DBA's have the DBs split between them so well that they backup 
> > almost the same amount of data, and finish within 30 minutes of each other.
> >  (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape, 
> > with /SKIPINTEGRITYCHECK specified.  Exchange DBs coming from V7000 
> > disk so should be spiffy speed there.).
> >
> > I tried setting resourceutilization 10 once before, was an 
> > impressive failure.  The backup appeared to be looping doing VSS 
> > snaps (or rather failing to); I think it was doing as you mentioned 
> > in 2 below, trying to snap the same LUN multiple times.
> >
> > Will go through the references you included, then open a performance 
> > PMR if no improvement.
> >
> > Thank you so much!
> >
> > W
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On 
> > Behalf Of Del Hoobler
> > Sent: Friday, February 07, 2014 6:48 PM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] Exchange 2010 backup performance
> >
> > Hi Wanda,
> >
> > I have a few ideas for you...
> >
> > --------------------------
> >
> > Are you running in a DAG environment? If so, you could do some load 
> > balancing between DAG Servers:
> >
> > Most of this in the Exchange book under "Managing Exchange Database 
> > Availability Group members by using a single policy":
> >
> >
> >
> > http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.
> > ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
> >
> > The key to "load balance" when setting up the scheduled backup 
> > script is to have a separate invocation of each database. For example:
> >
> > TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE 
> > TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE 
> > TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE 
> > TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE 
> > TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> >
> > Then, run this command from each of the Exchange servers at or about 
> > the same time.
> >
> > --------------------------
> >
> > Here are a few more things to look at:
> >
> > To help with some performance issues, some customers have split 
> > their backups into multiple "threads" or "processes" in two ways:
> >
> > 1. Increase the value of the RESOURCEUTILIZATION parameter in the
> >    DSM.OPT file for the DSMAGENT. Trying setting this to "10".
> >     Important: This needs to the DSM.OPT file for the DSMAGENT
> >                not the DP/Exchange options file.
> >
> > 2. Split the backups into multiple parallel instances of the
> >    TDPEXCC backup execution.
> >      i.e. the create separate invocations of DP/Exchange that back
> >      up a different set of databases. For example:
> >                  TDPEXCC BACKUP db1,db2,db3,db4 FULL
> >                  TDPEXCC BACKUP db5,db6,db7,db8 FULL
> >                  TDPEXCC BACKUP db9,db10,db11,db12 FULL
> >       Put these in separate command files and stagger the
> >       launching of them by 10 minutes or so.
> >       The key here is that you need to make sure that you don't
> >       have any LUNs that appears in more than one invocation.
> >       In other words, you don't want to snapshot the
> >       same LUN in separate invocations.
> >
> > Note: The integrity check is a Microsoft tool. IBM has no control 
> > over the speed of that tool. DP/Exchange invokes the Microsoft 
> > ESEUTIL program to perform the integrity check. It's a very I/O 
> > intensive program that must examine every page of the database file 
> > (.EDB) and all
> log files.
> >
> > --------------------------
> >
> > If none of these help, you should open a PMR to get the performance 
> > team to look at your environment.
> >
> >
> >
> > Thank you,
> >
> > Del
> >
> > ----------------------------------------------------
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 02/07/2014
> > 06:04:01 PM:
> >
> > > From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
> > > To: ADSM-L AT vm.marist DOT edu,
> > > Date: 02/07/2014 06:06 PM
> > > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist 
> > > Stor Manager" <ADSM-L AT vm.marist DOT edu>
> > >
> > > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in 
> > > the baclient dsm.opt?
> > > Or is there anything else I can tweak to improve TSM throughput of 
> > > a
> > > 2010 full backup?
> > > Got a 10G network, but Exchange full backup performance not impressive.
> > >
> > > Thanks for any ideas  - links to relevant doc also appreciated!
> > >
> > > Wanda
> > >
> > >
> > > **Please note new office phone:
> > > Wanda Prather  |  Senior Technical Specialist  | 
> > > Wanda.Prather AT icfi DOT com
> > |
> > > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF 
> > > International
> > > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md
> > > |443-718-4900 (o)
> > >
> > -----------------------------------------------------
> > Please see the following link for the BlueCross BlueShield of 
> > Tennessee E-mail disclaimer:
> > http://www.bcbst.com/email_disclaimer.shtm
> >
>