Hans,
What tcp-sizes are you using in AIX and on the nic? When you ran that test on
the file of zeros, you had client compression turned off, right? I would love
to get 200MB single-thread throughput from my clients.
Thanks,
-steve
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Hans Christian Riksheim
Sent: Monday, February 10, 2014 9:04 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Exchange 2010 backup performance
In my experience there is nothing wrong with the TCP stack in Windows.
Especially Windows2008R2 performs very well. For a single stream from a
2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km away
over 10Gig directly to LTO5 has a speed of around 200MB/ at our setup.
Bottleneck being the drive.
After too much experimenting I have found the critical factor to be to set
TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the tcp-sizes in
AIX(and override the tcp-settings on the NIC). Windows OS can be left alone as
its default is quite OK. YMMV of course.
Regards,
Hans Chr.
On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve <Steve_Schaub AT bcbst DOT
com>wrote:
> Wanda,
>
> I have fought with this problem myself, and here is what I concluded
> (at least in our environment, YMMV):
>
> 1. Running single-stream backups (one db at a time) you will never see
> the performance you expect, due to the Windows O/S tcpip stack. I
> haven't had a chance to stress-test Win2012-R2 yet, but at least
> through 2008-R2, there seems to be a single-thread constraint that
> prevents any backup from getting much more than about 20% of the bandwidth.
>
> 2. The only way to get around this is to do as Del suggests and
> parallelize your backups. If you can get 4-6 concurrent jobs running,
> you can push the network card pretty close to 100%. The catch, as
> Dell also pointed out, is that you can't run concurrent backups on
> databases that live on the same disk (since the vss snap is at the disk
> level).
>
> Bottom line is that you would need to divide up your Exchange
> databases so they are on different disks (or at least, create as many
> disks as you want to have concurrent backups, then create separate jobs to
> backup each group).
>
> Good luck,
>
> Steve Schaub
> System Engineer II, Backup/Recovery
> Blue Cross Blue Shield of Tennessee
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Prather, Wanda
> Sent: Sunday, February 09, 2014 1:08 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
> Del, you are a national treasure!
> You are very kind to take time to respond.
>
> My backups are already very well balanced, I have 2 servers, the DBA's
> have the DBs split between them so well that they backup almost the
> same amount of data, and finish within 30 minutes of each other.
> (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape,
> with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000
> disk so should be spiffy speed there.).
>
> I tried setting resourceutilization 10 once before, was an impressive
> failure. The backup appeared to be looping doing VSS snaps (or rather
> failing to); I think it was doing as you mentioned in 2 below, trying
> to snap the same LUN multiple times.
>
> Will go through the references you included, then open a performance
> PMR if no improvement.
>
> Thank you so much!
>
> W
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Del Hoobler
> Sent: Friday, February 07, 2014 6:48 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
> Hi Wanda,
>
> I have a few ideas for you...
>
> --------------------------
>
> Are you running in a DAG environment? If so, you could do some load
> balancing between DAG Servers:
>
> Most of this in the Exchange book under "Managing Exchange Database
> Availability Group members by using a single policy":
>
>
>
> http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.
> ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
>
> The key to "load balance" when setting up the scheduled backup script
> is to have a separate invocation of each database. For example:
>
> TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
> TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>
> Then, run this command from each of the Exchange servers at or about
> the same time.
>
> --------------------------
>
> Here are a few more things to look at:
>
> To help with some performance issues, some customers have split their
> backups into multiple "threads" or "processes" in two ways:
>
> 1. Increase the value of the RESOURCEUTILIZATION parameter in the
> DSM.OPT file for the DSMAGENT. Trying setting this to "10".
> Important: This needs to the DSM.OPT file for the DSMAGENT
> not the DP/Exchange options file.
>
> 2. Split the backups into multiple parallel instances of the
> TDPEXCC backup execution.
> i.e. the create separate invocations of DP/Exchange that back
> up a different set of databases. For example:
> TDPEXCC BACKUP db1,db2,db3,db4 FULL
> TDPEXCC BACKUP db5,db6,db7,db8 FULL
> TDPEXCC BACKUP db9,db10,db11,db12 FULL
> Put these in separate command files and stagger the
> launching of them by 10 minutes or so.
> The key here is that you need to make sure that you don't
> have any LUNs that appears in more than one invocation.
> In other words, you don't want to snapshot the
> same LUN in separate invocations.
>
> Note: The integrity check is a Microsoft tool. IBM has no control over
> the speed of that tool. DP/Exchange invokes the Microsoft ESEUTIL
> program to perform the integrity check. It's a very I/O intensive
> program that must examine every page of the database file (.EDB) and all log
> files.
>
> --------------------------
>
> If none of these help, you should open a PMR to get the performance
> team to look at your environment.
>
>
>
> Thank you,
>
> Del
>
> ----------------------------------------------------
>
> "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 02/07/2014
> 06:04:01 PM:
>
> > From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
> > To: ADSM-L AT vm.marist DOT edu,
> > Date: 02/07/2014 06:06 PM
> > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor
> > Manager" <ADSM-L AT vm.marist DOT edu>
> >
> > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in
> > the baclient dsm.opt?
> > Or is there anything else I can tweak to improve TSM throughput of a
> > 2010 full backup?
> > Got a 10G network, but Exchange full backup performance not impressive.
> >
> > Thanks for any ideas - links to relevant doc also appreciated!
> >
> > Wanda
> >
> >
> > **Please note new office phone:
> > Wanda Prather | Senior Technical Specialist |
> > Wanda.Prather AT icfi DOT com
> |
> > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF
> > International
> > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md
> > |443-718-4900 (o)
> >
> -----------------------------------------------------
> Please see the following link for the BlueCross BlueShield of
> Tennessee E-mail disclaimer:
> http://www.bcbst.com/email_disclaimer.shtm
>
-----------------------------------------------------
Please see the following link for the BlueCross BlueShield of Tennessee E-mail
disclaimer: http://www.bcbst.com/email_disclaimer.shtm
|
