Hi Sergio!
Yep, we had had some other issues with network disconnects, so we just 2 weeks
ago added NICS to the Exchange servers and put them on the same segment as the
TSM server. No routers, firewalls, hops.
Think my next step is a performance trace.....
W
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Sergio O. Fuentes
Sent: Monday, February 10, 2014 12:41 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Exchange 2010 backup performance
Have you examined the network topology altogether. Our Exchange environment
has network hops all over the place so our networking guys recommended adding a
VLAN tag to both the TSM server and the Exchange DB servers to do backups. In
that way, all hardware firewalls, IPS's, load-balancers, etc. didn't get in the
way of the backup stream. Haven't had to tweak performance on Exchange backups
at all yet. Of course, now that I say that...
SF
On 2/10/14 11:16 AM, "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM> wrote:
>Thank you - forgot to mention this is a Windows TSM server.
>I am curious that the drive is the bottleneck - a big file of zeros
>should compress, and give you > 200MB/sec on LTO5, yes?
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
>Of Hans Christian Riksheim
>Sent: Monday, February 10, 2014 9:04 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: [ADSM-L] Exchange 2010 backup performance
>
>In my experience there is nothing wrong with the TCP stack in Windows.
>Especially Windows2008R2 performs very well. For a single stream from a
>2008R2 client (dsm sel <big file of zeroes>) to an AIX TSM-server 500km
>away over 10Gig directly to LTO5 has a speed of around 200MB/ at our
>setup.
>Bottleneck being the drive.
>
>After too much experimenting I have found the critical factor to be to
>set TCPWINDOWSIZE 0 at both dsm.opt and dsmserv.opt and increase the
>tcp-sizes in AIX(and override the tcp-settings on the NIC). Windows OS
>can be left alone as its default is quite OK. YMMV of course.
>
>Regards,
>
>Hans Chr.
>
>
>On Mon, Feb 10, 2014 at 1:57 PM, Schaub, Steve
><Steve_Schaub AT bcbst DOT com>wrote:
>
>> Wanda,
>>
>> I have fought with this problem myself, and here is what I concluded
>> (at least in our environment, YMMV):
>>
>> 1. Running single-stream backups (one db at a time) you will never
>>see the performance you expect, due to the Windows O/S tcpip stack.
>>I haven't had a chance to stress-test Win2012-R2 yet, but at least
>>through 2008-R2, there seems to be a single-thread constraint that
>>prevents any backup from getting much more than about 20% of the
>>bandwidth.
>>
>> 2. The only way to get around this is to do as Del suggests and
>>parallelize your backups. If you can get 4-6 concurrent jobs running,
>>you can push the network card pretty close to 100%. The catch, as
>>Dell also pointed out, is that you can't run concurrent backups on
>>databases that live on the same disk (since the vss snap is at the
>>disk level).
>>
>> Bottom line is that you would need to divide up your Exchange
>>databases so they are on different disks (or at least, create as many
>>disks as you want to have concurrent backups, then create separate
>>jobs to backup each group).
>>
>> Good luck,
>>
>> Steve Schaub
>> System Engineer II, Backup/Recovery
>> Blue Cross Blue Shield of Tennessee
>>
>>
>> -----Original Message-----
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
>> Of Prather, Wanda
>> Sent: Sunday, February 09, 2014 1:08 PM
>> To: ADSM-L AT VM.MARIST DOT EDU
>> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>>
>> Del, you are a national treasure!
>> You are very kind to take time to respond.
>>
>> My backups are already very well balanced, I have 2 servers, the
>> DBA's have the DBs split between them so well that they backup almost
>> the same amount of data, and finish within 30 minutes of each other.
>> (3.7 TB each, takes 10 hours on a 10G network, direct to LTO5 tape,
>> with /SKIPINTEGRITYCHECK specified. Exchange DBs coming from V7000
>> disk so should be spiffy speed there.).
>>
>> I tried setting resourceutilization 10 once before, was an impressive
>> failure. The backup appeared to be looping doing VSS snaps (or
>> rather failing to); I think it was doing as you mentioned in 2 below,
>> trying to snap the same LUN multiple times.
>>
>> Will go through the references you included, then open a performance
>> PMR if no improvement.
>>
>> Thank you so much!
>>
>> W
>>
>> -----Original Message-----
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
>> Of Del Hoobler
>> Sent: Friday, February 07, 2014 6:48 PM
>> To: ADSM-L AT VM.MARIST DOT EDU
>> Subject: Re: [ADSM-L] Exchange 2010 backup performance
>>
>> Hi Wanda,
>>
>> I have a few ideas for you...
>>
>> --------------------------
>>
>> Are you running in a DAG environment? If so, you could do some load
>> balancing between DAG Servers:
>>
>> Most of this in the Exchange book under "Managing Exchange Database
>> Availability Group members by using a single policy":
>>
>>
>>
>> http://pic.dhe.ibm.com/infocenter/tsminfo/v6r4/index.jsp?topic=%2Fcom.
>> ibm.itsm.mail.exc.doc%2Ft_dpfcm_bup_reduce_redundant_exc.html
>>
>> The key to "load balance" when setting up the scheduled backup script
>> is to have a separate invocation of each database. For example:
>>
>> TDPEXCC BACKUP DB1 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>> TDPEXCC BACKUP DB2 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>> TDPEXCC BACKUP DB3 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>> TDPEXCC BACKUP DB4 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>> TDPEXCC BACKUP DB5 FULL /MINIMUMBACKUPINTERVAL=720 /PREFERDAGPASSIVE
>>
>> Then, run this command from each of the Exchange servers at or about
>> the same time.
>>
>> --------------------------
>>
>> Here are a few more things to look at:
>>
>> To help with some performance issues, some customers have split their
>> backups into multiple "threads" or "processes" in two ways:
>>
>> 1. Increase the value of the RESOURCEUTILIZATION parameter in the
>> DSM.OPT file for the DSMAGENT. Trying setting this to "10".
>> Important: This needs to the DSM.OPT file for the DSMAGENT
>> not the DP/Exchange options file.
>>
>> 2. Split the backups into multiple parallel instances of the
>> TDPEXCC backup execution.
>> i.e. the create separate invocations of DP/Exchange that back
>> up a different set of databases. For example:
>> TDPEXCC BACKUP db1,db2,db3,db4 FULL
>> TDPEXCC BACKUP db5,db6,db7,db8 FULL
>> TDPEXCC BACKUP db9,db10,db11,db12 FULL
>> Put these in separate command files and stagger the
>> launching of them by 10 minutes or so.
>> The key here is that you need to make sure that you don't
>> have any LUNs that appears in more than one invocation.
>> In other words, you don't want to snapshot the
>> same LUN in separate invocations.
>>
>> Note: The integrity check is a Microsoft tool. IBM has no control
>>over the speed of that tool. DP/Exchange invokes the Microsoft
>>ESEUTIL program to perform the integrity check. It's a very I/O
>>intensive program that must examine every page of the database file
>>(.EDB) and all log files.
>>
>> --------------------------
>>
>> If none of these help, you should open a PMR to get the performance
>> team to look at your environment.
>>
>>
>>
>> Thank you,
>>
>> Del
>>
>> ----------------------------------------------------
>>
>> "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 02/07/2014
>> 06:04:01 PM:
>>
>> > From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
>> > To: ADSM-L AT vm.marist DOT edu,
>> > Date: 02/07/2014 06:06 PM
>> > Subject: Exchange 2010 backup performance Sent by: "ADSM: Dist Stor
>> > Manager" <ADSM-L AT vm.marist DOT edu>
>> >
>> > Are Exchange 2010 VSS backups affected by TXNBYTELIMIT settings in
>> > the baclient dsm.opt?
>> > Or is there anything else I can tweak to improve TSM throughput of
>> > a
>> > 2010 full backup?
>> > Got a 10G network, but Exchange full backup performance not
>>impressive.
>> >
>> > Thanks for any ideas - links to relevant doc also appreciated!
>> >
>> > Wanda
>> >
>> >
>> > **Please note new office phone:
>> > Wanda Prather | Senior Technical Specialist |
>> > Wanda.Prather AT icfi DOT com
>> |
>> > www.icfi.com<http://www.icfi.com> | 410-868-4872 (m) ICF
>> > International
>> > | 7125 Thomas Edison Dr., Suite 100, Columbia, Md
>> > |443-718-4900 (o)
>> >
>> -----------------------------------------------------
>> Please see the following link for the BlueCross BlueShield of
>> Tennessee E-mail disclaimer:
>> http://www.bcbst.com/email_disclaimer.shtm
>>
|
