ADSM-L

Re: [ADSM-L] NDMP over IP Performance

2007-11-20 18:10:53
Subject: Re: [ADSM-L] NDMP over IP Performance
From: Ian-IT Smith <ian-it.smith AT DB DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 20 Nov 2007 20:34:53 +0000
Thanks Neil. I greatly appreciate your time on this. Everything you have
said is very useful!!!

In the mean time, we have found an issue that may be the cause. The
Network switch ports that the backup NIC is set to is configured to the
Flow Control -OFF.

Looking at this we are planning to enable flowcontrol for send and
recieve. I will keep you update at the performance. I have been advised by
NetApp that 28MB/s is a good performance per volume. Then to configure it
to backup 8 at the same time.

We are using Flexvols...

Ian Smith




"Strand, Neil B." <NBStrand AT LMUS.LEGGMASON DOT COM>
Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
20/11/2007 20:25
Please respond to
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>


To
ADSM-L AT VM.MARIST DOT EDU
cc

Subject
Re: [ADSM-L] NDMP over IP Performance






Ian,
Nothing special on the NetApp.
- The data being backed up is a snapshot of a LUN and there are no NAS
shares on the filer so there is little contention for ethernet.
- I do have multiple Gb ethernet interfaces VIFed on the filer.
- Verify that you are using a Gb interface and not the 100Mb onboard
interface if you are using a FAS960 or some other ancient filer.

Nothing special on the TSM server.
- I have virtualfsmap to the snapshot defined.  The snapshot is created
with snapmanager for Exchange once a day.
- Backups are run from an admin script on the TSM server.
- I backup about 5TB from 6 filers in 10 hours to a single TSM server
using NDMP over ethernet.  My bottleneck is the CPU on the TSM server
which is consistantly pegged at 100% during the backup evolution.

Things to look at:
1. Verify that your ethernet settings on both the TSM server and NAS
filer AND the switch between them are all consistent and correct.  There
should be no retransmits.  If there is a router between your filer and
TSM server, the routing delay may be a factor.  If the filer is a NAS
filer, there may be contention for ethernet bandwidth.

2. Take the tape out of the picture by creating a small disk pool of
known performance and designate it as the target and run a test backup.

3. If your SAN interface is the bottleneck, play around with your
num_cmd_elems, buffers and max_xfr_size (on AIX).

4. Ensure your filer is configured for performance.  Are you using flex
vols with the aggregrate spread across as many disks as possible (good
performance) or traditional vols with a couple of disks (not so good
performance)?



Cheers,
Neil Strand
Storage Engineer - Legg Mason
Baltimore, MD.
(410) 580-7491
Whatever you can do or believe you can, begin it.
Boldness has genius, power and magic.


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Ian-IT Smith
Sent: Tuesday, November 20, 2007 1:27 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] NDMP over IP Performance

That performance is superb!!

To get that, did you go with the defaults, or was any significant tuning
done?

Were there any filer settings that needed to be changed?

Also, was there any network gotchas that you encountered?


Thanks so so much!

Ian



"Strand, Neil B." <NBStrand AT LMUS.LEGGMASON DOT COM> Sent by: "ADSM: Dist
Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
20/11/2007 18:15
Please respond to
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>


To
ADSM-L AT VM.MARIST DOT EDU
cc

Subject
Re: [ADSM-L] NDMP over IP Performance






Ian,
   I am backing up a NetApp 3070 to a TSM 5.4.1.1 server on a P55a with
AIX 5.3 to a TS7520 VTL over ethernet.

There is a significant ethernet workload on the TSM server because I am
backing up 6 Netapp filers simultaneously.

Sar shows 100% cpu utilization on the TSM server during the processing
due to the high ethernet load.

I have one 50GB backup that takes 30 minutes (1.6GB/minute)

Make sure you are using TSM 5.4.1.1 or later because it addresses some
NDMP bugs in 5.4.1.0

You might check with your networking folks to ensure that all ethernet
settings are correct.


Cheers,
Neil Strand
Storage Engineer - Legg Mason
Baltimore, MD.
(410) 580-7491
Whatever you can do or believe you can, begin it.
Boldness has genius, power and magic.


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Ian-IT Smith
Sent: Tuesday, November 20, 2007 12:55 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L] NDMP over IP Performance

Hi

I am running a TSM 5.4 Server, trying to backup/restore a NetApp filer
over IP to a native LTO3 tape storage pool. The backups are NDMP images,
with the TOC. The TOC goes to a separate disk storage pool.

I am experiencing terrible performance. In the region of 2 MBs for
backup and 0.2MBs restore!!!

Have checked the basic settings tcpnodelay etc and disk performance
etc...

Can anyone think of anything, (sorry, its expansive!!!) Also any real
world experience when it comes to performance? expected throughputs etc
And how to get them!!!


Ian Smith


---

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures.

IMPORTANT:  E-mail sent through the Internet is not secure. Legg Mason
therefore recommends that you do not send any confidential or sensitive
information to us via electronic mail, including social security
numbers, account numbers, or personal identification numbers. Delivery,
and or timely delivery of Internet mail is not guaranteed. Legg Mason
therefore recommends that you do not send time sensitive or
action-oriented messages to us via electronic mail.

This message is intended for the addressee only and may contain
privileged or confidential information. Unless you are the intended
recipient, you may not use, copy or disclose to anyone any information
contained in this message. If you have received this message in error,
please notify the author by replying to this message and then kindly
delete the message.
Thank you.



---

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for
additional EU corporate and regulatory disclosures.

IMPORTANT:  E-mail sent through the Internet is not secure. Legg Mason
therefore recommends that you do not send any confidential or sensitive
information to us via electronic mail, including social security numbers,
account numbers, or personal identification numbers. Delivery, and or
timely delivery of Internet mail is not guaranteed. Legg Mason therefore
recommends that you do not send time sensitive
or action-oriented messages to us via electronic mail.

This message is intended for the addressee only and may contain privileged
or confidential information. Unless you are the intended recipient, you
may not use, copy or disclose to anyone any information contained in this
message. If you have received this message in error, please notify the
author by replying to this message and then kindly delete the message.
Thank you.



---

This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional 
EU corporate and regulatory disclosures.

<Prev in Thread] Current Thread [Next in Thread>