Over LAN Free, it flies approx 60MBs.

Still have performance problems over IP. The backup threads run at around
15 MBs, but a restore thread runs at 4MBs. Using virtualfsmapping means
the volume can be broken down into multiple threads. However, it is still
clear that per thread the performance especially on restore is horrific.


Ian Smith




Robert Clark <Robert_Clark AT MAC DOT COM>
Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
29/11/2007 05:28
Please respond to
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>


To
ADSM-L AT VM.MARIST DOT EDU
cc

Subject
Re: [ADSM-L] NDMP over IP Performance






I haven't run any throughput tests against NDMP yet, but I'm curious
to see what kind of numbers it can do.

If the TSM server runs on AIX, you can look at the output from
"vmstat -i 1" to see how many interrupts are are being processed per
second.

Going to jumbo packets and using interrupt mitigation can usually
increase overall network throughput.

I wonder what throughput will be like on 10Gbit ethernet?

[RC]

On Nov 20, 2007, at 12:34 PM, Ian-IT Smith wrote:

> Thanks Neil. I greatly appreciate your time on this. Everything you
> have
> said is very useful!!!
>
> In the mean time, we have found an issue that may be the cause. The
> Network switch ports that the backup NIC is set to is configured to
> the
> Flow Control -OFF.
>
> Looking at this we are planning to enable flowcontrol for send and
> recieve. I will keep you update at the performance. I have been
> advised by
> NetApp that 28MB/s is a good performance per volume. Then to
> configure it
> to backup 8 at the same time.
>
> We are using Flexvols...
>
> Ian Smith
>
>
>
>
> "Strand, Neil B." <NBStrand AT LMUS.LEGGMASON DOT COM>
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> 20/11/2007 20:25
> Please respond to
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
>
> To
> ADSM-L AT VM.MARIST DOT EDU
> cc
>
> Subject
> Re: [ADSM-L] NDMP over IP Performance
>
>
>
>
>
>
> Ian,
> Nothing special on the NetApp.
> - The data being backed up is a snapshot of a LUN and there are no NAS
> shares on the filer so there is little contention for ethernet.
> - I do have multiple Gb ethernet interfaces VIFed on the filer.
> - Verify that you are using a Gb interface and not the 100Mb onboard
> interface if you are using a FAS960 or some other ancient filer.
>
> Nothing special on the TSM server.
> - I have virtualfsmap to the snapshot defined.  The snapshot is
> created
> with snapmanager for Exchange once a day.
> - Backups are run from an admin script on the TSM server.
> - I backup about 5TB from 6 filers in 10 hours to a single TSM server
> using NDMP over ethernet.  My bottleneck is the CPU on the TSM server
> which is consistantly pegged at 100% during the backup evolution.
>
> Things to look at:
> 1. Verify that your ethernet settings on both the TSM server and NAS
> filer AND the switch between them are all consistent and correct.
> There
> should be no retransmits.  If there is a router between your filer and
> TSM server, the routing delay may be a factor.  If the filer is a NAS
> filer, there may be contention for ethernet bandwidth.
>
> 2. Take the tape out of the picture by creating a small disk pool of
> known performance and designate it as the target and run a test
> backup.
>
> 3. If your SAN interface is the bottleneck, play around with your
> num_cmd_elems, buffers and max_xfr_size (on AIX).
>
> 4. Ensure your filer is configured for performance.  Are you using
> flex
> vols with the aggregrate spread across as many disks as possible (good
> performance) or traditional vols with a couple of disks (not so good
> performance)?
>
>
>
> Cheers,
> Neil Strand
> Storage Engineer - Legg Mason
> Baltimore, MD.
> (410) 580-7491
> Whatever you can do or believe you can, begin it.
> Boldness has genius, power and magic.
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> Behalf Of
> Ian-IT Smith
> Sent: Tuesday, November 20, 2007 1:27 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] NDMP over IP Performance
>
> That performance is superb!!
>
> To get that, did you go with the defaults, or was any significant
> tuning
> done?
>
> Were there any filer settings that needed to be changed?
>
> Also, was there any network gotchas that you encountered?
>
>
> Thanks so so much!
>
> Ian
>
>
>
> "Strand, Neil B." <NBStrand AT LMUS.LEGGMASON DOT COM> Sent by: "ADSM: Dist
> Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> 20/11/2007 18:15
> Please respond to
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
>
> To
> ADSM-L AT VM.MARIST DOT EDU
> cc
>
> Subject
> Re: [ADSM-L] NDMP over IP Performance
>
>
>
>
>
>
> Ian,
>    I am backing up a NetApp 3070 to a TSM 5.4.1.1 server on a P55a
> with
> AIX 5.3 to a TS7520 VTL over ethernet.
>
> There is a significant ethernet workload on the TSM server because
> I am
> backing up 6 Netapp filers simultaneously.
>
> Sar shows 100% cpu utilization on the TSM server during the processing
> due to the high ethernet load.
>
> I have one 50GB backup that takes 30 minutes (1.6GB/minute)
>
> Make sure you are using TSM 5.4.1.1 or later because it addresses some
> NDMP bugs in 5.4.1.0
>
> You might check with your networking folks to ensure that all ethernet
> settings are correct.
>
>
> Cheers,
> Neil Strand
> Storage Engineer - Legg Mason
> Baltimore, MD.
> (410) 580-7491
> Whatever you can do or believe you can, begin it.
> Boldness has genius, power and magic.
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> Behalf Of
> Ian-IT Smith
> Sent: Tuesday, November 20, 2007 12:55 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: [ADSM-L] NDMP over IP Performance
>
> Hi
>
> I am running a TSM 5.4 Server, trying to backup/restore a NetApp filer
> over IP to a native LTO3 tape storage pool. The backups are NDMP
> images,
> with the TOC. The TOC goes to a separate disk storage pool.
>
> I am experiencing terrible performance. In the region of 2 MBs for
> backup and 0.2MBs restore!!!
>
> Have checked the basic settings tcpnodelay etc and disk performance
> etc...
>
> Can anyone think of anything, (sorry, its expansive!!!) Also any real
> world experience when it comes to performance? expected throughputs
> etc
> And how to get them!!!
>
>
> Ian Smith
>
>
> ---
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail.
> Any
> unauthorized copying, disclosure or distribution of the material in
> this
> e-mail is strictly forbidden.
>
> Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> additional EU corporate and regulatory disclosures.
>
> IMPORTANT:  E-mail sent through the Internet is not secure. Legg Mason
> therefore recommends that you do not send any confidential or
> sensitive
> information to us via electronic mail, including social security
> numbers, account numbers, or personal identification numbers.
> Delivery,
> and or timely delivery of Internet mail is not guaranteed. Legg Mason
> therefore recommends that you do not send time sensitive or
> action-oriented messages to us via electronic mail.
>
> This message is intended for the addressee only and may contain
> privileged or confidential information. Unless you are the intended
> recipient, you may not use, copy or disclose to anyone any information
> contained in this message. If you have received this message in error,
> please notify the author by replying to this message and then kindly
> delete the message.
> Thank you.
>
>
>
> ---
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail.
> Any
> unauthorized copying, disclosure or distribution of the material in
> this
> e-mail is strictly forbidden.
>
> Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> additional EU corporate and regulatory disclosures.
>
> IMPORTANT:  E-mail sent through the Internet is not secure. Legg Mason
> therefore recommends that you do not send any confidential or
> sensitive
> information to us via electronic mail, including social security
> numbers,
> account numbers, or personal identification numbers. Delivery, and or
> timely delivery of Internet mail is not guaranteed. Legg Mason
> therefore
> recommends that you do not send time sensitive
> or action-oriented messages to us via electronic mail.
>
> This message is intended for the addressee only and may contain
> privileged
> or confidential information. Unless you are the intended recipient,
> you
> may not use, copy or disclose to anyone any information contained
> in this
> message. If you have received this message in error, please notify the
> author by replying to this message and then kindly delete the message.
> Thank you.
>
>
>
> ---
>
> This e-mail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this e-mail
> in error) please notify the sender immediately and delete this e-
> mail. Any unauthorized copying, disclosure or distribution of the
> material in this e-mail is strictly forbidden.
>
> Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> additional EU corporate and regulatory disclosures.



---

This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly forbidden.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional 
EU corporate and regulatory disclosures.