When I'm doing my tests, I'm running
the job only on a single client. I noticed some high latencies on
the guest using the built in tools when running a test job yesterday, so
I'm going to pursue your ideas further. Thanks for your feedback!
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501
jack.forester AT mylan DOT com
Phone: +1.304.554.6039
Cell: +1.412.805.5313
"Martin, Jonathan"
<JMARTI05 AT intersil DOT com>
Sent by: veritas-bu-bounces AT mailman.eng.auburn DOT edu
07/07/2010 02:29 PM
|
To
| Jack.Forester AT mylan DOT com, veritas-bu AT mailman.eng.auburn DOT edu
|
cc
|
|
Subject
| Re: [Veritas-bu] Performance tuning
Windows 2008 client |
|
Are you hitting multiple
VMs simultaneously on the same datastore? Is the speed better when you
only run one backup at a time? We’ve identified serious performance issues
related to requesting to many random I/Os from a raid group as the same
time. Backing up multiple VMs and rebooting multiple VMs on the same datastore
can cause serious latency spikes. Latency is a bit tricky to track down,
but if you open the VC, increase metrics gather to maximum, and select
the host the VM is on you can see the latency for any given lun. Anything
sustained higher than 20ms is a performance problem waiting to happen.
Ideally we shoot for <10ms.
Let me know if you need more
information or help tracking down these metrics.
-Jonathan
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Jack.Forester AT mylan DOT com
Sent: Wednesday, July 07, 2010 1:56 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Performance tuning Windows 2008 client
> Do these contain lots and lots of little files? If so, have
you considered FlashBackup?
The backup is 19GB over 35,000 files. That's pretty typical. It
took 17 hours to run. That's not typical. We hope to upgrade
to NBU7 later this year and take advantage of the new features for backing
up VMs
> Do you have enough memory on these guests. We've seen some issues
where one of our VMware admins decided to give all of the new guests VERY
little memory,
forgetting that after he provisioned the guest he was supposed to increase
it to a reasonable size.
The guests all have 4GB memory.
> If you do network exerciser from the guest to the media server, what
sort of performance do you get? For example, have you tried something
like an FTP or a simple
file copy directly from the guest to the media server?
I copied a folder containing 215MB of files from the guest to the media
server. It took nearly two minutes. A lot slower than I'd have
expected for a 1Gb LAN, so to have a basis of comparason, I copied 206MB
of files from a 2k8 server running on bare metal to the same media server,
and that copy took about half the time. Not stellar performance,
but still better than the VM. This is all repeatable.
What I've observed is that at the start of the backup, we get good performance,
but it slows after 20 minutes or so. The activity also seems to happen
in bursts with periods of good speeds followed by periods of poor performance.
I'm a UNIX administrator by trade, so troubleshooting Windows issues is
a bit foreign to me.
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501
jack.forester AT mylan DOT com
Phone: +1.304.554.6039
Cell: +1.412.805.5313
"Ed Wilts" <ewilts AT ewilts DOT org>
Sent by: veritas-bu-bounces AT mailman.eng.auburn DOT edu
07/07/2010 10:50 AM
|
To
| Jack.Forester AT mylan DOT com
|
cc
| veritas-bu AT mailman.eng.auburn DOT edu
|
Subject
| Re: [Veritas-bu] Performance tuning Windows
2008 client |
|
On Wed, Jul 7, 2010 at 9:28 AM, <Jack.Forester AT mylan DOT com>
wrote:
We're observing some significant performance issues with some of our Windows
2008 SP2 clients.
Backing up to a DD880 VTL, one client in particular is running at just
over 300KB/sec. Others are running 2-3MB/sec.
The clients in question are all virtual machines running on VMWare ESX
3.5.0 238493. Our Windows 2008 clients running on bare metal are
performing well.
Our NetBackup environment is version 6.5.3 on the master/media server and
the client.
We've tried tuning the net buffer size, but that made no difference. Are
there other things we can try?
Do these contain lots and lots of little files? If so, have you considered
FlashBackup?
Do you have enough memory on these guests. We've seen some issues
where one of our VMware admins decided to give all of the new guests VERY
little memory,
forgetting that after he provisioned the guest he was supposed to increase
it to a reasonable size.
If you do network exerciser from the guest to the media server, what sort
of performance do you get? For example, have you tried something
like an FTP or a simple
file copy directly from the guest to the media server?
.../Ed
Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
ewilts AT ewilts DOT org
Linkedin
We have a case open with Symantec, but I thought I'd check here to see
if anyone else has run into this issue.
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501
jack.forester AT mylan DOT com
Phone: +1.304.554.6039
Cell: +1.412.805.5313
==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted
with it may contain legally privileged, proprietary and/or confidential
information intended solely for the use of the addressee. If you
are not the intended recipient, you are hereby notified that any review,
dissemination, distribution, duplication or other use of this message and/or
its attachments is strictly prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies
of the original message and its attachments. Thank you.
==============================================================================
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
==============================================================================
CONFIDENTIALITY NOTICE: This
e-mail message and all attachments transmitted with it may contain legally
privileged, proprietary and/or confidential information intended solely
for the use of the addressee. If you are not the intended recipient,
you are hereby notified that any review, dissemination, distribution, duplication
or other use of this message and/or its attachments is strictly prohibited.
If you are not the intended recipient, please contact the sender
by reply e-mail and destroy all copies of the original message and its
attachments. Thank you.
==============================================================================_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
==============================================================================
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|