Are you hitting multiple VMs simultaneously on the same
datastore? Is the speed better when you only run one backup at a time? We’ve
identified serious performance issues related to requesting to many random I/Os
from a raid group as the same time. Backing up multiple VMs and rebooting multiple
VMs on the same datastore can cause serious latency spikes. Latency is a bit
tricky to track down, but if you open the VC, increase metrics gather to
maximum, and select the host the VM is on you can see the latency for any given
lun. Anything sustained higher than 20ms is a performance problem waiting to
happen. Ideally we shoot for <10ms.
Let me know if you need more information or help tracking down
these metrics.
-Jonathan
From:
veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Jack.Forester AT mylan DOT com
Sent: Wednesday, July 07, 2010 1:56 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Performance tuning Windows 2008 client
> Do these contain lots and lots of little files? If so, have you
considered FlashBackup?
The backup is 19GB over 35,000 files. That's pretty typical. It
took 17 hours to run. That's not typical. We hope to upgrade to
NBU7 later this year and take advantage of the new features for backing up VMs
> Do you have enough memory on these guests. We've seen some issues
where one of our VMware admins decided to give all of the new guests VERY
little memory,
forgetting that after he provisioned the guest he was supposed to increase it
to a reasonable size.
The guests all have 4GB memory.
> If you do network exerciser from the guest to the media server, what sort
of performance do you get? For example, have you tried something like an
FTP or a simple
file copy directly from the guest to the media server?
I copied a folder containing 215MB of files from the guest to the media server.
It took nearly two minutes. A lot slower than I'd have expected for
a 1Gb LAN, so to have a basis of comparason, I copied 206MB of files from a 2k8
server running on bare metal to the same media server, and that copy took about
half the time. Not stellar performance, but still better than the VM.
This is all repeatable.
What I've observed is that at the start of the backup, we get good performance,
but it slows after 20 minutes or so. The activity also seems to happen in
bursts with periods of good speeds followed by periods of poor performance.
I'm a UNIX administrator by trade, so troubleshooting Windows issues is a bit
foreign to me.
--
Jack
Forester, Jr.
Sr. Data
Protection Administrator
Global
Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501
jack.forester AT mylan DOT com
Phone: +1.304.554.6039
Cell:
+1.412.805.5313
"Ed
Wilts" <ewilts AT ewilts DOT org>
Sent by:
veritas-bu-bounces AT mailman.eng.auburn DOT edu
07/07/2010
10:50 AM
|
To
|
Jack.Forester AT mylan DOT com
|
cc
|
veritas-bu AT mailman.eng.auburn DOT edu
|
Subject
|
Re:
[Veritas-bu] Performance tuning Windows 2008 client
|
|
On Wed, Jul 7, 2010 at 9:28 AM, <Jack.Forester AT mylan DOT com>
wrote:
We're observing some significant performance issues with some of our Windows
2008 SP2 clients.
Backing up to a
DD880 VTL, one client in particular is running at just over 300KB/sec.
Others are running 2-3MB/sec.
The clients in
question are all virtual machines running on VMWare ESX 3.5.0 238493. Our
Windows 2008 clients running on bare metal are performing well.
Our NetBackup
environment is version 6.5.3 on the master/media server and the client.
We've tried tuning the net buffer size, but that made no difference. Are
there other things we can try?
Do these contain lots and lots of little files? If so, have you
considered FlashBackup?
Do you have enough memory on these guests. We've seen some issues where
one of our VMware admins decided to give all of the new guests VERY little
memory,
forgetting that after he provisioned the guest he was supposed to increase it
to a reasonable size.
If you do network exerciser from the guest to the media server, what sort of
performance do you get? For example, have you tried something like an FTP
or a simple
file copy directly from the guest to the media server?
.../Ed
Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
ewilts AT ewilts DOT org
Linkedin
We have a case open with Symantec, but I thought I'd check here to see if
anyone else has run into this issue.
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501
jack.forester AT mylan DOT com
Phone: +1.304.554.6039
Cell: +1.412.805.5313
==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments
transmitted with it may contain legally privileged, proprietary and/or
confidential information intended solely for the use of the addressee. If
you are not the intended recipient, you are hereby notified that any review,
dissemination, distribution, duplication or other use of this message and/or
its attachments is strictly prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies of
the original message and its attachments. Thank you.
==============================================================================
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
==============================================================================
CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.
==============================================================================