Are you hitting multiple VMs simultaneously on the same datastore? Is the speed better when you only run one backup at a time? We’ve identified serious performance issues related to requesting to many random I/Os from a raid group as the same time. Backing up multiple VMs and rebooting multiple VMs on the same datastore can cause serious latency spikes. Latency is a bit tricky to track down, but if you open the VC, increase metrics gather to maximum, and select the host the VM is on you can see the latency for any given lun. Anything sustained higher than 20ms is a performance problem waiting to happen. Ideally we shoot for <10ms.

Let me know if you need more information or help tracking down these metrics.

-Jonathan

> Do these contain lots and lots of little files?  If so, have you considered FlashBackup?

The backup is 19GB over 35,000 files.  That's pretty typical.  It took 17 hours to run.  That's not typical.  We hope to upgrade to NBU7 later this year and take advantage of the new features for backing up VMs

> Do you have enough memory on these guests.  We've seen some issues where one of our VMware admins decided to give all of the new guests VERY little memory,
forgetting that after he provisioned the guest he was supposed to increase it to a reasonable size.

The guests all have 4GB memory.

> If you do network exerciser from the guest to the media server, what sort of performance do you get?  For example, have you tried something like an FTP or a simple
file copy directly from the guest to the media server?

I copied a folder containing 215MB of files from the guest to the media server.  It took nearly two minutes.  A lot slower than I'd have expected for a 1Gb LAN, so to have a basis of comparason, I copied 206MB of files from a 2k8 server running on bare metal to the same media server, and that copy took about half the time.  Not stellar performance, but still better than the VM.  This is all repeatable.

What I've observed is that at the start of the backup, we get good performance, but it slows after 20 minutes or so.  The activity also seems to happen in bursts with periods of good speeds followed by periods of poor performance.

I'm a UNIX administrator by trade, so troubleshooting Windows issues is a bit foreign to me.

--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501

jack.forester AT mylan DOT com

Phone: +1.304.554.6039
Cell: +1.412.805.5313

On Wed, Jul 7, 2010 at 9:28 AM, <Jack.Forester AT mylan DOT com> wrote:

We're observing some significant performance issues with some of our Windows 2008 SP2 clients.  
Backing up to a DD880 VTL, one client in particular is running at just over 300KB/sec.  Others are running 2-3MB/sec.  
The clients in question are all virtual machines running on VMWare ESX 3.5.0 238493.  Our Windows 2008 clients running on bare metal are performing well.   
Our NetBackup environment is version 6.5.3 on the master/media server and the client.

We've tried tuning the net buffer size, but that made no difference.  Are there other things we can try?

Do these contain lots and lots of little files?  If so, have you considered FlashBackup?
Do you have enough memory on these guests.  We've seen some issues where one of our VMware admins decided to give all of the new guests VERY little memory,
forgetting that after he provisioned the guest he was supposed to increase it to a reasonable size.

If you do network exerciser from the guest to the media server, what sort of performance do you get?  For example, have you tried something like an FTP or a simple
file copy directly from the guest to the media server?

   .../Ed

Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
ewilts AT ewilts DOT org
Linkedin

 

We have a case open with Symantec, but I thought I'd check here to see if anyone else has run into this issue.
--
Jack Forester, Jr.
Sr. Data Protection Administrator
Global Technology Services - AHS
Mylan, Inc.
5005 Greenbag Road
Morgantown, WV 26501

jack.forester AT mylan DOT com

Phone: +1.304.554.6039
Cell: +1.412.805.5313

==============================================================================
CONFIDENTIALITY NOTICE:  This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee.  If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited.  If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments.  Thank you.
==============================================================================


_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

==============================================================================

CONFIDENTIALITY NOTICE:  This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee.  If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited.  If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments.  Thank you.

==============================================================================