Veritas-bu

Re: [Veritas-bu] netbackup master single threaded vs. multithreaded

2009-06-25 19:43:27
Subject: Re: [Veritas-bu] netbackup master single threaded vs. multithreaded
From: "Kevin Corley" <Kevin.Corley AT apollogrp DOT edu>
To: <william.d.brown AT gsk DOT com>, <veritas-bu AT mailman.eng.auburn DOT edu>
Date: Thu, 25 Jun 2009 16:39:50 -0700
William, 
Thank you for the response.

I think staggering jobs is really my only option in the short-term,
which I've already done to a large extent. Due to a strict 'non-business
hours' window, which is ~7500 jobs a night and growing, I'm trying to
defuse a ticking time-bomb. 

The T5220's have been a great media servers for us, we rarely push them
to any limits. I actually made a big push for a couple M4000's as a
replacement master/standby master improve the environments ability to
scale, but management is refusing to deploy anymore SPARC in the
environment in favor of x86. 

As far as IOP requirements to support the Sybase db itself, we ended up
moving the entire /nbu volume to SAN luns striped across 100+ spindles,
which ending up having no effect on the eval timee. I've had an ongoing
case with Symantec open to try and get them to fess-up about the
single-threaded model of the NBRB, but it is simply a limitation of the
product they will have to address in a future release, hopefully 6.5.5!

Good luck with the M4000's, let me know if you see any benefit from
tweaking the Sybase cache usage. It seems to me as long as your max
cache limit is high enough, it will resize in realtime to keep up with
EMM requests.

Thanks,
Kevin

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of
william.d.brown AT gsk DOT com
Sent: Thursday, June 25, 2009 3:04 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] netbackup master single threaded vs.
multithreaded

I'll admit straight up that no, I have no *experience* of this.  However

we did have long discussions with Sun and Symantec about this, as Sun 
raised it as a possible issue as the Sybase ASA is single threaded.

We had decided that on balance the T6320 (which is the same as the T5220

but in a blade format) would be OK as we would be dividing our NetBackup

environment into a low number of separate 'medium sized' domains, which 
related to the DR design.   The 'balance' is that the multi-threading is

very suitable for the numerous daemons that the Master Server runs, and
we 
can have 10GbE directly off the processor using the XAUI pass-through,
so 
it should cope well with the chit-chat on the network.

Our DR plan changed so we went back to a single large domain, and we've 
gone back to an M4000 - faster CPU cores but many fewer threads.  We've 
not implemented it yet so I'll have to take a rain check on how it works

out.  It is a very expensive solution compared to the T5220, but then
one 
M4000 is cheaper than 8 T5220s.

My understanding is that there is a plan to get a multi-threaded Sybase 
'sometime' but I guess a cheap database has cheap features.  As you've
got 
the T5220 I'd persevere, but maybe try and get Sun to help look using 
dtrace to make sure there are no tweaks in the OS - there are some
potted 
solutions on the net, though none I've seen specific to NetBackup.  I'd 
also look at the database quite closely.  Put your DBA hat on.   We plan

to make the changes to the cache that are available as we have tons of 
RAM.  We are separating the database, index, and transaction logs onto 3

separate file systems.  Until we use seriously it I don't know if the
EMM 
database disk performance is an issue - but it is worth looking for fast

performance so use RAID 0+1 not RAID5 etc.  I'd love to try out the Sun 
SSDs on the EMM db!  They say that measured in IOPS they are already 
cheaper than getting the same IOPS with ordinary disks - of course in 
capacity they are still way expensive, but if that was your bottleneck
it 
might be cheaper than another new server.

There is also tuning advice to stagger the start windows of your
backups, 
as the resource broker has a self-limiting ability to find resources for

jobs.  That means that if all jobs have a start window say at 18:00,
some 
will start way past that just because the scheduler hasn't even thought 
about them.  You can see that if you create a vast pile of jobs eligible

to be considered, it is going to take ages for the scheduler to go
through 
the numerous options to pick the winner.  If you can stagger the
windows, 
it will work through the list quicker, and jobs will start more 
predictably near the start of their windows.  Look at page 50 of the 
tuning guide under ?Staggering the submission of jobs for better load 
distribution?.  May mean changing the habits of a lifetime.

William D L Brown


veritas-bu-bounces AT mailman.eng.auburn DOT edu wrote on 25/06/2009 22:07:47:

> Hello,
> Has anyone here had performance problems with 6.5 masters running on
> Solaris with T5220 or T2 Sparc class processors? I have noticed that
> during periods of heavy job load the nbrb evaluation cycle, which 
> appears to handled via a single threaded dbsrv9 process, can take 
> forever and leave jobs stuck between requesting & granted resources 
> states. Sometime for 15 minutes a time, depending on the job load.
> I am familiar with nbrb.conf tweaks, however this doesn't make much 
> difference in my environment where we have many disk-based 
> destinations. Seems like in large environments, the master needs to 
> be on a processor better suited for single threaded performance.
> Thoughts?
> 
> 
> This message is private and confidential. If you have received it in
> error, please notify the sender and remove it from your system.
> 
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu


-----------------------------------------------------------
This e-mail was sent by GlaxoSmithKline Services Unlimited 
(registered in England and Wales No. 1047315), which is a 
member of the GlaxoSmithKline group of companies. The 
registered address of GlaxoSmithKline Services Unlimited 
is 980 Great West Road, Brentford, Middlesex TW8 9GS.
-----------------------------------------------------------

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

This message is private and confidential. If you have received it in error, 
please notify the sender and remove it from your system.
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

<Prev in Thread] Current Thread [Next in Thread>