Veritas-bu

[Veritas-bu] Serious master issue...

2007-02-14 14:29:23
Subject: [Veritas-bu] Serious master issue...
From: hampus.lind at rps.police.se (Hampus Lind)
Date: Wed, 14 Feb 2007 20:29:23 +0100
I have no idea


 

I have sent tons of bpdbm logs to support and the cant see whats wrong
 And
now the support guy in US says that this is perfectly normal


 

We cant do any larger restore because bpdbm takes all CPU from our master..

 

I am going mad!!

 

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail: hampus.lind at rps.police.se

-----Ursprungligt meddelande-----
Fr?n: Jeff Lightner [mailto:jlightner at water.com] 
Skickat: den 14 februari 2007 20:25
Till: Hampus Lind
?mne: RE: [Veritas-bu] Serious master issue...

 

So you?re running 42 bpdbm processes without any backups running?   What
kicks off that number ? as I noted we only have 5 running.

 

  _____  

From: Hampus Lind [mailto:hampus.lind at rps.police.se] 
Sent: Wednesday, February 14, 2007 2:24 PM
To: Jeff Lightner
Subject: SV: [Veritas-bu] Serious master issue...

 

Okay.. Thanks anyway..

 

We have the db placed on a StorageTek Flexline 380 (LSI Logic`s engenio),
and I don?t really see the problem there, but we also have a DMX 2000, so I
can move netbackup on to that one and see if it solves the problem. But as a
said, I don?t think the problem is on the array.

 

We use LVM and vxfs.

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail: hampus.lind at rps.police.se

-----Ursprungligt meddelande-----
Fr?n: Jeff Lightner [mailto:jlightner at water.com] 
Skickat: den 14 februari 2007 20:16
Till: Hampus Lind
?mne: RE: [Veritas-bu] Serious master issue...

 

The NetBackup storage is all done over fibre to an EMC Symmetrix DMX.   If
you?re on internal drives or something less than a DMX my stats wouldn?t
help because it is top of the line EMC storage (read $Millions) so most of
the performance for I/O is there rather than in box..   Also because we
mount the BCVs I previously mentioned it would take a fair amount of output
(798 disks) to give it to you or a fair amount of work for me to separate
out what was part of /usr/openv from what was part of the BCV mounted DB.

 

By the way we use VxVM rather than LVM.   We don?t stripe at that level ? we
use concatenation.

 

  _____  

From: Hampus Lind [mailto:hampus.lind at rps.police.se] 
Sent: Wednesday, February 14, 2007 2:05 PM
To: Jeff Lightner
Subject: SV: [Veritas-bu] Serious master issue...

 

And how are your CPU usage? 

 

Could you also check a ?sar ?d 1 10? and ?iostat 1?. Do you have busy
netbackup disks?

 

Have you stiped tour netbackup disk at the os level? 

 

 

Thank you very much for helping.. I really appreciate it..

 

 

 

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail: hampus.lind at rps.police.se

-----Ursprungligt meddelande-----
Fr?n: Jeff Lightner [mailto:jlightner at water.com] 
Skickat: den 14 februari 2007 19:57
Till: Hampus Lind
?mne: RE: [Veritas-bu] Serious master issue...

 

5 and 2 respectively at the moment.

 

  _____  

From: Hampus Lind [mailto:hampus.lind at rps.police.se] 
Sent: Wednesday, February 14, 2007 1:39 PM
To: Jeff Lightner
Subject: SV: [Veritas-bu] Serious master issue...

 

Thanks,

 

On your master server
 How man bpdbm and bprd processes are active?

 

 

 

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail: hampus.lind at rps.police.se

-----Ursprungligt meddelande-----
Fr?n: Jeff Lightner [mailto:jlightner at water.com] 
Skickat: den 14 februari 2007 19:35
Till: Hampus Lind; Veritas-bu at mailman.eng.auburn.edu
?mne: RE: [Veritas-bu] Serious master issue...

 

We use an RP4440 as the master.   We also have 18 media servers.   For the
Databases (4 TB in size each) we do the backup using the host on which the
database resides (or BCV to the RP4440 and do the backup from the BCV
mounts) so that all the backups are going fibre from the host to the Library
rather than backing up via the network.   For the hundreds of smaller
clients we have we do backup over the network but most of our windows
clients are on a dedicated backup network for that purpose.

 

  _____  

From: veritas-bu-bounces at mailman.eng.auburn.edu
[mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of Hampus Lind
Sent: Wednesday, February 14, 2007 1:17 PM
To: Veritas-bu at mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Serious master issue...
Importance: High

 

All,

 

Now I have been transferred to USA support
 God bless America!

 

They have told me that they haven?t seen such a big installation in over a
year
. Strange, I have about 200 clients and backup a couple a TB per day..
I was under the impression that this was kinda small installation..??

 

However, they have told me that this is perfectly normal behaviour with
netbackup. That it produces heavy disk IO and eat all CPU power. And I was
really stupid and told them that I also had an case with HP earlier on this
disk IO problem, so now Symantec support are pointing all there fingers at
HP and our disk setup.

 

Our DB is about 60-65 GB and resides on a StorageTek Flexline 380 disk array
(SAN). We run a RAID 5 on 146GB FC drives.. I don?t really see the
bottleneck there, but I will create a RAID 5 on 73GB 15K FC drives just to
shut netbackup support up


 

We run a two CPU HP rp2470  with HP-UX 11.11 as a master server. Shouldn?t
this be enough for this installation?

 

Ooh well
 

 

If support cant help me, what should I do?? I am desperate!!!

 

 

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail: hampus.lind at rps.police.se

-----Ursprungligt meddelande-----
Fr?n: veritas-bu-bounces at mailman.eng.auburn.edu
[mailto:veritas-bu-bounces at mailman.eng.auburn.edu] F?r Hampus Lind
Skickat: den 14 februari 2007 12:48
Till: Veritas-bu at mailman.eng.auburn.edu
?mne: [Veritas-bu] Serious master issue...
Prioritet: H?g

 

Hi,

 

We have a serious issue here with our master server. The problem occurred a
couple of weeks ago, or at least I found out about it then..

 

I was looking at IO`s and scsi queue depth on my master (hp-ux 11.11) when a
say that we had 4000-6000 SCSI commands in que, and a disk utilisation of
100% for the /usr/openv/netbackup/db disk.

 

I have patched hpux to the latest patch bundle and we run NBU 5.1 MP4.

 

HP support sad that bpdbm was leaking memory.

 

Veritas support still investigating.. But we have about 30 bpdbm and bprd
processes active on our master which eats both my CPU`s and produces tons of
IO against our db disk.

 

I actived verbose = 5 on the master, and after 15 minutes the bpdbm log had
reached the file size limit on our filsystem, 2 GB


 

Any one had similar problems? 

 

 

Thanks and regards,

 

Hampus Lind
Rikspolisstyrelsen
National Police Board
Tel dir: +46 (0)8 - 401 99 43
Tel mob: +46 (0)70 - 217 92 66
E-mail:  <mailto:hampus.lind at rps.police.se> hampus.lind at rps.police.se

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20070214/3737e89f/attachment-0001.htm