Veritas-bu

[Veritas-bu] Issues with NetBackup after applying Solaris patches

2006-04-06 10:14:33
Subject: [Veritas-bu] Issues with NetBackup after applying Solaris patches
From: jlightner AT water DOT com (Jeff Lightner)
Date: Thu, 6 Apr 2006 10:14:33 -0400
This is a multi-part message in MIME format.

------_=_NextPart_001_01C65984.6D5B7653
Content-Type: text/plain;
        charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Haven't used Solaris in a while but this reminds me of a time I did an
update on some Solaris production servers and found the patch included a
default st.conf that overwrote the specific entries we'd made for our
AIT libraries.    Maybe your patch overwrote some key file?

=20

________________________________

From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of Greenberg,
Katherine A
Sent: Thursday, April 06, 2006 9:39 AM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] Issues with NetBackup after applying Solaris
patches

=20

Hey all!=20

We've begun rolling out our annual patch cluster for Solaris (we're
running 8 right now) and as soon as it was installed, we began to
experience some very strange issues with NetBackup.

Environment:=20

Solaris 8 (patch cluster installed was 10/2005). We have also since
patched the CE card and TCP to the latest available patch levels

Sun F15k domain (4x8, single board)=20
4 IP addresses across 4 subnets on both the master/media and the media
server=20
NetBackup 5.0 MP6 for Solaris=20
NetBackup 5.0 MP5 for NDMP=20
Filers are:=20
        NetApp  w/tape attached is Release 6.5.2R1P10  =20
        NetApp w/out tape are Release 6.4.5=20

Issues:=20

1.  All NDMP backups with DAR enabled no longer work. However, if we set
the HIST_FILE =3D N in the policy include configuration, it works fine. =
3
NetApps, same problem with each one. 1 has tape attached thru the SAN
and the others back up thru it.

2.  211, 213 and 195 errors happen nightly on clients. Backups running
thru the master/media itself do not experience issues, however, backups
running thru media servers or SAN media servers will get these errors.
The backups will generally re-run successfully, if they run thru another
media server. SAN Media server backups will fail completely. This
doesn't happen on the same clients every night and some nights doesn't
happen at all.

3.  Backups get sent to the worklist but the PID dies so quickly that
the job looks to be running (bpdbjobs) but the PID isn't active on the
master anymore, nor is there any real activity within Netbackup (no
tapes mounted, not writing going on, no database entries being created,
etc.). The only way to get rid of these *active* jobs is to completely
recycle Netbackup (bp.kill_all).

Steps taken so far:=20

We have disabled all but the PRIMARY interface on the Master and Media
server and are no longer getting 211, 213 and 195 errors, however the
NDMP/w DAR still do not work. We're going to start adding interfaces
back in and see at what point it breaks again.=20

If anyone has seen anything like this can you please let me know if
you've resolved it? We're at backline w/ Sun and about to be escalated
to backline w/ Veritas. =20

Thanks!
Kate=20

________________________________



This e-mail may contain confidential or privileged information. If
you
think you have received this e-mail in error, please advise the
sender by
reply e-mail and then delete this e-mail immediately. Thank you.
Aetna



------_=_NextPart_001_01C65984.6D5B7653
Content-Type: text/html;
        charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40";>

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Issues with NetBackup after applying Solaris patches</title>
<style>
<!--
 /* Font Definitions */
 @font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
        {font-family:Georgia;
        panose-1:2 4 5 2 5 4 5 2 3 3;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
a:link, span.MsoHyperlink
        {color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {color:purple;
        text-decoration:underline;}
p
        {mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman";}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:Arial;
        color:navy;}
@page Section1
        {size:8.5in 11.0in;
        margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
        {page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Haven&#8217;t used Solaris in a =
while but
this reminds me of a time I did an update on some Solaris production =
servers
and found the patch included a default st.conf that overwrote the =
specific
entries we&#8217;d made for our AIT libraries. &nbsp;&nbsp;&nbsp;Maybe =
your
patch overwrote some key file?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'>
veritas-bu-admin AT mailman.eng.auburn DOT edu =
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]
<b><span style=3D'font-weight:bold'>On Behalf Of </span></b>Greenberg, =
Katherine
A<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Thursday, April 06, =
2006
9:39 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b>
veritas-bu AT mailman.eng.auburn DOT edu<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> [Veritas-bu] =
Issues with
NetBackup after applying Solaris patches</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Hey
all!</span></font> <o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>We've
begun rolling out our annual patch cluster for Solaris (we're running 8 =
right
now) and as soon as it was installed, we began to experience some very =
strange
issues with NetBackup.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Environment:</span></font>=

<o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Solaris
8 (patch cluster installed was 10/2005). We have also since patched the =
CE card
and TCP to the latest available patch =
levels</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Sun
F15k domain (4x8, single board)</span></font> <br>
<font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>4
IP addresses across 4 subnets on both the master/media and the media =
server</span></font>
<br>
<font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>NetBackup
5.0 MP6 for Solaris</span></font> <br>
<font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>NetBackup
5.0 MP5 for NDMP</span></font> <br>
<font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Filers
are:</span></font> <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <font size=3D2 =
face=3DGeorgia><span
style=3D'font-size:10.0pt;font-family:Georgia'>NetApp&nbsp; w/tape =
attached is
Release 6.5.2R1P10&nbsp;&nbsp; </span></font><br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <font size=3D2 =
face=3DGeorgia><span
style=3D'font-size:10.0pt;font-family:Georgia'>NetApp w/out tape are =
Release
6.4.5</span></font> <o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Issues:</span></font>
<o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>1.&nbsp;
All NDMP backups with DAR enabled no longer work. However, if we set the =
HIST_FILE
=3D N in the policy include configuration, it works fine. 3 NetApps, =
same problem
with each one. 1 has tape attached thru the SAN and the others back up =
thru it.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>2.&nbsp;
211, 213 and 195 errors happen nightly on clients. Backups running thru =
the
master/media itself do not experience issues, however, backups running =
thru
media servers or SAN media servers will get these errors. The backups =
will
generally re-run successfully, if they run thru another media server. =
SAN Media
server backups will fail completely. This doesn't happen on the same =
clients
every night and some nights doesn't happen at =
all.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>3.&nbsp;
Backups get sent to the worklist but the PID dies so quickly that the =
job looks
to be running (bpdbjobs) but the PID isn't active on the master anymore, =
nor is
there any real activity within Netbackup (no tapes mounted, not writing =
going
on, no database entries being created, etc.). The only way to get rid of =
these
*active* jobs is to completely recycle Netbackup =
(bp.kill_all).</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Steps
taken so far:</span></font> <o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>We
have disabled all but the PRIMARY interface on the Master and Media =
server and
are no longer getting 211, 213 and 195 errors, however the NDMP/w DAR =
still do
not work. We're going to start adding interfaces back in and see at what =
point
it breaks again. </span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>If
anyone has seen anything like this can you please let me know if you've
resolved it? We're at backline w/ Sun and about to be escalated to =
backline w/
Veritas.&nbsp; </span></font><o:p></o:p></p>

<p><font size=3D2 face=3DGeorgia><span =
style=3D'font-size:10.0pt;font-family:Georgia'>Thanks!<br>
Kate</span></font> <o:p></o:p></p>

</div>

</body>

</html>
<HTML><BODY><P><hr size=3D1></P><br>
<P><STRONG><br>
This e-mail may contain confidential or privileged information.  If<br>
you<br>
think you have received this e-mail in error, please advise the<br>
sender by<br>
reply e-mail and then delete this e-mail immediately.  Thank you.<br>
Aetna<br>
</STRONG></P></BODY>
------_=_NextPart_001_01C65984.6D5B7653--