Veritas-bu

[Veritas-bu] Issues with NetBackup after applying Solaris patches

2006-04-06 10:21:12
Subject: [Veritas-bu] Issues with NetBackup after applying Solaris patches
From: GreenbergKA AT aetna DOT com (Greenberg, Katherine A)
Date: Thu, 6 Apr 2006 10:21:12 -0400
This is a multi-part message in MIME format.


------_=_NextPart_001_01C65985.5BA256D4
Content-Type: text/plain;
        charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Don't think that is it, but who knows=2E I would hope that w/ Sun and=0D=0A=
Veritas both heavily on the case someone would've come up with that by=0D=
=0Anow ;)=0D=0A =0D=0A =0D=0A-----Original Message-----=0D=0AFrom: Jeff Lig=
htner [mailto:jlightner@water=2Ecom] =0D=0ASent: Thursday, April 06, 2006 1=
0:15 AM=0D=0ATo: Greenberg, Katherine A; veritas-bu@mailman=2Eeng=2Eauburn=
=2Eedu=0D=0ASubject: RE: [Veritas-bu] Issues with NetBackup after applying =
Solaris=0D=0Apatches=0D=0A=0D=0A=0D=0A=0D=0AHaven't used Solaris in a while=
 but this reminds me of a time I did an=0D=0Aupdate on some Solaris product=
ion servers and found the patch included a=0D=0Adefault st=2Econf that over=
wrote the specific entries we'd made for our=0D=0AAIT libraries=2E    Maybe=
 your patch overwrote some key file?=0D=0A=0D=0A =0D=0A=0D=0A______________=
__________________=0D=0A=0D=0AFrom: veritas-bu-admin@mailman=2Eeng=2Eauburn=
=2Eedu=0D=0A[mailto:veritas-bu-admin@mailman=2Eeng=2Eauburn=2Eedu] On Behal=
f Of Greenberg,=0D=0AKatherine A=0D=0ASent: Thursday, April 06, 2006 9:39 A=
M=0D=0ATo: veritas-bu@mailman=2Eeng=2Eauburn=2Eedu=0D=0ASubject: [Veritas-b=
u] Issues with NetBackup after applying Solaris=0D=0Apatches=0D=0A=0D=0A =
=0D=0A=0D=0AHey all! =0D=0A=0D=0AWe've begun rolling out our annual patch c=
luster for Solaris (we're=0D=0Arunning 8 right now) and as soon as it was i=
nstalled, we began to=0D=0Aexperience some very strange issues with NetBack=
up=2E=0D=0A=0D=0AEnvironment: =0D=0A=0D=0ASolaris 8 (patch cluster installe=
d was 10/2005)=2E We have also since=0D=0Apatched the CE card and TCP to th=
e latest available patch levels=0D=0A=0D=0ASun F15k domain (4x8, single boa=
rd) =0D=0A4 IP addresses across 4 subnets on both the master/media and the =
media=0D=0Aserver =0D=0ANetBackup 5=2E0 MP6 for Solaris =0D=0ANetBackup 5=
=2E0 MP5 for NDMP =0D=0AFilers are: =0D=0A        NetApp  w/tape attached i=
s Release 6=2E5=2E2R1P10   =0D=0A        NetApp w/out tape are Release 6=2E=
4=2E5 =0D=0A=0D=0AIssues: =0D=0A=0D=0A1=2E  All NDMP backups with DAR enabl=
ed no longer work=2E However, if we set=0D=0Athe HIST_FILE =3D N in the pol=
icy include configuration, it works fine=2E 3=0D=0ANetApps, same problem wi=
th each one=2E 1 has tape attached thru the SAN=0D=0Aand the others back up=
 thru it=2E=0D=0A=0D=0A2=2E  211, 213 and 195 errors happen nightly on clie=
nts=2E Backups running=0D=0Athru the master/media itself do not experience =
issues, however, backups=0D=0Arunning thru media servers or SAN media serve=
rs will get these errors=2E=0D=0AThe backups will generally re-run successf=
ully, if they run thru another=0D=0Amedia server=2E SAN Media server backup=
s will fail completely=2E This=0D=0Adoesn't happen on the same clients ever=
y night and some nights doesn't=0D=0Ahappen at all=2E=0D=0A=0D=0A3=2E  Back=
ups get sent to the worklist but the PID dies so quickly that=0D=0Athe job =
looks to be running (bpdbjobs) but the PID isn't active on the=0D=0Amaster =
anymore, nor is there any real activity within Netbackup (no=0D=0Atapes mou=
nted, not writing going on, no database entries being created,=0D=0Aetc=2E)=
=2E The only way to get rid of these *active* jobs is to completely=0D=0Are=
cycle Netbackup (bp=2Ekill_all)=2E=0D=0A=0D=0ASteps taken so far: =0D=0A=0D=
=0AWe have disabled all but the PRIMARY interface on the Master and Media=
=0D=0Aserver and are no longer getting 211, 213 and 195 errors, however the=
=0D=0ANDMP/w DAR still do not work=2E We're going to start adding interface=
s=0D=0Aback in and see at what point it breaks again=2E =0D=0A=0D=0AIf anyo=
ne has seen anything like this can you please let me know if=0D=0Ayou've re=
solved it? We're at backline w/ Sun and about to be escalated=0D=0Ato backl=
ine w/ Veritas=2E  =0D=0A=0D=0AThanks!=0D=0AKate =0D=0A=0D=0A______________=
__________________=0D=0A=0D=0A=0D=0A=0D=0AThis e-mail may contain confident=
ial or privileged information=2E If=0D=0Ayou=0D=0Athink you have received t=
his e-mail in error, please advise the=0D=0Asender by=0D=0Areply e-mail and=
 then delete this e-mail immediately=2E Thank you=2E=0D=0AAetna=0D=0A=0D=0A=
=0D=0A=0D=0A
------_=_NextPart_001_01C65985.5BA256D4
Content-Type: text/html;
        charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4=2E0 Transitional//EN">=0D=0A<HTML=
 xmlns=3D"http://www=2Ew3=2Eorg/TR/REC-html40"; xmlns:v =3D =0D=0A"urn:schem=
as-microsoft-com:vml" xmlns:o =3D =0D=0A"urn:schemas-microsoft-com:office:o=
ffice" xmlns:w =3D =0D=0A"urn:schemas-microsoft-com:office:word"><HEAD><TIT=
LE>Message</TITLE>=0D=0A<META http-equiv=3DContent-Type content=3D"text/htm=
l; charset=3Dus-ascii">=0D=0A<META content=3D"MSHTML 6=2E00=2E2800=2E1529" =
name=3DGENERATOR><!--[if !mso]>=0D=0A<STYLE>=0D=0Av\:* {behavior:url(#defau=
lt#VML);}=0D=0Ao\:* {behavior:url(#default#VML);}=0D=0Aw\:* {behavior:url(#=
default#VML);}=0D=0A=2E=2Eshape {behavior:url(#default#VML);}=0D=0A</STYLE>=
=0D=0A<![endif]-->=0D=0A<STYLE>=0D=0A<!--=0D=0A /* Font Definitions */=0D=
=0A @font-face=0D=0A    {font-family:Tahoma;=0D=0A      panose-1:2 11 6 4 3 5 4 
4 2=
 4;}=0D=0A@font-face=0D=0A      {font-family:Georgia;=0D=0A     panose-1:2 4 5 
2 5 4=
 5 2 3 3;}=0D=0A /* Style Definitions */=0D=0A p=2EMsoNormal, li=2EMsoNorma=
l, div=2EMsoNormal=0D=0A        {margin:0in;=0D=0A      
margin-bottom:=2E0001pt;=0D=0A  =
font-size:12=2E0pt;=0D=0A       font-family:"Times New Roman";}=0D=0Aa:link, 
span=
=2EMsoHyperlink=0D=0A   {color:blue;=0D=0A      
text-decoration:underline;}=0D=0Aa=
:visited, span=2EMsoHyperlinkFollowed=0D=0A     {color:purple;=0D=0A    
text-decor=
ation:underline;}=0D=0Ap=0D=0A  {mso-margin-top-alt:auto;=0D=0A margin-right=
:0in;=0D=0A     mso-margin-bottom-alt:auto;=0D=0A       margin-left:0in;=0D=0A  
font-s=
ize:12=2E0pt;=0D=0A     font-family:"Times New Roman";}=0D=0Aspan=2EEmailStyle1=
8=0D=0A {mso-style-type:personal-reply;=0D=0A   font-family:Arial;=0D=0A        
colo=
r:navy;}=0D=0A@page Section1=0D=0A      {size:8=2E5in 11=2E0in;=0D=0A   
margin:1=
=2E0in 1=2E25in 1=2E0in 1=2E25in;}=0D=0Adiv=2ESection1=0D=0A    {page:Section1=
;}=0D=0A-->=0D=0A</STYLE>=0D=0A</HEAD>=0D=0A<BODY lang=3DEN-US vLink=3Dpurp=
le link=3Dblue>=0D=0A<DIV><SPAN class=3D593252014-06042006><FONT face=3DGeo=
rgia color=3D#0000ff =0D=0Asize=3D2>Don't think that is it, but who knows=
=2E I would hope that w/ Sun and =0D=0AVeritas both heavily on the case som=
eone would've come up with that by now =0D=0A;)</FONT></SPAN></DIV>=0D=0A<D=
IV><SPAN class=3D593252014-06042006><FONT face=3DGeorgia color=3D#0000ff =
=0D=0Asize=3D2></FONT></SPAN>&nbsp;</DIV>=0D=0A<DIV><SPAN class=3D593252014=
-06042006></SPAN>&nbsp;</DIV>=0D=0A<DIV></DIV>=0D=0A<DIV><FONT face=3DTahom=
a size=3D2>-----Original Message-----<BR><B>From:</B> Jeff =0D=0ALightner [=
mailto:jlightner@water=2Ecom] <BR><B>Sent:</B> Thursday, April 06, 2006 =0D=
=0A10:15 AM<BR><B>To:</B> Greenberg, Katherine A; =0D=0Averitas-bu@mailman=
=2Eeng=2Eauburn=2Eedu<BR><B>Subject:</B> RE: [Veritas-bu] Issues =0D=0Awith=
 NetBackup after applying Solaris patches<BR><BR></DIV></FONT>=0D=0A<DIV cl=
ass=3DSection1>=0D=0A<P class=3DMsoNormal><FONT face=3DArial color=3Dnavy s=
ize=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Ari=
al">Haven&#8217;t used Solaris in =0D=0Aa while but this reminds me of a ti=
me I did an update on some Solaris production =0D=0Aservers and found the p=
atch included a default st=2Econf that overwrote the =0D=0Aspecific entries=
 we&#8217;d made for our AIT libraries=2E &nbsp;&nbsp;&nbsp;Maybe your =0D=
=0Apatch overwrote some key file?<o:p></o:p></SPAN></FONT></P>=0D=0A<P clas=
s=3DMsoNormal><FONT face=3DArial color=3Dnavy size=3D2><SPAN =0D=0Astyle=3D=
"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN>=
</FONT></P>=0D=0A<DIV>=0D=0A<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: cen=
ter" align=3Dcenter><FONT =0D=0Aface=3D"Times New Roman" size=3D3><SPAN sty=
le=3D"FONT-SIZE: 12pt">=0D=0A<HR tabIndex=3D-1 align=3Dcenter width=3D"100%=
" SIZE=3D2>=0D=0A</SPAN></FONT></DIV>=0D=0A<P class=3DMsoNormal><B><FONT fa=
ce=3DTahoma size=3D2><SPAN =0D=0Astyle=3D"FONT-WEIGHT: bold; FONT-SIZE: 10p=
t; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT =0D=0Aface=3DTahoma si=
ze=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> =0D=0Averitas-=
bu-admin@mailman=2Eeng=2Eauburn=2Eedu =0D=0A[mailto:veritas-bu-admin@mailma=
n=2Eeng=2Eauburn=2Eedu] <B><SPAN =0D=0Astyle=3D"FONT-WEIGHT: bold">On Behal=
f Of </SPAN></B>Greenberg, Katherine =0D=0AA<BR><B><SPAN style=3D"FONT-WEIG=
HT: bold">Sent:</SPAN></B> Thursday, April 06, =0D=0A2006 9:39 AM<BR><B><SP=
AN style=3D"FONT-WEIGHT: bold">To:</SPAN></B> =0D=0Averitas-bu@mailman=2Een=
g=2Eauburn=2Eedu<BR><B><SPAN =0D=0Astyle=3D"FONT-WEIGHT: bold">Subject:</SP=
AN></B> [Veritas-bu] Issues with NetBackup =0D=0Aafter applying Solaris pat=
ches</SPAN></FONT><o:p></o:p></P></DIV>=0D=0A<P class=3DMsoNormal><FONT fac=
e=3D"Times New Roman" size=3D3><SPAN =0D=0Astyle=3D"FONT-SIZE: 12pt"><o:p>&=
nbsp;</o:p></SPAN></FONT></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =
=0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">Hey all!</SPAN></FONT=
> =0D=0A<o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =0D=0As=
tyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">We've begun rolling out our =
annual =0D=0Apatch cluster for Solaris (we're running 8 right now) and as s=
oon as it was =0D=0Ainstalled, we began to experience some very strange iss=
ues with =0D=0ANetBackup=2E</SPAN></FONT><o:p></o:p></P>=0D=0A<P><FONT face=
=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Geor=
gia">Environment:</SPAN></FONT> =0D=0A<o:p></o:p></P>=0D=0A<P><FONT face=3D=
Georgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia=
">Solaris 8 (patch cluster installed =0D=0Awas 10/2005)=2E We have also sin=
ce patched the CE card and TCP to the latest =0D=0Aavailable patch levels</=
SPAN></FONT><o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =0D=
=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">Sun F15k domain (4x8, si=
ngle =0D=0Aboard)</SPAN></FONT> <BR><FONT face=3DGeorgia size=3D2><SPAN =0D=
=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">4 IP addresses across 4 =
subnets on =0D=0Aboth the master/media and the media server</SPAN></FONT> <=
BR><FONT face=3DGeorgia =0D=0Asize=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT=
-FAMILY: Georgia">NetBackup 5=2E0 MP6 for =0D=0ASolaris</SPAN></FONT> <BR><=
FONT face=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAM=
ILY: Georgia">NetBackup 5=2E0 MP5 for =0D=0ANDMP</SPAN></FONT> <BR><FONT fa=
ce=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Ge=
orgia">Filers are:</SPAN></FONT> =0D=0A<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; <FONT face=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10=
pt; FONT-FAMILY: Georgia">NetApp&nbsp; w/tape attached is =0D=0ARelease 6=
=2E5=2E2R1P10&nbsp;&nbsp; =0D=0A</SPAN></FONT><BR>&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; <FONT face=3DGeorgia =0D=0Asize=3D2><SPAN style=3D"FONT-S=
IZE: 10pt; FONT-FAMILY: Georgia">NetApp w/out tape are =0D=0ARelease 6=2E4=
=2E5</SPAN></FONT> <o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><S=
PAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">Issues:</SPAN></F=
ONT> =0D=0A<o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =0D=
=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">1=2E&nbsp; All NDMP back=
ups with DAR =0D=0Aenabled no longer work=2E However, if we set the HIST_FI=
LE =3D N in the policy =0D=0Ainclude configuration, it works fine=2E 3 NetA=
pps, same problem with each one=2E 1 =0D=0Ahas tape attached thru the SAN a=
nd the others back up thru =0D=0Ait=2E</SPAN></FONT><o:p></o:p></P>=0D=0A<P=
><FONT face=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-F=
AMILY: Georgia">2=2E&nbsp; 211, 213 and 195 errors =0D=0Ahappen nightly on =
clients=2E Backups running thru the master/media itself do not =0D=0Aexperi=
ence issues, however, backups running thru media servers or SAN media =0D=
=0Aservers will get these errors=2E The backups will generally re-run succe=
ssfully, =0D=0Aif they run thru another media server=2E SAN Media server ba=
ckups will fail =0D=0Acompletely=2E This doesn't happen on the same clients=
 every night and some nights =0D=0Adoesn't happen at all=2E</SPAN></FONT><o=
:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FO=
NT-SIZE: 10pt; FONT-FAMILY: Georgia">3=2E&nbsp; Backups get sent to the =0D=
=0Aworklist but the PID dies so quickly that the job looks to be running (b=
pdbjobs) =0D=0Abut the PID isn't active on the master anymore, nor is there=
 any real activity =0D=0Awithin Netbackup (no tapes mounted, not writing go=
ing on, no database entries =0D=0Abeing created, etc=2E)=2E The only way to=
 get rid of these *active* jobs is to =0D=0Acompletely recycle Netbackup (b=
p=2Ekill_all)=2E</SPAN></FONT><o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia =
size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">Steps =
taken so far:</SPAN></FONT> =0D=0A<o:p></o:p></P>=0D=0A<P><FONT face=3DGeor=
gia size=3D2><SPAN =0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">We=
 have disabled all but the =0D=0APRIMARY interface on the Master and Media =
server and are no longer getting 211, =0D=0A213 and 195 errors, however the=
 NDMP/w DAR still do not work=2E We're going to =0D=0Astart adding interfac=
es back in and see at what point it breaks again=2E =0D=0A</SPAN></FONT><o:=
p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =0D=0Astyle=3D"FON=
T-SIZE: 10pt; FONT-FAMILY: Georgia">If anyone has seen anything like =0D=0A=
this can you please let me know if you've resolved it? We're at backline w/=
 Sun =0D=0Aand about to be escalated to backline w/ Veritas=2E&nbsp; =0D=0A=
</SPAN></FONT><o:p></o:p></P>=0D=0A<P><FONT face=3DGeorgia size=3D2><SPAN =
=0D=0Astyle=3D"FONT-SIZE: 10pt; FONT-FAMILY: Georgia">Thanks!<BR>Kate</SPAN=
></FONT> =0D=0A<o:p></o:p></P></DIV>=0D=0A<P>=0D=0A<HR SIZE=3D1>=0D=0A=0D=
=0A<P></P><BR>=0D=0A<P><STRONG><BR>This e-mail may contain confidential or =
privileged information=2E =0D=0AIf<BR>you<BR>think you have received this e=
-mail in error, please advise =0D=0Athe<BR>sender by<BR>reply e-mail and th=
en delete this e-mail immediately=2E Thank =0D=0Ayou=2E<BR>Aetna<BR></STRON=
G></P></BODY></HTML>=0D=0A=0D=0A=0D=0A<HTML><BODY><P><hr size=3D1></P><br>=
=0D=0A<P><STRONG><br>=0D=0AThis e-mail may contain confidential or privileg=
ed information=2E  If<br>=0D=0Ayou<br>=0D=0Athink you have received this e-=
mail in error, please advise the<br>=0D=0Asender by<br>=0D=0Areply e-mail a=
nd then delete this e-mail immediately=2E  Thank you=2E<br>=0D=0AAetna<br>=
=0D=0A</STRONG></P></BODY></HTML>=0D=0A
------_=_NextPart_001_01C65985.5BA256D4--