Veritas-bu

[Veritas-bu] Very ODD problem with NB5.1MP3a + 2 Solaris Clients

2005-11-08 12:34:53
Subject: [Veritas-bu] Very ODD problem with NB5.1MP3a + 2 Solaris Clients
From: jpiszcz AT servervault DOT com (Piszcz, Justin)
Date: Tue, 8 Nov 2005 12:34:53 -0500
This is a multi-part message in MIME format.

------_=_NextPart_001_01C5E48A.D0E3EF6C
Content-Type: text/plain;
        charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

They are multi-homed but with completely different subnet masks and
routes.

Again, there are 5 machines with only 1 path to the backup server, there
are other nics but with separate interfaces+routes+Subnets.

=20

I see what you are saying but this is a completely vlan+server-isolated
backup network.

=20

I've checked with the network guys they are telling me is a box problem.

=20

Any other ideas?

=20

Justin.

=20

________________________________

From: Paul Keating [mailto:pkeating AT bank-banque-canada DOT ca]=20
Sent: Tuesday, November 08, 2005 12:16 PM
To: Piszcz, Justin
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Very ODD problem with NB5.1MP3a + 2 Solaris
Clients

=20

Ummm....are any of your systems multi homed?

=20

looks like an asymmetrical routing issue we had here recently.

=20

backup server talks to client on NIC A, but client replies back to
server from NIC B via an alternate path.

=20

the ARP table on your switch that the backup server and NIC A are
connected to doesn't get populated with the MAC of the client NIC A
untill the client ARPs or tries to talk to via NIC A....then
communication works for 5-10 minutes or untill your ARP cache refreshes,
then communication is broken till the client tries to talk to the server
again.

=20

I'm thinking it's likely not an issue if your clients all are single NIC
or connected to a single switch along with the backup server.

=20

check with your network guys to see if they're seeing any broadcasting
on the switches your backup server and clients are attached to.

=20

Paul

        -----Original Message-----
        From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of Piszcz,
Justin
        Sent: November 8, 2005 11:59 AM
        To: Piszcz, Justin; Patrick Whelan
        Cc: veritas-bu AT mailman.eng.auburn DOT edu
        Subject: RE: [Veritas-bu] Very ODD problem with NB5.1MP3a + 2
Solaris Clients

        Name resolution is setup through /etc/hosts on the backup
server, no external DNS servers are used.

        The subnet is the same yes.

        The bp.conf is the same for the most part, the error occurs on
both systems.

        =20

        MASTER -> CLIENTS:

        jpiszcz@backup01# telnet 172.16.0.135 bpcd

        Trying 172.16.0.135...

        ^C

        jpiszcz@backup01# telnet 172.16.0.136 bpcd

        Trying 172.16.0.136...

        =20

        HANGs and HANGs..-How would I debug this problem?

        =20

        CLIENTS -> MASTER

        box1# telnet backup01 bpcd

        Trying 172.16.0.2...

        Connected to backup01.

        Escape character is '^]'.

        =20

        box2# telnet backup01 bpcd

        Trying 172.16.0.2...

        Connected to backup01.

        Escape character is '^]'.

        =20

        Now the EXTREMELY WEIRD PART!!! (now I can MASTER->CLIENTS) - NO
ISSUE

        jpiszcz@backup01# telnet 172.16.0.135 bpcd

        Trying 172.16.0.135...

        Connected to 172.16.0.135.

        Escape character is '^]'.

        ^]

        telnet> Connection to 172.16.0.135 closed.

        jpiszcz@backup01# telnet 172.16.0.136 bpcd

        Trying 172.16.0.136...

        Connected to 172.16.0.136.

        Escape character is '^]'.

        =20

        Any freaking clue what is going on here?

        =20


------_=_NextPart_001_01C5E48A.D0E3EF6C
Content-Type: text/html;
        charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40";>

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Message</title>
<style>
<!--
 /* Font Definitions */
 @font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
a:link, span.MsoHyperlink
        {color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {color:purple;
        text-decoration:underline;}
p
        {mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman";}
code
        {font-family:"Courier New";}
span.EmailStyle19
        {mso-style-type:personal;
        font-family:Arial;
        color:windowtext;}
span.EmailStyle20
        {mso-style-type:personal;
        font-family:Arial;
        color:navy;}
span.EmailStyle21
        {mso-style-type:personal;
        font-family:Arial;
        color:navy;}
span.EmailStyle22
        {mso-style-type:personal-reply;
        font-family:Arial;
        color:navy;}
@page Section1
        {size:8.5in 11.0in;
        margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
        {page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>They are multi-homed but with =
completely
different subnet masks and routes.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Again, there are 5 machines with =
only 1
path to the backup server, there are other nics but with separate
interfaces+routes+Subnets.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I see what you are saying but this =
is a
completely vlan+server-isolated backup =
network.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I&#8217;ve checked with the network =
guys
they are telling me is a box problem.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Any other =
ideas?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Justin.<o:p></o:p></span></font></p>=


<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Paul =
Keating
[mailto:pkeating AT bank-banque-canada DOT ca] <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, November =
08, 2005
12:16 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Piszcz, Justin<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b>
veritas-bu AT mailman.eng.auburn DOT edu<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: [Veritas-bu] =
Very ODD
problem with NB5.1MP3a + 2 Solaris Clients</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>Ummm....are any of your systems =
multi
homed?</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>looks like an asymmetrical =
routing issue
we had here recently.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>backup server talks to client on =
NIC A,
but client replies back to server from NIC B via an alternate =
path.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>the ARP table on your switch that =
the
backup server and NIC A&nbsp;are connected to doesn't get populated with =
the
MAC of the&nbsp;client NIC A untill the client ARPs or&nbsp;tries to =
talk to
via NIC A....then communication works for 5-10 minutes or untill your =
ARP cache
refreshes, then communication is broken till the client tries to talk to =
the
server again.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>I'm thinking it's likely not an =
issue if
your clients all are single NIC or connected to a single switch along =
with the
backup server.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>check with your network guys to =
see if
they're seeing any broadcasting on the switches your backup server and =
clients
are attached to.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:maroon'>Paul</span></font><o:p></o:p></p>

</div>

<blockquote style=3D'border:none;border-left:solid maroon =
1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'=
>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma'>-----Original =
Message-----<br>
<b><span style=3D'font-weight:bold'>From:</span></b>
veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] <b><span =
style=3D'font-weight:
bold'>On Behalf Of </span></b>Piszcz, Justin<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> November 8, 2005 =
11:59 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Piszcz, Justin; =
Patrick Whelan<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b>
veritas-bu AT mailman.eng.auburn DOT edu<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: [Veritas-bu] =
Very ODD
problem with NB5.1MP3a + 2 Solaris Clients</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Name resolution is setup through
/etc/hosts on the backup server, no external DNS servers are =
used.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The subnet is the same =
yes.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The bp.conf is the same for the =
most part,
the error occurs on both systems.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-bottom:solid windowtext =
1.0pt;padding:0in 0in 1.0pt 0in'>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>MASTER -&gt; =
CLIENTS:<o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>jpiszcz@backup01# telnet =
172.16.0.135 bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.135...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>^C<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>jpiszcz@backup01# telnet =
172.16.0.136 bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.136...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>HANGs and HANGs..&#8212;How would I =
debug
this problem?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-bottom:solid windowtext =
1.0pt;padding:0in 0in 1.0pt 0in'>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>CLIENTS -&gt; =
MASTER<o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>box1# telnet backup01 =
bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.2...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Connected to =
backup01.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Escape character is =
'^]'.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>box2# telnet backup01 =
bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.2...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Connected to =
backup01.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Escape character is =
'^]'.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-bottom:solid windowtext =
1.0pt;padding:0in 0in 1.0pt 0in'>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Now the EXTREMELY WEIRD PART!!! =
(now I can
MASTER-&gt;CLIENTS) &#8211; NO ISSUE<o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>jpiszcz@backup01# telnet =
172.16.0.135 bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.135...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Connected to =
172.16.0.135.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Escape character is =
'^]'.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>^]<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>telnet&gt; Connection to =
172.16.0.135
closed.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>jpiszcz@backup01# telnet =
172.16.0.136 bpcd<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Trying =
172.16.0.136...<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Connected to =
172.16.0.136.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Escape character is =
'^]'.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Any freaking clue what is going on =
here?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</blockquote>

</div>

</body>

</html>

------_=_NextPart_001_01C5E48A.D0E3EF6C--