Veritas-bu

[Veritas-bu] Drive Down logs?

2002-01-11 04:27:19
Subject: [Veritas-bu] Drive Down logs?
From: BrochF AT europe.stortek DOT com (Brochart, Fabrice)
Date: Fri, 11 Jan 2002 09:27:19 -0000
This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C19A82.2AE37420
Content-Type: text/plain;
        charset="iso-8859-1"

                Hi,

Just for explanations, the library don't decide to clean a drive at any
moment.

When a clean is required by drive, it will "ask" to library by internal
interface (drive -> lmu (library management unit)).
Then the library mount the cleaning cartridge at the next mount required by
backup software (or any soft which use the library).

So, if you use DLT7000, the time of cleaning by the cartridge grow up every
time (it don't use the same part of the cleaning cartridge until the "dead"
of this cleaning cart (20 cleans).At the end of life, the time of cleaning
can take up to 7 minutes !!Too long for NetBackup, so it down the drive !

The last Veritas patch NB_34_1 should correct this problem.

Description:
   Add robotic auto cleaning support for STK L series robots

I suppose that they grow up the timeout interval or something like that (as
we do with Acsls)

HTH

F@b

-----Original Message-----
From: George Gagnon [mailto:ggagnon AT ibasis DOT net]
Sent: jeudi 10 janvier 2002 20:00
To: 'Sokolowski Ric-ERS004'; 'Robert.L.Harris AT rdlg DOT net'
Cc: 'veritas-bu AT mailman.eng.auburn DOT edu'
Subject: RE: [Veritas-bu] Drive Down logs?


Here is a script that automatically up's any downed drives. This has solved
my problem with the L700 cleaning tape downing the drive.

#! /bin/sh
#
############################################################################
##################
##
##    up_drives - Bring any DOWN drives back on-line
##                This script is intended to be run out of cron to keep any
downed
##                drives up. This will help prevent the vaults from hanging
for
##                long periods of time while waiting for tapes in downed
drives.
##                If there is a serious problem with a drive this script
will ignore
##                that and keep bringing it up which could be bad. A better
approach
##                would be to monitor the down-to-up transitions and only
re-up the
##                drives a set number of times. That would require a state
file to be
##                kept which is tougher.
##                Another solution would be to watch the vault log and only
go after
##                the tapes if the vault has been hanging for a long time.
Also a more
##                complicated solution.
##
##
############################################################################
##################
#

LOGFILE=/tmp/up_drive.log.$$
TMPFILE=/tmp/t_stats.$$
RCPT_LIST=""
DONT_CHECK=""

if [ $# -eq 1 ]
then
   case $1 in
       -[nN] )
           NOTIFY="Y"
           ;;

       * )
           echo "Unknown option: $1"
           exit 2
           ;;
   esac

else

   NOTIFY="N"

fi

if [ ! `/usr/openv/volmgr/bin/vmoprcmd -d ds | /usr/bin/grep -c DOWN` -eq 0
]
then

   ENABLED_DRIVES="N"
   /usr/bin/echo "Down drives detected on `uname -n`\n" >$LOGFILE
   /usr/openv/volmgr/bin/vmoprcmd -d ds >>$LOGFILE
   /usr/bin/echo "" >>$LOGFILE

   /usr/openv/volmgr/bin/vmoprcmd -d ds | /usr/bin/grep DOWN >$TMPFILE

   for drive in `/usr/bin/cat $TMPFILE | /usr/bin/awk '{print $1}'`
   do
      if [ "`/usr/bin/echo $DONT_CHECK | /usr/bin/grep -w  $drive`" != "" ]
      then
         /usr/bin/echo "*** Skipping drive $drive ***" >>$LOGFILE
      else
         /usr/bin/echo "Setting drive $drive to UP status" >>$LOGFILE
         /usr/openv/volmgr/bin/vmoprcmd -up $drive
         ENABLED_DRIVES="Y"
      fi
   done

   /usr/bin/echo "\nPlease verify that the drives are all set.\n" >>$LOGFILE

   # Send email to notify staff that drives were reset.
   if [ $NOTIFY = "Y" -a $ENABLED_DRIVES = "Y" ] ;
   then
      /usr/bin/cat $LOGFILE |/usr/bin/mailx -s "Drives enabled on
`/usr/bin/uname -n`" $RCPT_LIST
   else
      /usr/bin/more $LOGFILE
   fi

   /usr/bin/rm -rf $TMPFILE $LOGFILE

fi

exit 0

 -----Original Message-----
From:   Sokolowski Ric-ERS004 [mailto:Ric.Sokolowski AT motorola DOT com] 
Sent:   Thursday, January 10, 2002 1:53 PM
To:     'Robert.L.Harris AT rdlg DOT net'
Cc:     'veritas-bu AT mailman.eng.auburn DOT edu'
Subject:        RE: [Veritas-bu] Drive Down logs?

I've seen similar problems when NetBackup is unaware that a cleaning tape is
in the drive.  We use library controlled cleaning (L700) and occasionally,
the library will decide to clean a drive and when NB attempts to put another
tape into that drive, whamo, the drive gets downed.  Once the cleaning is
finished, the CLN tape is put back where it belongs but you're stuck with a
down drive.  With verbose logging on all Media Manager daemons, you should
see movement of the CLN tapes show up in syslog on your robotic controller
which corresponds to the downing of the drive.  HTH!

--
Regards,

Ric Sokolowski (Ric.Sokolowski AT motorola DOT com)
Staff Systems Engineer
Phone: (954) 723-6332  Pager: 8776050503 AT skytel DOT com
Motorola, Inc.  / CGISS / Enterprise Computing
8000 West Sunrise Blvd, MS 22-2F, Plantation, FL 33322

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

------_=_NextPart_001_01C19A82.2AE37420
Content-Type: text/html;
        charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
5.5.2653.12">
<TITLE>RE: [Veritas-bu] Drive Down logs?</TITLE>
</HEAD>
<BODY>

<P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <FONT SIZE=3D2>Hi,</FONT>
</P>

<P><FONT SIZE=3D2>Just for explanations, the library don't decide to =
clean a drive at any moment.</FONT>
</P>

<P><FONT SIZE=3D2>When a clean is required by drive, it will =
&quot;ask&quot; to library by internal interface (drive -&gt; lmu =
(library management unit)).</FONT></P>

<P><FONT SIZE=3D2>Then the library mount the cleaning cartridge at the =
next mount required by backup software (or any soft which use the =
library).</FONT></P>

<P><FONT SIZE=3D2>So, if you use DLT7000, the time of cleaning by the =
cartridge grow up every time (it don't use the same part of the =
cleaning cartridge until the &quot;dead&quot; of this cleaning cart (20 =
cleans).At the end of life, the time of cleaning can take up to 7 =
minutes !!Too long for NetBackup, so it down the drive !</FONT></P>

<P><FONT SIZE=3D2>The last Veritas patch NB_34_1 should correct this =
problem.</FONT>
</P>

<P><FONT SIZE=3D2>Description:</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; Add robotic auto cleaning support for =
STK L series robots</FONT>
</P>

<P><FONT SIZE=3D2>I suppose that they grow up the timeout interval or =
something like that (as we do with Acsls)</FONT>
</P>

<P><FONT SIZE=3D2>HTH</FONT>
</P>

<P><FONT SIZE=3D2>F@b</FONT>
</P>

<P><FONT SIZE=3D2>-----Original Message-----</FONT>
<BR><FONT SIZE=3D2>From: George Gagnon [<A =
HREF=3D"mailto:ggagnon AT ibasis DOT net">mailto:ggagnon AT ibasis DOT 
net</A>]</FONT>=

<BR><FONT SIZE=3D2>Sent: jeudi 10 janvier 2002 20:00</FONT>
<BR><FONT SIZE=3D2>To: 'Sokolowski Ric-ERS004'; =
'Robert.L.Harris AT rdlg DOT net'</FONT>
<BR><FONT SIZE=3D2>Cc: 'veritas-bu AT mailman.eng.auburn DOT edu'</FONT>
<BR><FONT SIZE=3D2>Subject: RE: [Veritas-bu] Drive Down logs?</FONT>
</P>
<BR>

<P><FONT SIZE=3D2>Here is a script that automatically up's any downed =
drives. This has solved</FONT>
<BR><FONT SIZE=3D2>my problem with the L700 cleaning tape downing the =
drive.</FONT>
</P>

<P><FONT SIZE=3D2>#! /bin/sh</FONT>
<BR><FONT SIZE=3D2>#</FONT>
<BR><FONT =
SIZE=3D2>###############################################################=
#############</FONT>
<BR><FONT SIZE=3D2>##################</FONT>
<BR><FONT SIZE=3D2>##</FONT>
<BR><FONT SIZE=3D2>##&nbsp;&nbsp;&nbsp; up_drives - Bring any DOWN =
drives back on-line</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; This script is intended to be run out of =
cron to keep any</FONT>
<BR><FONT SIZE=3D2>downed</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; drives up. This will help prevent the =
vaults from hanging</FONT>
<BR><FONT SIZE=3D2>for</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; long periods of time while waiting for =
tapes in downed</FONT>
<BR><FONT SIZE=3D2>drives.</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; If there is a serious problem with a =
drive this script</FONT>
<BR><FONT SIZE=3D2>will ignore</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; that and keep bringing it up which could =
be bad. A better</FONT>
<BR><FONT SIZE=3D2>approach</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; would be to monitor the down-to-up =
transitions and only</FONT>
<BR><FONT SIZE=3D2>re-up the</FONT>
<BR><FONT SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; drives a set number of times. =
That would require a state</FONT>
<BR><FONT SIZE=3D2>file to be</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kept which is tougher.</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Another solution would be to watch the =
vault log and only</FONT>
<BR><FONT SIZE=3D2>go after</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; the tapes if the vault has been hanging =
for a long time.</FONT>
<BR><FONT SIZE=3D2>Also a more</FONT>
<BR><FONT =
SIZE=3D2>##&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; complicated solution.</FONT>
<BR><FONT SIZE=3D2>##</FONT>
<BR><FONT SIZE=3D2>##</FONT>
<BR><FONT =
SIZE=3D2>###############################################################=
#############</FONT>
<BR><FONT SIZE=3D2>##################</FONT>
<BR><FONT SIZE=3D2>#</FONT>
</P>

<P><FONT SIZE=3D2>LOGFILE=3D/tmp/up_drive.log.$$</FONT>
<BR><FONT SIZE=3D2>TMPFILE=3D/tmp/t_stats.$$</FONT>
<BR><FONT SIZE=3D2>RCPT_LIST=3D&quot;&quot;</FONT>
<BR><FONT SIZE=3D2>DONT_CHECK=3D&quot;&quot;</FONT>
</P>

<P><FONT SIZE=3D2>if [ $# -eq 1 ]</FONT>
<BR><FONT SIZE=3D2>then</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; case $1 in</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -[nN] )</FONT>
<BR><FONT =
SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
NOTIFY=3D&quot;Y&quot;</FONT>
<BR><FONT =
SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
;;</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * )</FONT>
<BR><FONT =
SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
echo &quot;Unknown option: $1&quot;</FONT>
<BR><FONT =
SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
exit 2</FONT>
<BR><FONT =
SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
;;</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; esac</FONT>
</P>

<P><FONT SIZE=3D2>else</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; NOTIFY=3D&quot;N&quot;</FONT>
</P>

<P><FONT SIZE=3D2>fi</FONT>
</P>

<P><FONT SIZE=3D2>if [ ! `/usr/openv/volmgr/bin/vmoprcmd -d ds | =
/usr/bin/grep -c DOWN` -eq 0</FONT>
<BR><FONT SIZE=3D2>]</FONT>
<BR><FONT SIZE=3D2>then</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; ENABLED_DRIVES=3D&quot;N&quot;</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; /usr/bin/echo &quot;Down drives =
detected on `uname -n`\n&quot; &gt;$LOGFILE</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; /usr/openv/volmgr/bin/vmoprcmd -d ds =
&gt;&gt;$LOGFILE</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; /usr/bin/echo &quot;&quot; =
&gt;&gt;$LOGFILE</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; /usr/openv/volmgr/bin/vmoprcmd -d ds | =
/usr/bin/grep DOWN &gt;$TMPFILE</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; for drive in `/usr/bin/cat $TMPFILE | =
/usr/bin/awk '{print $1}'`</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; do</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if [ =
&quot;`/usr/bin/echo $DONT_CHECK | /usr/bin/grep -w&nbsp; $drive`&quot; =
!=3D &quot;&quot; ]</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; then</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
/usr/bin/echo &quot;*** Skipping drive $drive ***&quot; =
&gt;&gt;$LOGFILE</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
/usr/bin/echo &quot;Setting drive $drive to UP status&quot; =
&gt;&gt;$LOGFILE</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
/usr/openv/volmgr/bin/vmoprcmd -up $drive</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
ENABLED_DRIVES=3D&quot;Y&quot;</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fi</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; done</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; /usr/bin/echo &quot;\nPlease verify that =
the drives are all set.\n&quot; &gt;&gt;$LOGFILE</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; # Send email to notify staff that drives =
were reset.</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; if [ $NOTIFY =3D &quot;Y&quot; -a =
$ENABLED_DRIVES =3D &quot;Y&quot; ] ;</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; then</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /usr/bin/cat $LOGFILE =
|/usr/bin/mailx -s &quot;Drives enabled on</FONT>
<BR><FONT SIZE=3D2>`/usr/bin/uname -n`&quot; $RCPT_LIST</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; else</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /usr/bin/more =
$LOGFILE</FONT>
<BR><FONT SIZE=3D2>&nbsp;&nbsp; fi</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;&nbsp; /usr/bin/rm -rf $TMPFILE $LOGFILE</FONT>
</P>

<P><FONT SIZE=3D2>fi</FONT>
</P>

<P><FONT SIZE=3D2>exit 0</FONT>
</P>

<P><FONT SIZE=3D2>&nbsp;-----Original Message-----</FONT>
<BR><FONT SIZE=3D2>From: &nbsp; Sokolowski Ric-ERS004 [<A =
HREF=3D"mailto:Ric.Sokolowski AT motorola DOT com">mailto:Ric.Sokolowski@motoro=
la.com</A>] </FONT>
<BR><FONT SIZE=3D2>Sent:&nbsp;&nbsp; Thursday, January 10, 2002 1:53 =
PM</FONT>
<BR><FONT SIZE=3D2>To:&nbsp;&nbsp;&nbsp;&nbsp; =
'Robert.L.Harris AT rdlg DOT net'</FONT>
<BR><FONT SIZE=3D2>Cc:&nbsp;&nbsp;&nbsp;&nbsp; =
'veritas-bu AT mailman.eng.auburn DOT edu'</FONT>
<BR><FONT SIZE=3D2>Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
RE: [Veritas-bu] Drive Down logs?</FONT>
</P>

<P><FONT SIZE=3D2>I've seen similar problems when NetBackup is unaware =
that a cleaning tape is</FONT>
<BR><FONT SIZE=3D2>in the drive.&nbsp; We use library controlled =
cleaning (L700) and occasionally,</FONT>
<BR><FONT SIZE=3D2>the library will decide to clean a drive and when NB =
attempts to put another</FONT>
<BR><FONT SIZE=3D2>tape into that drive, whamo, the drive gets =
downed.&nbsp; Once the cleaning is</FONT>
<BR><FONT SIZE=3D2>finished, the CLN tape is put back where it belongs =
but you're stuck with a</FONT>
<BR><FONT SIZE=3D2>down drive.&nbsp; With verbose logging on all Media =
Manager daemons, you should</FONT>
<BR><FONT SIZE=3D2>see movement of the CLN tapes show up in syslog on =
your robotic controller</FONT>
<BR><FONT SIZE=3D2>which corresponds to the downing of the drive.&nbsp; =
HTH!</FONT>
</P>

<P><FONT SIZE=3D2>--</FONT>
<BR><FONT SIZE=3D2>Regards,</FONT>
</P>

<P><FONT SIZE=3D2>Ric Sokolowski (Ric.Sokolowski AT motorola DOT com)</FONT>
<BR><FONT SIZE=3D2>Staff Systems Engineer</FONT>
<BR><FONT SIZE=3D2>Phone: (954) 723-6332&nbsp; Pager: =
8776050503 AT skytel DOT com</FONT>
<BR><FONT SIZE=3D2>Motorola, Inc.&nbsp; / CGISS / Enterprise =
Computing</FONT>
<BR><FONT SIZE=3D2>8000 West Sunrise Blvd, MS 22-2F, Plantation, FL =
33322</FONT>
</P>

<P><FONT =
SIZE=3D2>_______________________________________________</FONT>
<BR><FONT SIZE=3D2>Veritas-bu maillist&nbsp; -&nbsp; =
Veritas-bu AT mailman.eng.auburn DOT edu</FONT>
<BR><FONT SIZE=3D2><A =
HREF=3D"http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu"; =
TARGET=3D"_blank">http://mailman.eng.auburn.edu/mailman/listinfo/veritas=
-bu</A></FONT>
<BR><FONT =
SIZE=3D2>_______________________________________________</FONT>
<BR><FONT SIZE=3D2>Veritas-bu maillist&nbsp; -&nbsp; =
Veritas-bu AT mailman.eng.auburn DOT edu</FONT>
<BR><FONT SIZE=3D2><A =
HREF=3D"http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu"; =
TARGET=3D"_blank">http://mailman.eng.auburn.edu/mailman/listinfo/veritas=
-bu</A></FONT>
</P>

</BODY>
</HTML>
------_=_NextPart_001_01C19A82.2AE37420--

<Prev in Thread] Current Thread [Next in Thread>