Veritas-bu

[Veritas-bu] rman backups are slow

2006-03-22 06:56:11
Subject: [Veritas-bu] rman backups are slow
From: atif76 AT gmail DOT com (Atif Munir)
Date: Wed, 22 Mar 2006 16:56:11 +0500
------=_Part_13270_10608931.1143028571074
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Thanks for your reply.I have increased the channels and now the speed is
70MB/sec but still it is not good enough.

As for FS backups i can achieve 100MB/sec.

For test backup i have turn the optimization on but for production i will
keep it off and we are using crosscheck option to sync the catalog.

Regards,
atif

On 3/22/06, andrew.x.smith AT jpmchase DOT com <andrew.x.smith AT jpmchase DOT 
com> wrote=
:
>
>
> Ok so your script is limiting your parallisms to 2, you could try adding
> more streams by adding in additional allocated channel commands. To run f=
our
> streams you would do:
>
>
> ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
> ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
> ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
> ALLOCATE CHANNEL ch04 TYPE 'SBT_TAPE';
>
>
> Also FYI CONFIGURE BACKUP OPTIMIZATION ON; can cause data loss on your
> backups if you don't fully understand the implications of this.
>
>  Use caution when enabling backup optimization if you use a media manager
> with its own internal expiration policy. Run CROSSCHECK periodically to
> synchronize the RMAN repository with the media manager. Otherwise, RMAN m=
ay
> skip backups due to optimization without recognizing that the media manag=
er
> has discarded backups stored on tape.
>
>
> But I really would recommend getting a good DBA to help you out with all
> of this....
>
>
>
>
> *"Atif Munir" <atif76 AT gmail DOT com>*
> Sent by: veritas-bu-admin AT mailman.eng.auburn DOT edu
>
> 22/03/2006 10:54
>
>         To:        "andrew.x.smith AT jpmchase DOT com" <
> andrew.x.smith AT jpmchase DOT com>
>         cc:        veritas-bu AT mailman.eng.auburn DOT edu
>         Subject:        Re: [Veritas-bu] rman backups are slow
>
>
>
> RUN {
> ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
> ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
> SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';
> BACKUP
>    $BACKUP_TYPE
>    FILESPERSET 20
>    # recommended format
>    FORMAT 'bk_%s_%p_%t'
>    DATABASE;
>    sql 'alter system archive log current';
> RELEASE CHANNEL ch01;
> RELEASE CHANNEL ch02;
> # backup all archive logs
> ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
> ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
> SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';
> BACKUP
>   filesperset 20
>   FORMAT 'al_%s_%p_%t'
>   ARCHIVELOG ALL DELETE INPUT;
> RELEASE CHANNEL ch01;
> RELEASE CHANNEL ch02;
> #
> # Note: During the process of backing up the database, RMAN also backs up
> the
> # control file.  This version of the control file does not contain the
> # information about the current backup because "nocatalog" has been
> specified.
> # To include the information about the current backup, the control file
> should
> # be backed up as the last step of the RMAN section.  This step would not
> be
> # necessary if we were using a recovery catalog.
> #
> ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
> SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';
> BACKUP
>    # recommended format
>    FORMAT 'cntrl_%s_%p_%t'
>    CURRENT CONTROLFILE;
> RELEASE CHANNEL ch00;
> }
> EOF
>
>
> On 3/22/06, *andrew.x.smith AT jpmchase DOT com* <andrew.x.smith AT jpmchase 
> DOT com> <=
*andrew.x.smith AT jpmchase DOT com
> * <andrew.x.smith AT jpmchase DOT com>> wrote:
>
> This is moving into the realms of RMAN performance tuning so you may get
> better advise from an Oracle board on this.
> Your default settings look ok to me, you could try testing with different
> values for CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4; to see which
> gives the best performance. Also all of the settings you provided can be
> overwritten by the RMAN script, can you post this too?
>
>
>
>
> *"Atif Munir" <**atif76 AT gmail DOT com* <atif76 AT gmail DOT com>*>*
> Sent by: *veritas-bu-admin AT mailman.eng.auburn DOT 
> edu*<veritas-bu-admin@mailm=
an.eng.auburn.edu>
>
> 22/03/2006 06:44
>
>        To:        *veritas-bu AT mailman.eng.auburn DOT 
> edu*<veritas-bu@mailman.=
eng.auburn.edu>
>        cc:
>        Subject:        [Veritas-bu] rman backups are slow
>
>
>
> I am unable to get more than 50MB/sec for rman backups. Although
> filesystem backups are running 100MB/sec.
>
> Is there any conf changes required in the rman. I have rman setting like
> this.
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> RMAN> show all;
>
> RMAN configuration parameters are:
> CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
> CONFIGURE BACKUP OPTIMIZATION ON;
> CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
> CONFIGURE CONTROLFILE AUTOBACKUP ON;
> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; #
> default
> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO
> '%F'; # default
> CONFIGURE DEVICE TYPE DISK PARALLELISM 4;
> CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4;
> CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
> CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
> CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
> CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; #
> default
> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  "BLKSIZE=3D1048576";
> CONFIGURE MAXSETSIZE TO UNLIMITED; # default
> CONFIGURE SNAPSHOT CONTROLFILE NAME TO
> '/oracle/product/9.2.0.4/dbs/snapcf_ebizprod.f'; # default
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
>
> _______________________________________________
> Veritas-bu maillist  -  *Veritas-bu AT mailman.eng.auburn DOT 
> edu*<Veritas-bu@ma=
ilman.eng.auburn.edu>
> *
> **http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu*<http://mailm=
an.eng.auburn.edu/mailman/listinfo/veritas-bu>
>
>
>
>
>

------=_Part_13270_10608931.1143028571074
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Thanks for your reply.I have increased the channels and now the speed is 70=
MB/sec but still it is not good enough.<br>
<br>
As for FS backups i can achieve 100MB/sec.<br>
<br>
For test backup i have turn the <font face=3D"Times New Roman" size=3D"3">o=
ptimization on but for production i will keep it off and we are using cross=
check option to sync the catalog.<br>
<br>
Regards,<br>
atif<br>
</font><br><div><span class=3D"gmail_quote">On 3/22/06, <b class=3D"gmail_s=
endername"><a href=3D"mailto:andrew.x.smith AT jpmchase DOT 
com">andrew.x.smith@jp=
mchase.com</a></b> &lt;<a href=3D"mailto:andrew.x.smith AT jpmchase DOT 
com">andre=
w.x.smith AT jpmchase DOT com
</a>&gt; wrote:</span><blockquote class=3D"gmail_quote" style=3D"border-lef=
t: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1=
ex;"><div style=3D"direction: ltr;">
<br><font face=3D"sans-serif" size=3D"2">Ok so your script is limiting your
parallisms to 2, you could try adding more streams by adding in
additional allocated channel commands. To run four streams you would do:</f=
ont>
</div><div style=3D"direction: ltr;"><span class=3D"q"><br>
<br><font face=3D"Times New Roman" size=3D"3">ALLOCATE CHANNEL ch01 TYPE 'S=
BT_TAPE';<br>
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';</font>
<br></span></div><div style=3D"direction: ltr;"><font face=3D"Times New Rom=
an" size=3D"3">ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';<br>
ALLOCATE CHANNEL ch04 TYPE 'SBT_TAPE';</font>
<br>
<br>
<br><font face=3D"Times New Roman" size=3D"3">Also FYI </font><font face=3D=
"Courier New" size=3D"2">CONFIGURE BACKUP OPTIMIZATION ON;</font><font face=
=3D"Times New Roman" size=3D"3"> can cause data loss on your backups if you=
 don't fully understand the implications of this.
</font>
<br>
<br>
<table align=3D"center">
<tbody><tr>
<td><font face=3D"Times New Roman" size=3D"3">Use caution when enabling bac=
kup optimization if you use a media manager with its own internal expiratio=
n policy. Run </font><font face=3D"Courier New" size=3D"3">CROSSCHECK</font=
><font face=3D"Times New Roman" size=3D"3">

periodically to synchronize the RMAN repository with the media manager.
Otherwise, RMAN may skip backups due to optimization without
recognizing that the media manager has discarded backups stored on tape.</f=
ont></td></tr></tbody></table>
<br>
<br>
<br><font face=3D"Times New Roman" size=3D"3">But I really would recommend =
getting a good DBA to help you out with all of this....</font>
<br>
<br>
<br>
<br>
<table width=3D"100%">
<tbody><tr valign=3D"top">
<td>
        <br>
</td><td><div style=3D"direction: ltr;"><span class=3D"q"><font face=3D"san=
s-serif" size=3D"1"><b>&quot;Atif Munir&quot; &lt;<a href=3D"mailto:atif76@=
gmail.com" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,ev=
ent,this)">
atif76 AT gmail DOT com</a>&gt;</b></font>
<br><font face=3D"sans-serif" size=3D"1">Sent by: <a href=3D"mailto:veritas=
-bu-admin AT mailman.eng.auburn DOT edu" target=3D"_blank" onclick=3D"return 
top.j=
s.OpenExtLink(window,event,this)">veritas-bu-admin AT mailman.eng.auburn DOT 
edu</=
a></font>

</span></div><div style=3D"direction: ltr;"><p><font face=3D"sans-serif" si=
ze=3D"1">22/03/2006 10:54</font>
<br>
</p></div></td><td><font face=3D"Arial" size=3D"1">&nbsp; &nbsp; &nbsp; &nb=
sp; </font>
<br><font face=3D"sans-serif" size=3D"1">&nbsp; &nbsp; &nbsp; &nbsp; To: &n=
bsp; &nbsp; &nbsp; &nbsp;&quot;<a href=3D"mailto:andrew.x.smith AT jpmchase DOT 
co=
m" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,event,this=
)">andrew.x.smith AT jpmchase DOT com</a>&quot; &lt;
<a href=3D"mailto:andrew.x.smith AT jpmchase DOT com" target=3D"_blank" 
onclick=3D=
"return top.js.OpenExtLink(window,event,this)">andrew.x.smith AT jpmchase DOT 
com<=
/a>&gt;</font>
<br><font face=3D"sans-serif" size=3D"1">&nbsp; &nbsp; &nbsp; &nbsp; cc: &n=
bsp; &nbsp; &nbsp; &nbsp;<a href=3D"mailto:veritas-bu AT mailman.eng.auburn DOT 
ed=
u" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,event,this=
)">veritas-bu AT mailman.eng.auburn DOT edu</a></font>

<br><font face=3D"sans-serif" size=3D"1">&nbsp; &nbsp; &nbsp; &nbsp; Subjec=
t: &nbsp; &nbsp; &nbsp; &nbsp;Re: [Veritas-bu] rman backups are slow</font>=
</td></tr></tbody></table></div><div style=3D"direction: ltr;"><span class=
=3D"e" id=3D"q_10a21c52748b27ca_5">
<br>
<br>
<br><font face=3D"Times New Roman" size=3D"3">RUN {<br>
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';<br>
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';<br>
SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';<br>
BACKUP<br>
 &nbsp; &nbsp;$BACKUP_TYPE<br>
 &nbsp; &nbsp;FILESPERSET 20<br>
 &nbsp; &nbsp;# recommended format<br>
 &nbsp; &nbsp;FORMAT 'bk_%s_%p_%t'<br>
 &nbsp; &nbsp;DATABASE;<br>
 &nbsp; &nbsp;sql 'alter system archive log current';<br>
RELEASE CHANNEL ch01;<br>
RELEASE CHANNEL ch02;<br>
# backup all archive logs<br>
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';<br>
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';<br>
SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';<br>
BACKUP<br>
 &nbsp; filesperset 20<br>
 &nbsp; FORMAT 'al_%s_%p_%t'<br>
 &nbsp; ARCHIVELOG ALL DELETE INPUT;<br>
RELEASE CHANNEL ch01;<br>
RELEASE CHANNEL ch02;<br>
#<br>
# Note: During the process of backing up the database, RMAN also backs up t=
he<br>
# control file. &nbsp;This version of the control file does not contain the=
<br>
# information about the current backup because &quot;nocatalog&quot; has be=
en specified.<br>
# To include the information about the current backup, the control file sho=
uld<br>
# be backed up as the last step of the RMAN section. &nbsp;This step would =
not be<br>
# necessary if we were using a recovery catalog.<br>
#<br>
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';<br>
SEND 'NB_ORA_SERV=3Dserver-backup,NB_ORA_CLIENT=3Dserver';<br>
BACKUP<br>
 &nbsp; &nbsp;# recommended format<br>
 &nbsp; &nbsp;FORMAT 'cntrl_%s_%p_%t'<br>
 &nbsp; &nbsp;CURRENT CONTROLFILE;<br>
RELEASE CHANNEL ch00;<br>
}<br>
EOF<br>
<br>
</font>
<br><font face=3D"Times New Roman" size=3D"3">On 3/22/06, </font><a href=3D=
"mailto:andrew.x.smith AT jpmchase DOT com" target=3D"_blank" onclick=3D"return 
to=
p.js.OpenExtLink(window,event,this)"><font color=3D"blue" face=3D"Times New=
 Roman" size=3D"3">
<b><u>andrew.x.smith AT jpmchase DOT com</u></b></font></a><font face=3D"Times 
New=
 Roman" size=3D"3"> &lt;</font><a href=3D"mailto:andrew.x.smith AT jpmchase DOT 
co=
m" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,event,this=
)"><font color=3D"blue" face=3D"Times New Roman" size=3D"3">
<u>andrew.x.smith AT jpmchase DOT com </u></font></a><font face=3D"Times New 
Roman=
" size=3D"3">&gt; wrote:</font>
<br><font face=3D"sans-serif" size=3D"2"><br>
This is moving into the realms of RMAN performance tuning so you may get be=
tter advise from an Oracle board on this.</font><font face=3D"Times New Rom=
an" size=3D"3"> </font><font face=3D"sans-serif" size=3D"2"><br>
Your default settings look ok to me, you could try testing with different v=
alues for </font><font face=3D"Courier New" size=3D"2">CONFIGURE DEVICE TYP=
E 'SBT_TAPE' PARALLELISM 4;</font><font face=3D"sans-serif" size=3D"2">
to see which gives the best performance. Also all of the settings you
provided can be overwritten by the RMAN script, can you post this too?</fon=
t><font face=3D"Times New Roman" size=3D"3"> <br>
</font><font face=3D"sans-serif" size=3D"2"><br>
 </font><font face=3D"Times New Roman" size=3D"3">&nbsp;<br>
<br>
</font>
<table width=3D"100%">
<tbody><tr valign=3D"top">
<td width=3D"2%">
        <br>
</td><td width=3D"48%"><font face=3D"sans-serif" size=3D"1"><b>&quot;Atif M=
unir&quot; &lt;</b></font><a href=3D"mailto:atif76 AT gmail DOT com" 
target=3D"_bl=
ank" onclick=3D"return top.js.OpenExtLink(window,event,this)"><font color=
=3D"blue" face=3D"sans-serif" size=3D"1">
<b><u>atif76 AT gmail DOT com</u></b></font></a><font face=3D"sans-serif" 
size=3D"=
1"><b>&gt;</b></font><font face=3D"Times New Roman" size=3D"3"> </font><fon=
t face=3D"sans-serif" size=3D"1"><br>
Sent by: </font><a href=3D"mailto:veritas-bu-admin AT mailman.eng.auburn DOT 
edu" =
target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,event,this)">=
<font color=3D"blue" face=3D"sans-serif" size=3D"1"><u>veritas-bu-admin@mai=
lman.eng.auburn.edu
</u></font></a><font face=3D"Times New Roman" size=3D"3"> </font>
<p><font face=3D"sans-serif" size=3D"1">22/03/2006 06:44</font><font face=
=3D"Times New Roman" size=3D"3"> </font>
</p></td><td width=3D"48%"><font face=3D"Arial" size=3D"1">&nbsp; &nbsp; &n=
bsp; &nbsp; </font><font face=3D"sans-serif" size=3D"1"><br>
 &nbsp; &nbsp; &nbsp; &nbsp;To: &nbsp; &nbsp; &nbsp; &nbsp;</font><a href=
=3D"mailto:veritas-bu AT mailman.eng.auburn DOT edu" target=3D"_blank" 
onclick=3D"=
return top.js.OpenExtLink(window,event,this)"><font color=3D"blue" face=3D"=
sans-serif" size=3D"1"><u>veritas-bu AT mailman.eng.auburn DOT edu
</u></font></a><font face=3D"Times New Roman" size=3D"3"> </font><font face=
=3D"sans-serif" size=3D"1"><br>
 &nbsp; &nbsp; &nbsp; &nbsp;cc: &nbsp; &nbsp; &nbsp; &nbsp;</font><font fac=
e=3D"Times New Roman" size=3D"3"> </font><font face=3D"sans-serif" size=3D"=
1"><br>
 &nbsp; &nbsp; &nbsp; &nbsp;Subject: &nbsp; &nbsp; &nbsp; &nbsp;[Veritas-bu=
] rman backups are slow</font></td></tr></tbody></table>
<br><font face=3D"Times New Roman" size=3D"3"><br>
<br>
</font>
<br><font face=3D"Courier New" size=3D"2">I am unable to get more than 50MB=
/sec for rman backups. Although <br>
filesystem backups are running 100MB/sec.<br>
<br>
Is there any conf changes required in the rman. I have rman setting like th=
is.<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D<br>
RMAN&gt; show all;<br>
<br>
RMAN configuration parameters are:<br>
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default<br>
CONFIGURE BACKUP OPTIMIZATION ON;<br>
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default<br>
CONFIGURE CONTROLFILE AUTOBACKUP ON;<br>
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # def=
ault<br>
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO<br>
'%F'; # default<br>
CONFIGURE DEVICE TYPE DISK PARALLELISM 4;<br>
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4;<br>
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default<br>
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default<b=
r>
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default<br>
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default=
<br>
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS &nbsp;&quot;BLKSIZE=3D104857=
6&quot;;<br>
CONFIGURE MAXSETSIZE TO UNLIMITED; # default<br>
CONFIGURE SNAPSHOT CONTROLFILE NAME TO<br>
'/oracle/product/9.2.0.4/dbs/snapcf_ebizprod.f'; # default<br>
<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D<br>
</font>
<br><font face=3D"Courier New" size=3D"2">_________________________________=
______________<br>
Veritas-bu maillist &nbsp;- &nbsp;</font><a href=3D"mailto:Veritas-bu@mailm=
an.eng.auburn.edu" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(w=
indow,event,this)"><font color=3D"blue" face=3D"Courier New" size=3D"2"><u>=
Veritas-bu AT mailman.eng.auburn DOT edu
</u></font></a><font color=3D"blue" face=3D"Courier New" size=3D"2"><u><br>
</u></font><a href=3D"http://mailman.eng.auburn.edu/mailman/listinfo/verita=
s-bu" target=3D"_blank" onclick=3D"return top.js.OpenExtLink(window,event,t=
his)"><font color=3D"blue" face=3D"Courier New" size=3D"2"><u>http://mailma=
n.eng.auburn.edu/mailman/listinfo/veritas-bu
</u></font></a><font face=3D"Times New Roman" size=3D"3"><br>
<br>
</font>
<br>
<br>
<br>
</span></div></blockquote></div><br>

------=_Part_13270_10608931.1143028571074--

<Prev in Thread] Current Thread [Next in Thread>