Veritas-bu

Re: [Veritas-bu] 10GB network + perf questions etc

2007-08-24 15:43:56
Subject: Re: [Veritas-bu] 10GB network + perf questions etc
From: "Nick Majeran" <nmajeran AT gmail DOT com>
To: veritas-bu AT mailman.eng.auburn DOT edu, "Curtis Preston" <cpreston AT glasshouse DOT com>
Date: Fri, 24 Aug 2007 14:23:07 -0500
On one of our media servers, I can receive 250 MB/s into my 6850 with
4 GigE connections bonded into two LACP bonds, stream that out to six
fc LTO-3 tape drives, and the box is plenty usable.  We are upgrading
our network here, and I'd be surprised if I couldn't get closer to
300-350MB/s in the future.  The E6900 is still PCI-X, correct?  PCI-e
is a must to do anything that I/O intensive, or so I would think.
But, I'm probably not as smart as those Sun d00ds.
;-)

-- nick

> ------------------------------
>
> Message: 4
> Date: Fri, 24 Aug 2007 01:42:33 -0400
> From: "Curtis Preston" <cpreston AT glasshouse DOT com>
> Subject: Re: [Veritas-bu] 10GB network + perf questions etc
> To: "Dominik Pietrzykowski" <dominik_pietrzykowski AT toll.com DOT au>,
>         <veritas-bu AT mailman.eng.auburn DOT edu>
> Message-ID:
>         <4FBA0941CF3D9347889AA5FF23A809BEAD0558 AT ghmail02.glasshousetech 
> DOT com>
> Content-Type: text/plain;       charset="us-ascii"
>
> Check out my blog entry on this topic:
> http://www.backupcentral.com/content/view/133/47/
>
> I just spent some time today with some REALLY smart folks who were using
> Intel's 10 GbE NICs with Sun 6900s and Solaris 10.  They can do about
> 250 MB/s and have the box still function.  They can get it up to 400
> MB/s, but when they do that, the box won't respond.  They couldn't run
> top, they can't login, they can't run ps, etc -- NOTHING.
>
> So I'm thinking that with Solaris, you're not going to get anywhere near
> 10,000 Mb/s (1200 MB/s).  Maybe with Linux or Windows and a TOE (TCP
> offload engine) NIC, you might have a chance.  (I only say those OSs
> because that's where they're making TOE NICs.)  One vendor replied to my
> blog post and I'm looking into it.
>
> ---
> W. Curtis Preston
> Backup Blog @ www.backupcentral.com
> VP Data Protection, GlassHouse Technologies
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Dominik
> Pietrzykowski
> Sent: Monday, August 13, 2007 4:42 PM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] 10GB network + perf questions etc
>
>
>
>
> Hi Group,
>
> Just curious to know if anyone is using 10GB network (fibre/copper, is
> copper available yet at 10GB ???) on their media servers or clients and
> what
> sort of data transfer rates are they getting ???
>
> I'm waiting for my 10GB blade for the switch to come from the US, it's
> taking ages !!!!
>
> Also, are you using Solaris or windows on these ???
>
> What's the best you've seen on 100MB and 1GB copper/fibre networks ???
>
> What data rates are you seeing via HBAs 2/4GB ????
>
> Are you using standard backup method or flashbackup ?
>
> I've played around with flash and on a server with about 4.7TB of small
> files, I have gone from 10MB/s -> 70MB/s. It's a V490
>
> Thanks in advance,
>
> Dominik
>
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
>
> ------------------------------
>
> Message: 5
> Date: Fri, 24 Aug 2007 03:07:27 -0700
> From: "Peter Marelas" <peter_marelas AT symantec DOT com>
> Subject: Re: [Veritas-bu] 10GB network + perf questions etc
> To: "Curtis Preston" <cpreston AT glasshouse DOT com>, "Dominik
>         Pietrzykowski" <dominik_pietrzykowski AT toll.com DOT au>,
>         <veritas-bu AT mailman.eng.auburn DOT edu>
> Message-ID:
>         <B3E98EAC5926D5498DDD341AE4B7D21C01C4FC6C AT 
> TUS1XCHCLUPIN06.enterprise.veritas DOT com>
>
> Content-Type: text/plain;       charset="us-ascii"
>
> If it's the card I'm thinking about the TCP checksum is already offload
> to hardware. In fact most gigaswift cards support this today with 1 Gbit
> technology.
>
> I think a great deal of tuning would be required to achieve anywhere
> near 10 GB w/ a single TCP stream.
>
> Regards
> Peter Marelas
>
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Curtis
> Preston
> Sent: Friday, 24 August 2007 3:43 PM
> To: Dominik Pietrzykowski; veritas-bu AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] 10GB network + perf questions etc
>
> Check out my blog entry on this topic:
> http://www.backupcentral.com/content/view/133/47/
>
> I just spent some time today with some REALLY smart folks who were using
> Intel's 10 GbE NICs with Sun 6900s and Solaris 10.  They can do about
> 250 MB/s and have the box still function.  They can get it up to 400
> MB/s, but when they do that, the box won't respond.  They couldn't run
> top, they can't login, they can't run ps, etc -- NOTHING.
>
> So I'm thinking that with Solaris, you're not going to get anywhere near
> 10,000 Mb/s (1200 MB/s).  Maybe with Linux or Windows and a TOE (TCP
> offload engine) NIC, you might have a chance.  (I only say those OSs
> because that's where they're making TOE NICs.)  One vendor replied to my
> blog post and I'm looking into it.
>
> ---
> W. Curtis Preston
> Backup Blog @ www.backupcentral.com
> VP Data Protection, GlassHouse Technologies
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Dominik
> Pietrzykowski
> Sent: Monday, August 13, 2007 4:42 PM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] 10GB network + perf questions etc
>
>
>
>
> Hi Group,
>
> Just curious to know if anyone is using 10GB network (fibre/copper, is
> copper available yet at 10GB ???) on their media servers or clients and
> what sort of data transfer rates are they getting ???
>
> I'm waiting for my 10GB blade for the switch to come from the US, it's
> taking ages !!!!
>
> Also, are you using Solaris or windows on these ???
>
> What's the best you've seen on 100MB and 1GB copper/fibre networks ???
>
> What data rates are you seeing via HBAs 2/4GB ????
>
> Are you using standard backup method or flashbackup ?
>
> I've played around with flash and on a server with about 4.7TB of small
> files, I have gone from 10MB/s -> 70MB/s. It's a V490
>
> Thanks in advance,
>
> Dominik
>
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
>
> ------------------------------
>
> Message: 6
> Date: Fri, 24 Aug 2007 07:58:46 -0400
> From: "Sesar, Steven L." <ssesar AT mitre DOT org>
> Subject: Re: [Veritas-bu] Has anyone run a database consistency check
>         recently?
> To: <Ian.Fehring AT nab.com DOT au>, <veritas-bu AT mailman.eng.auburn DOT 
> edu>
> Message-ID: <7DF33D4FED916C43A609D7C2B55772400476F7 AT IMCSRV7.MITRE DOT ORG>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Yes, many times. This was a huge exposure for us at the time. The issue
> was that we couldn't run a consistency check on our own, mainly due to
> the size of our catalog. We were relegated, as you are, to sending
> Symantec dumps of the catalog/databases, opening up a case and waiting
> for them to produce a report.
>
> The critical problem with that, is that by the time we actually got the
> report and acted on it, the next backup run was in flight, making the
> report  largely obsolete because the catalog/databases were now being
> written to again. Typically, it took a few days to clear up our
> inconsistencies. Also, typically, we ran this bi-weekly to keep this
> problem in check.
>
> This has been a serious issue with NBU for years. Because the
> possibility exists, as you have found out, that NBU could mark media
> with live images on it as scratch media, customers are always subject
> to losing data. Symantec knows this.
>
> At least in our environment, we, nor Symantec we never able to
> determine what we could do to keep these inconsistencies from occurring
> in the first place.
>
> At the time, the utility that Symantec used to determine
> inconsistencies was kept from customers. It was/is called NBCC. My
> understanding is that it may be available to end users now, but for a
> long time, customers were, in essence, unable to help themselves in
> this regard.
>
> So, yes, I am suggesting that inherently, NBU *is* subject to data
> loss, silently and at any time, due to the fact that their
> catalog/databases cannot remain consistent.
>
> ===================================
>
>    Steven L. Sesar
>    Lead Operating Systems Programmer/Analyst
>    UNIX Application Services R101
>    The MITRE Corporation
>    202 Burlington Road - MS K101
>    Bedford, MA 01730
>    tel: (781) 271-7702
>    fax: (781) 271-2600
>    mobile: (617) 519-8933
>    email: ssesar AT mitre DOT org
>
> ===================================
>
>
>
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu on behalf of
> Ian.Fehring AT nab.com DOT au
> Sent: Fri 8/24/2007 12:04 AM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] Has anyone run a database consistency check
> recently?
>
>
> Hi all,
> We have been having issues with our catalog (running NBU 6.0 MP4) and
> have
> found that our catalog is very inconsistent. We have been running a
> utility from Symantec called consistency_gather.pl, which basically
> grabs
> image, media and vol DB information. We send that through to Symantec
> to
> produce a report, which we receive back.
> We have two NetBackup 6.0 environment, and both are reporting that we
> have
> inconsistencies in our catalog.
> Has anyone else run this utility and obtained a report from Symantec.
> In one of our report, it stats that of the 375000 images that we have,
> about 4700 of them could be lost due to being overwritten. We also have
>
> multiple tapes that appear to be assigned to multiple media servers,
> and
> the list goes on.
> Anyone else run into this type of issue?
> Regards
> Ian.
>
>
> National Australia Bank Ltd - ABN 12 004 044 937
> This email may contain confidential information. If you are not the
> intended recipient, please immediately notify us at
> postmaster AT nab.com DOT au or by replying to the sender, and then destroy
> all copies of this email. Except where this email indicates otherwise,
> views expressed in this email are those of the sender and not of
> National Australia Bank Ltd. Advice in this email does not take account
> of your objectives, financial situation, or needs. It is important for
> you to consider these matters and, if the e-mail refers to a
> product(s), you should read the relevant Product Disclosure
> Statement(s)/other disclosure document(s) before making any decisions.
> If you do not want email marketing from us in future, forward this
> email with "unsubscribe" in the subject line to
> Unsubscriptions AT nab.com DOT au in order to stop marketing emails from this
> sender. National Australia Bank Ltd does not represent that this email
> is free of errors, viruses or interference.
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20070824/116adf6b/attachment.htm
>
> ------------------------------
>
> Message: 7
> Date: Fri, 24 Aug 2007 15:56:38 +0300
> From: <smpt1 AT peppas DOT gr>
> Subject: [Veritas-bu] Remote media server connection speed line
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Message-ID: <20070824125638.32188.qmail AT host202 DOT com>
> Content-Type: text/plain; charset="windows-1253"
>
> Does anyone know the minimum line speed requirements for a master and a media 
> server communication?
>
> I have a remote site that connects with the primary with a 1 Mbps line. Will 
> a media server operate successfully at my remote site?
>
> thanks
>
>
> ------------------------------
>
> Message: 8
> Date: Fri, 24 Aug 2007 09:12:11 -0400
> From: "Preston, Douglas L" <dpreston AT LANDAM DOT com>
> Subject: Re: [Veritas-bu] Has anyone run a database consistency check
>         recently?
> To: <Ian.Fehring AT nab.com DOT au>, <veritas-bu AT mailman.eng.auburn DOT 
> edu>
> Message-ID:
>         <FA30C379B98F0A4592BE462AC0ED356F06048059 AT 
> SVARIEXMB07.landam.landamerica DOT net>
>
> Content-Type: text/plain;       charset="us-ascii"
>
> In windows you can run this command.
>
> "C:\Program Files\VERITAS\netbackup\bin\bpdbm" -consistency
> >>f:\catalog.consistency.txt
>
> On a *nix machine it should still be the same command
>
> While it checks the catalog it will move corrupt entries to
> C:\Program Files\VERITAS\netbackup\db.corrupt and your catalog backups
> will complete.  Using this command to do the consistency check and fix
> it does not delete the corrupt entries so if a fix for the corrupt
> entries ever happen you still have them.  On our catalog it takes
> approxamately 2 hours to run.  Our catalog is 60+ gigs.
>
> Doug Preston
> Systems Engineer
> Land America Tax and Flood Services
> Phone 626-339-5221 Ext 1104
> Email  dlpreston AT landam DOT com
>
>
> ------------------------------------------------------------------------
> ------------
> NOTICE: This electronic mail transmission may constitute a communication
> that is legally privileged. It is not intended for transmission to, or
> receipt by, any unauthorized persons. If you have received this
> electronic mail transmission in error, please delete it from your system
> without copying it, and notify the sender by reply e-mail, so that our
> address record can be corrected.
> ------------------------------------------------------------------------
> ------------
>
>
>
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of
> Ian.Fehring AT nab.com DOT au
> Sent: Thursday, August 23, 2007 9:05 PM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] Has anyone run a database consistency check
> recently?
>
> Hi all,
>
> We have been having issues with our catalog (running NBU 6.0 MP4) and
> have found that our catalog is very inconsistent. We have been running a
> utility from Symantec called consistency_gather.pl, which basically
> grabs image, media and vol DB information. We send that through to
> Symantec to produce a report, which we receive back.
>
> We have two NetBackup 6.0 environment, and both are reporting that we
> have inconsistencies in our catalog.
>
> Has anyone else run this utility and obtained a report from Symantec.
>
> In one of our report, it stats that of the 375000 images that we have,
> about 4700 of them could be lost due to being overwritten. We also have
> multiple tapes that appear to be assigned to multiple media servers, and
> the list goes on.
>
> Anyone else run into this type of issue?
>
> Regards
> Ian.
>
> National Australia Bank Ltd - ABN 12 004 044 937 This email may contain
> confidential information. If you are not the intended recipient, please
> immediately notify us at postmaster AT nab.com DOT au or by replying to the
> sender, and then destroy all copies of this email. Except where this
> email indicates otherwise, views expressed in this email are those of
> the sender and not of National Australia Bank Ltd. Advice in this email
> does not take account of your objectives, financial situation, or needs.
> It is important for you to consider these matters and, if the e-mail
> refers to a product(s), you should read the relevant Product Disclosure
> Statement(s)/other disclosure document(s) before making any decisions.
> If you do not want email marketing from us in future, forward this email
> with "unsubscribe" in the subject line to Unsubscriptions AT nab.com DOT au in
> order to stop marketing emails from this sender. National Australia Bank
> Ltd does not represent that this email is free of errors, viruses or
> interference.
>
>
>
>
> ------------------------------
>
> Message: 9
> Date: Fri, 24 Aug 2007 09:13:10 -0400 (EDT)
> From: Justin Piszcz <jpiszcz AT lucidpixels DOT com>
> Subject: Re: [Veritas-bu] Remote media server connection speed line
> To: smpt1 AT peppas DOT gr
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Message-ID: <Pine.LNX.4.64.0708240912340.18608 AT p34.internal DOT lan>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> Should be fine the master is just telling the clients/media servers what
> to do and where to send the data.  As long as you have no tape drives
> attached to it-- however, if you backup millions of files those files will
> be written to in the catalog on the master.
>
> On Fri, 24 Aug 2007, smpt1 AT peppas DOT gr wrote:
>
> > Does anyone know the minimum line speed requirements for a master and a 
> > media server communication?
> >
> > I have a remote site that connects with the primary with a 1 Mbps line. 
> > Will a media server operate successfully at my remote site?
> >
> > thanks
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >
>
>
> ------------------------------
>
> Message: 10
> Date: Fri, 24 Aug 2007 09:18:08 -0400
> From: "Steven L. Sesar" <ssesar AT mitre DOT org>
> Subject: Re: [Veritas-bu] Has anyone run a database consistency check
>         recently?
> To: "Preston, Douglas L" <dpreston AT LANDAM DOT com>
> Cc: Ian.Fehring AT nab.com DOT au, veritas-bu AT mailman.eng.auburn DOT edu
> Message-ID: <46CEDA90.6030502 AT mitre DOT org>
> Content-Type: text/plain; charset="iso-8859-1"
>
> This took way too long (over 24 hours) in our environment. It was never
> an option for us, unfortunately.
>
>
> Preston, Douglas L wrote:
> > In windows you can run this command.
> >
> > "C:\Program Files\VERITAS\netbackup\bin\bpdbm" -consistency
> >
> >>> f:\catalog.consistency.txt
> >>>
> >
> > On a *nix machine it should still be the same command
> >
> > While it checks the catalog it will move corrupt entries to
> > C:\Program Files\VERITAS\netbackup\db.corrupt and your catalog backups
> > will complete.  Using this command to do the consistency check and fix
> > it does not delete the corrupt entries so if a fix for the corrupt
> > entries ever happen you still have them.  On our catalog it takes
> > approxamately 2 hours to run.  Our catalog is 60+ gigs.
> >
> > Doug Preston
> > Systems Engineer
> > Land America Tax and Flood Services
> > Phone 626-339-5221 Ext 1104
> > Email  dlpreston AT landam DOT com
> >
> >
> > ------------------------------------------------------------------------
> > ------------
> > NOTICE: This electronic mail transmission may constitute a communication
> > that is legally privileged. It is not intended for transmission to, or
> > receipt by, any unauthorized persons. If you have received this
> > electronic mail transmission in error, please delete it from your system
> > without copying it, and notify the sender by reply e-mail, so that our
> > address record can be corrected.
> > ------------------------------------------------------------------------
> > ------------
> >
> >
> >
> >
> > -----Original Message-----
> > From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of
> > Ian.Fehring AT nab.com DOT au
> > Sent: Thursday, August 23, 2007 9:05 PM
> > To: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: [Veritas-bu] Has anyone run a database consistency check
> > recently?
> >
> > Hi all,
> >
> > We have been having issues with our catalog (running NBU 6.0 MP4) and
> > have found that our catalog is very inconsistent. We have been running a
> > utility from Symantec called consistency_gather.pl, which basically
> > grabs image, media and vol DB information. We send that through to
> > Symantec to produce a report, which we receive back.
> >
> > We have two NetBackup 6.0 environment, and both are reporting that we
> > have inconsistencies in our catalog.
> >
> > Has anyone else run this utility and obtained a report from Symantec.
> >
> > In one of our report, it stats that of the 375000 images that we have,
> > about 4700 of them could be lost due to being overwritten. We also have
> > multiple tapes that appear to be assigned to multiple media servers, and
> > the list goes on.
> >
> > Anyone else run into this type of issue?
> >
> > Regards
> > Ian.
> >
> > National Australia Bank Ltd - ABN 12 004 044 937 This email may contain
> > confidential information. If you are not the intended recipient, please
> > immediately notify us at postmaster AT nab.com DOT au or by replying to the
> > sender, and then destroy all copies of this email. Except where this
> > email indicates otherwise, views expressed in this email are those of
> > the sender and not of National Australia Bank Ltd. Advice in this email
> > does not take account of your objectives, financial situation, or needs.
> > It is important for you to consider these matters and, if the e-mail
> > refers to a product(s), you should read the relevant Product Disclosure
> > Statement(s)/other disclosure document(s) before making any decisions.
> > If you do not want email marketing from us in future, forward this email
> > with "unsubscribe" in the subject line to Unsubscriptions AT nab.com DOT au 
> > in
> > order to stop marketing emails from this sender. National Australia Bank
> > Ltd does not represent that this email is free of errors, viruses or
> > interference.
> >
> >
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >
>
>
> --
> ===================================
>
>    Steven L. Sesar
>    Lead Operating Systems Programmer/Analyst
>    UNIX Application Services R101
>    The MITRE Corporation
>    202 Burlington Road - MS K101
>    Bedford, MA 01730
>    tel: (781) 271-7702
>    fax: (781) 271-2600
>    mobile: (617) 519-8933
>    email: ssesar AT mitre DOT org
>
> ===================================
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20070824/93b290a6/attachment.html
>
> ------------------------------
>
> Message: 11
> Date: Fri, 24 Aug 2007 01:19:25 -0700
> From: Paikekar <netbackup-forum AT backupcentral DOT com>
> Subject: [Veritas-bu] Re: Has anyone run a database consistency check
>         recently?
> To: VERITAS-BU AT mailman.eng.auburn DOT edu
> Message-ID: <1187943565.m2f.176680 AT www.backupcentral DOT com>
>
>
> You may find many inconsistancies , try to resolve only
>
> 1. Media inconsistancies
>  a : media assign time and allocated time
>  b : media assigned to multiple media servers
>  c : remove all media servers which are not into existance
> 2. Image inconsistancies
>
>
> Regards ,
>
> Rohit Paikekar
>
> +----------------------------------------------------------------------
> |This was sent by rohit_paikekar AT yahoo DOT com via Backup Central.
> |Forward SPAM to abuse AT backupcentral DOT com.
> +----------------------------------------------------------------------
>
>
>
>
> ------------------------------
>
> Message: 12
> Date: Fri, 24 Aug 2007 16:23:25 +0200
> From: Didier BRUN <didier.brun AT fr.thalesgroup DOT com>
> Subject: [Veritas-bu]   TIR with NDMP on Celerra ? up
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Message-ID: <46CEE9DD.1080507 AT fr.thalesgroup DOT com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi all,
>
>      Anybody knows a solution to make a TIR ( True image Restore )
> restoration  with NDMP on Celerra ?
>
> Many thanks,
>
> Didier
>
>
>
> ------------------------------
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
> End of Veritas-bu Digest, Vol 16, Issue 45
> ******************************************
>
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu