I experienced problems with SSO drives on Windows media servers in the past due
to SCSI target ID's changing whenever the media servers were rebooted (at least
once a month for OS patching). I manage several SL3000 and SL500 libraries with
a mix of HP LTO4 and IBM LTO3 drives (22 and 8 respectively). The media servers
are W2K3 x64 NBU 6.5.5 with Emulex LP11000 HBA's. All media servers are zoned
to all drives. This may not be directly related to the SCSI reservation locking
problem but the issue I had with SSO was fixed by defining persistent SCSI
target ID to LUN mappings on each HBA. This particular issue and solution is
well documented. http://seer.entsupport.symantec.com/docs/256261.htm
Regards,
Jack Roche
IMF Storage Admin
-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of
veritas-bu-request AT mailman.eng.auburn DOT edu
Sent: Tuesday, May 25, 2010 9:07 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Veritas-bu Digest, Vol 49, Issue 24
Send Veritas-bu mailing list submissions to
veritas-bu AT mailman.eng.auburn DOT edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
or, via email, send a message with subject or body 'help' to
veritas-bu-request AT mailman.eng.auburn DOT edu
You can reach the person managing the list at
veritas-bu-owner AT mailman.eng.auburn DOT edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Veritas-bu digest..."
Today's Topics:
1. Re: SCSI reserves on SSO windows drives (WEAVER, Simon (external))
2. SCSI reserves on SSO windows drives (Mike Wigington)
3. NBU 6.5.5 Eject Issue. (Ulises Rodriguez)
4. Scheduled policy backups are not starting child jobs
(mbattini AT bloomberg DOT com)
5. Re: RMAN crosscheck (Jared Still)
6. Re: RMAN crosscheck (Jared Still)
7. Re: NBU 6.5.5 Eject Issue. (Scott Jacobson)
8. Re: RMAN crosscheck (Dean)
9. Re: RMAN crosscheck (Jared Still)
10. Re: NBU 6.5.5 Eject Issue. (Mark Phillips)
----------------------------------------------------------------------
Message: 1
Date: Tue, 25 May 2010 15:05:26 +0100
From: "WEAVER, Simon \(external\)" <simon.weaver AT astrium.eads DOT net>
Subject: Re: [Veritas-bu] SCSI reserves on SSO windows drives
To: "stefanos" <smpt1 AT peppas DOT gr>, <michael.ketley AT orange-ftgroup DOT
com>,
<VERITAS-BU AT mailman.eng.auburn DOT edu>
Message-ID:
<8BCBC55B17626149A568F4E4AA24659E5F8AE6 AT VSMAIL1.uk.astrium DOT corp>
Content-Type: text/plain; charset="us-ascii"
Hi Stefanos
No its not irrelevant, as I was given the very same article from
Symantec Support. Luckily for us, the latest HP firmware corrected the
majority of our problems.
Although its still not uncommon for some engineers to be working on the
Fabric while backups are running to cause a few "minor" issues :-)
Simon
________________________________
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of stefanos
Sent: Tuesday, May 25, 2010 2:44 PM
To: michael.ketley AT orange-ftgroup DOT com; VERITAS-BU AT mailman.eng.auburn
DOT edu
Subject: Re: [Veritas-bu] SCSI reserves on SSO windows drives
Have you follow this technote for the win3k systems?
http://seer.entsupport.symantec.com/docs/268245.htm
It may seems irrelevant, but it is.
stefanos
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of
michael.ketley AT orange-ftgroup DOT com
Sent: Tuesday, May 25, 2010 4:30 PM
To: VERITAS-BU AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] SCSI reserves on SSO windows drives
Not sure if anyone can help it, but i have a most annoying problem with
SSO windows drives and reservation conflicts.
I have 8 IBM Ultrium 2 drives that are shared between 10 windows media
servers (2000+2003). Backups can work fine for a few days and then
'RESERVATION CONFLICTS' appear. Occassionally vmoprcmd
-crawlreleasebyname will release the reserve but more often than not a
trip to the datacentres and a physical reset of drive is the only
recourse to break the reserve.
Drives are all same firmware revision and tape drivers are all IBM
6.1.3.5. Netbackup 5.1 MP6, it's legacy but no current path to upgrade
so i'm stuck for a while.
Does anyone know where i should be looking to identifying what's placing
this reservation, it's not Netbackup i don't think? Or is there anything
i can do to reduce the reserves?
Regards
Mike
************************************************************************
**************
This e-mail (including any attachments) is intended for the above-named
person(s). It is confidential and may contain legally privileged
information. Any opinions expressed are not necessarily those of the
company. If you receive it in error please delete it, inform the sender
and do not copy, distribute or take any action in reliance upon it.
We may monitor all incoming and outgoing emails in line with current
legislation. We have taken steps to ensure that this email and
attachments are free from any virus, but it remains your responsibility
to ensure that viruses do not adversely affect you.
Orange Personal Communications Services Limited is an English company
(registered no. 2178917) with its address at St James Court, Great Park
Road, Almondsbury Park, Bradley Stoke, Bristol BS32 4QJ.
Orange Retail Limited is an English company (registered no. 2439104)
with its address at St James Court, Great Park Road, Almondsbury Park,
Bradley Stoke, Bristol BS32 4QJ.
Orange Home UK Limited is an English company (registered no. 3014367)
with its address at St James Court, Great Park Road, Almondsbury Park,
Bradley Stoke, Bristol BS32 4QJ.
.
************************************************************************
**************
*********************************
This message and any attachments (the "message") are confidential and
intended solely for the addressees.
Any unauthorised use or dissemination is prohibited.
Messages are susceptible to alteration.
France Telecom Group shall not be liable for the message if altered,
changed or falsified.
If you are not the intended addressee of this message, please cancel it
immediately and inform the sender.
********************************
This email (including any attachments) may contain confidential
and/or privileged information or information otherwise protected
from disclosure. If you are not the intended recipient, please
notify the sender immediately, do not copy this message or any
attachments and do not use it for any purpose or disclose its
content to any person, but delete this message and any attachments
from your system. Astrium disclaims any and all liability if this
email transmission was virus corrupted, altered or falsified.
-o-
Astrium Limited, Registered in England and Wales No. 2449259
Registered Office:
Gunnels Wood Road, Stevenage, Hertfordshire, SG1 2AS, England
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/131c3bdb/attachment-0001.htm
------------------------------
Message: 2
Date: Tue, 25 May 2010 11:11:22 -0700 (PDT)
From: Mike Wigington <redclay30253 AT yahoo DOT com>
Subject: [Veritas-bu] SCSI reserves on SSO windows drives
To: VERITAS-BU AT mailman.eng.auburn DOT edu
Message-ID: <191405.1806.qm AT web65707.mail.ac4.yahoo DOT com>
Content-Type: text/plain; charset="iso-8859-1"
Re-install the IBM tape drivers using install.exe -n for Non-Execlusive.
Non-Execlusive = no SCSI reserves.
Also,?run the DisableDPF.reg file, in the reg folder, to set the registry.?
This will reduce the number of problems but will not eliminate the problems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/a36d9128/attachment-0001.htm
------------------------------
Message: 3
Date: Tue, 25 May 2010 14:22:27 -0400
From: Ulises Rodriguez <ulises.rodriguez AT wallst DOT com>
Subject: [Veritas-bu] NBU 6.5.5 Eject Issue.
To: "'veritas-bu AT mailman.eng.auburn DOT edu'"
<veritas-bu AT mailman.eng.auburn DOT edu>
Message-ID:
<D3F095514DFF1D4D95DA857CAA9182992CC441B437 AT IS-EXCHANGEC101 DOT
wsod.local>
Content-Type: text/plain; charset="us-ascii"
Hello All,
I have an issue when I try to eject tapes, the process hangs but it only ejects
one tape. I am running NBU 6.5.5 on Win2k3 64x, Storagetek L700 library. Has
anybody seen or have ideas how to fix this?
Thank you,
Ulises.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/f796bf67/attachment-0001.htm
------------------------------
Message: 4
Date: Tue, 25 May 2010 14:44:41 -0400
From: "mbattini AT bloomberg DOT com" <mbattini AT bloomberg DOT com>
Subject: [Veritas-bu] Scheduled policy backups are not starting child
jobs
To: "veritas-bu AT mailman.eng.auburn DOT edu"
<veritas-bu AT mailman.eng.auburn DOT edu>
Cc: "lmascarenhas AT bloomberg DOT com" <lmascarenhas AT bloomberg DOT com>,
"YWu4 AT bloomberg DOT com" <YWu4 AT bloomberg DOT com>
Message-ID:
<52630004D308054DA9B6C66FC1070AE24EE370C8 AT NY2580.corp.bloomberg DOT
com>
Content-Type: text/plain; charset="us-ascii"
Hi All,
We are facing a peculiar problem , where policies starting around midnight are
launching only parent jobs, but none of those parent jobs launch child jobs for
each listed client in policy. The same client works fine if we start manual
backup with same policy. Working with Symantec actively, but if anyone faced
the problem already, would like to know how it got fixed.
Thanks
Mallik
Env: Netabckup 6.5.3.1 running on solaris 10. All master, media server and
client same versions.
Job details shows as follows:
05/24/2010 23:55:00 - requesting resource ADIC_LAN_MS
05/24/2010 23:55:00 - requesting resource nynbuprod1.NBU_CLIENT.MAXJOBS.n071-s
05/24/2010 23:55:00 - requesting resource
nynbuprod1.NBU_POLICY.MAXJOBS.fs_2355_ip
05/24/2010 23:55:08 - granted resource nynbuprod1.NBU_CLIENT.MAXJOBS.n071-s
05/24/2010 23:55:08 - granted resource nynbuprod1.NBU_POLICY.MAXJOBS.fs_2355_ip
05/24/2010 23:55:08 - granted resource N00877
05/24/2010 23:55:08 - granted resource nyprod1ms4_tld0_20cbn
05/24/2010 23:55:08 - granted resource nyprod1ms4-hcart3-tld-0
05/24/2010 23:55:11 - estimated 49077 kbytes needed
05/24/2010 23:55:11 - begin Parent Job
05/24/2010 23:55:11 - begin Stream Discovery: Start Notify Script
05/24/2010 23:55:13 - started process RUNCMD (pid=11187)
Operation Status: 0
05/24/2010 23:55:15 - end Stream Discovery: Start Notify Script; elapsed time
0:00:04
05/24/2010 23:55:15 - begin Stream Discovery: Stream Discovery
05/24/2010 23:55:19 - started process bpmount (pid=6261)
Operation Status: 0
05/24/2010 23:55:21 - end Stream Discovery: Stream Discovery; elapsed time
0:00:06
05/24/2010 23:55:21 - begin Stream Discovery: Policy Execution Manager
Preprocessed
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/f068690d/attachment-0001.htm
------------------------------
Message: 5
Date: Tue, 25 May 2010 14:38:40 -0700
From: Jared Still <jkstill AT gmail DOT com>
Subject: Re: [Veritas-bu] RMAN crosscheck
To: David McMullin <David.McMullin AT cbc-companies DOT com>
Cc: "veritas-bu AT mailman.eng.auburn DOT edu"
<veritas-bu AT mailman.eng.auburn DOT edu>
Message-ID:
<AANLkTikj3tpw2oXfjvFc6mYgEzNrL_l7-5H5-NCW0Dsf AT mail.gmail DOT com>
Content-Type: text/plain; charset="iso-8859-1"
On Wed, May 19, 2010 at 1:16 PM, David McMullin <
David.McMullin AT cbc-companies DOT com> wrote:
>
> NOTE THIS PART! but this list is incomplete if an image spans several
> media.
>
> I haven't experienced that myself.
If that were true, RMAN would be unable to restore.
Having used RMAN/NetBackup to do many restores, that statement seems
incorrect to me.
One thing that I have seen is that if the tape has been vaulted, the media
ID changes and
RMAN no longer knows the correct media to ask for.
When that happen, I have to supply a date range to our Backup Admin, and he
requests
the right tapes to restore to the time period.
As re-cataloging NetBackup tapes back into the RMAN repository is nearly
impossible,
I never expire anything from RMAN. That way when an old tape is put back
into NBU,
RMAN still has the correct names to request.
I guess it is possible to re-catalog NBU tapes back into RMAN, it just isn't
documented by Oracle.
Here's an undocumented CATALOG command you may find interesting:
http://erpondb.blogspot.com/2008/09/catalog-rman-backuppieces.html
Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Oracle Blog: http://jkstill.blogspot.com
Home Page: http://jaredstill.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/e1a1e1fd/attachment-0001.htm
------------------------------
Message: 6
Date: Tue, 25 May 2010 14:41:39 -0700
From: Jared Still <jkstill AT gmail DOT com>
Subject: Re: [Veritas-bu] RMAN crosscheck
To: "Lightner, Jeff" <jlightner AT water DOT com>
Cc: veritas-bu AT mailman.eng.auburn DOT edu, Kevin Corley
<Kevin.Corley AT apollogrp DOT edu>
Message-ID:
<AANLkTilIKVmuMsxoOdRy_2qs0zJkeqDx3er-XjSMM0Kh AT mail.gmail DOT com>
Content-Type: text/plain; charset="windows-1252"
On Mon, May 17, 2010 at 9:38 AM, Lightner, Jeff <jlightner AT water DOT com>
wrote:
> That?s unfortunate. It seems that there ought to be a way to have RMAN
> rebuild its catalog from NBU much the same the way that NBU can recover
> images into its catalog from NBU expired but not yet overwritten tapes.
>
>
>
It apparently can be done, though it requires some manual intervention:
http://erpondb.blogspot.com/2008/09/catalog-rman-backuppieces.html
Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Oracle Blog: http://jkstill.blogspot.com
Home Page: http://jaredstill.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/eb9dd797/attachment-0001.htm
------------------------------
Message: 7
Date: Tue, 25 May 2010 16:12:49 -0600
From: "Scott Jacobson" <SJACOBSO AT novell DOT com>
Subject: Re: [Veritas-bu] NBU 6.5.5 Eject Issue.
To: "'veritas-bu AT mailman.eng.auburn DOT edu'"
<veritas-bu AT mailman.eng.auburn DOT edu>, "Ulises Rodriguez"
<ulises.rodriguez AT wallst DOT com>
Message-ID: <4BFBF7010200003E000ACBF8 AT sinclair.provo.novell DOT com>
Content-Type: text/plain; charset="us-ascii"
I'd be interested in knowing if anyone is seeing this issue with an SL8500
running ACSLS.
Thanks
-sj
>>> Ulises Rodriguez <ulises.rodriguez AT wallst DOT com> 5/25/2010 12:22 PM >>>
Hello All,
I have an issue when I try to eject tapes, the process hangs but it only ejects
one tape. I am running NBU 6.5.5 on Win2k3 64x, Storagetek L700 library. Has
anybody seen or have ideas how to fix this?
Thank you,
Ulises.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/23e33068/attachment-0001.htm
------------------------------
Message: 8
Date: Wed, 26 May 2010 09:54:46 +1000
From: Dean <dean.deano AT gmail DOT com>
Subject: Re: [Veritas-bu] RMAN crosscheck
To: Jared Still <jkstill AT gmail DOT com>
Cc: "veritas-bu AT mailman.eng.auburn DOT edu"
<veritas-bu AT mailman.eng.auburn DOT edu>
Message-ID:
<AANLkTinmYrCp_xib1aAx4_C1S_kif3sXdpjaFGEC60pR AT mail.gmail DOT com>
Content-Type: text/plain; charset="iso-8859-1"
Are you sure RMAN actually knows about the tape an image is on? If it does,
that is just silly. It should just ask for a backupid, then NBU will give it
the correct tape, regardless of whether it's a duped or vault copy or
whatever. I've always been sure that that's the way it worked (Although, I'm
not a DBA and don't know that much about RMAN).
Why would RMAN, if it's using the sbt_tape "pipe" to talk to NBU, need to
know media details? It doesn't make any sense. I can understand it if you're
using RMAN without NBU to write straight to tape, but if RMAN is just piping
data to NBU, why would it need to double up on NBUs media management?
Does RMAN also track disk STUs? I know in our environment, we write all our
Oracle archive log backups to a DSSU, then they get moved out to tape and
deleted from the DSSU within a day or so. We restore regularly (to
development), and it always works fine, regardless of whether the archive
log backups are still on disk or have been moved to tape. Does that mean
RMAN is aware of the destage process? I highly doubt it.
On Wed, May 26, 2010 at 7:38 AM, Jared Still <jkstill AT gmail DOT com> wrote:
> On Wed, May 19, 2010 at 1:16 PM, David McMullin <
> David.McMullin AT cbc-companies DOT com> wrote:
>
>>
>> NOTE THIS PART! but this list is incomplete if an image spans several
>> media.
>>
>> I haven't experienced that myself.
>
> If that were true, RMAN would be unable to restore.
> Having used RMAN/NetBackup to do many restores, that statement seems
> incorrect to me.
>
> One thing that I have seen is that if the tape has been vaulted, the media
> ID changes and
> RMAN no longer knows the correct media to ask for.
>
> When that happen, I have to supply a date range to our Backup Admin, and he
> requests
> the right tapes to restore to the time period.
>
> As re-cataloging NetBackup tapes back into the RMAN repository is nearly
> impossible,
> I never expire anything from RMAN. That way when an old tape is put back
> into NBU,
> RMAN still has the correct names to request.
>
> I guess it is possible to re-catalog NBU tapes back into RMAN, it just
> isn't documented by Oracle.
>
> Here's an undocumented CATALOG command you may find interesting:
> http://erpondb.blogspot.com/2008/09/catalog-rman-backuppieces.html
>
>
> Jared Still
> Certifiable Oracle DBA and Part Time Perl Evangelist
> Oracle Blog: http://jkstill.blogspot.com
> Home Page: http://jaredstill.com
>
>
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100526/fef568b8/attachment-0001.htm
------------------------------
Message: 9
Date: Tue, 25 May 2010 17:32:03 -0700
From: Jared Still <jkstill AT gmail DOT com>
Subject: Re: [Veritas-bu] RMAN crosscheck
To: Dean <dean.deano AT gmail DOT com>
Cc: "veritas-bu AT mailman.eng.auburn DOT edu"
<veritas-bu AT mailman.eng.auburn DOT edu>
Message-ID:
<AANLkTile6n-q_KeC3cD6gZC_yOav_Qkvl3dANEL-WPHb AT mail.gmail DOT com>
Content-Type: text/plain; charset="iso-8859-1"
On Tue, May 25, 2010 at 4:54 PM, Dean <dean.deano AT gmail DOT com> wrote:
> Are you sure RMAN actually knows about the tape an image is on? If it does,
> that is just silly. It should just ask for a backupid, then NBU will give it
> the correct tape, regardless of whether it's a duped or vault copy or
> whatever. I've always been sure that that's the way it worked (Although, I'm
> not a DBA and don't know that much about RMAN).
>
>
I think you're right, it just keeps the Media ID.
It's been a few months since I have looked closely at it.
Regarding vaulted tapes: RMAN will not know that a tape has been vaulted,
so it will have the wrong media ID for requesting the tape.
If the tapes are restored into the NBU catalog, RMAN will then ask for the
correct backup pieces by name.
This not an issue with RMAN, as the API has that capability, at least
according to the docs.
It just has not been implemented by Veritas.
This would bear further investigation, as that situation may have changed
since I last looked into it.
Does RMAN also track disk STUs? I know in our environment, we write all our
> Oracle archive log backups to a DSSU, then they get moved out to tape and
> deleted from the DSSU within a day or so. We restore regularly (to
> development), and it always works fine, regardless of whether the archive
> log backups are still on disk or have been moved to tape. Does that mean
> RMAN is aware of the destage process? I highly doubt it.
>
>
There is a way for RMAN in later version to track backups that are staged to
disk and
then backed up on to tape.
I have never used it, and don't know any details about it.
Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Oracle Blog: http://jkstill.blogspot.com
Home Page: http://jaredstill.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100525/738ed4ba/attachment-0001.htm
------------------------------
Message: 10
Date: Wed, 26 May 2010 10:36:37 +0930
From: Mark Phillips <Mark.Phillips AT unisa.edu DOT au>
Subject: Re: [Veritas-bu] NBU 6.5.5 Eject Issue.
To: Scott Jacobson <SJACOBSO AT novell DOT com>,
"'veritas-bu AT mailman.eng.auburn DOT edu'"
<veritas-bu AT mailman.eng.auburn DOT edu>, Ulises Rodriguez
<ulises.rodriguez AT wallst DOT com>
Message-ID:
<FDC0E8F0B26A334E92DE2BE4ECDAFF03627D38E8 AT
ITUPC-EX1MBOX.UniNet.unisa.edu DOT au>
Content-Type: text/plain; charset="us-ascii"
We are running NBU 6.5.5 on RHEL5.
4 identical RHEL5 media servers each attached to its own DELL ML6000 tape
library, the drives being shared.
We have this issue on only one of the libraries.
After seeing this problem the firmware on all of the libraries was upgraded to
the latest version available.
The firmware upgrade didn't fix the problem.
There were no hardware errors before or after the firmware upgrade.
It's intermittent , sometimes working and there doesn't seem to be a pattern to
the failures.
On occasion we've had to resort to using robtest to get the tapes into the
access port.
Planning on going to NBU 6.5.6 in the next few weeks, maybe this will fix the
problem.
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Scott
Jacobson
Sent: Wednesday, 26 May 2010 7:43 AM
To: 'veritas-bu AT mailman.eng.auburn DOT edu'; Ulises Rodriguez
Subject: Re: [Veritas-bu] NBU 6.5.5 Eject Issue.
I'd be interested in knowing if anyone is seeing this issue with an SL8500
running ACSLS.
Thanks
-sj
>>> Ulises Rodriguez <ulises.rodriguez AT wallst DOT com> 5/25/2010 12:22 PM >>>
Hello All,
I have an issue when I try to eject tapes, the process hangs but it only ejects
one tape. I am running NBU 6.5.5 on Win2k3 64x, Storagetek L700 library. Has
anybody seen or have ideas how to fix this?
Thank you,
Ulises.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20100526/c3633d14/attachment.htm
------------------------------
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
End of Veritas-bu Digest, Vol 49, Issue 24
******************************************
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
