Hi Tim -
Well, I've just determined that my hba drivers are current and
I've successfully used ufsrestore on the third ufsdump on a tape.
I've read the man page for tcopy and am not clear on what it would actually
show me about either the ufsdump tape or a netbackup tape.
It really looks like I'm running out of options here...
I'll let you know how the tcopy goes.
On Wed, 12 Jan 2005, Tim Hoke wrote:
> Date: Wed, 12 Jan 2005 13:25:38 -0600
> From: Tim Hoke <thoke AT northpeak DOT org>
> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> Subject: Re: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov. 2004)
>
> keep in mind that ufsdump isn't going to do the block checking like
> NBU... and may not care if the block sizes are broken up.
>
>
> On Jan 12, 2005, at 12:40 PM, Kathryn Hemness wrote:
>
> > Hi Tim -
> >
> > I haven't done anything with tcopy yet, but I successfully stacked
> > 3 separate ufsdumps onto one tape. I just finished a ufsrestore from
> > the third ufsdump.
> >
> > I'm checking on the firmware level of the HBAs now...but it seems like
> > old firmware would have affected the ufsdump and restore.
> >
> > On Wed, 12 Jan 2005, Tim Hoke wrote:
> >
> >> Date: Wed, 12 Jan 2005 09:41:47 -0600
> >> From: Tim Hoke <thoke AT northpeak DOT org>
> >> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >> Subject: Re: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov. 2004)
> >>
> >> Kathryn,
> >>
> >> I don't know much about the HBA firmware/driver revisions, so can't
> >> help with that one.
> >>
> >> Were you able to run tcopy on one of the tapes?
> >>
> >> -Tim
> >>
> >> On Jan 12, 2005, at 9:04 AM, Kathryn Hemness wrote:
> >>
> >>>
> >>> I'm current up to 113277-26. I'll be looking at the HBA drivers next.
> >>>
> >>> Here's what the lsiutil reports for my HBAs:
> >>>
> >>> Port Name Chip Vendor/Type MPT Rev Firmware Rev
> >>> 1. itmpt0 LSI Logic FC919X 103 01020000
> >>> 2. itmpt1 LSI Logic FC919X 103 01020000
> >>>
> >>> These HBAs are also only 2 months old, so I'd expect the FW to
> >>> be current.
> >>>
> >>>
> >>>
> >>> On Wed, 12 Jan 2005, Tim Hoke wrote:
> >>>
> >>>> Date: Wed, 12 Jan 2005 08:43:36 -0600
> >>>> From: Tim Hoke <thoke AT northpeak DOT org>
> >>>> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>
> >>>> Cc: "Chapman, Scott" <Scott.Chapman AT icbc DOT com>
> >>>> Subject: Re: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>> 2004)
> >>>>
> >>>> When you say VERITAS Drivers, that would typically mean Windows
> >>>> platforms. The only "VERITAS Driver" that is provided for Solaris
> >>>> is
> >>>> the sg driver (scsi passthru).
> >>>>
> >>>> So, you should be using the SUN st driver (SCSI Tape).
> >>>>
> >>>> According to my records (Sunsolve), the st driver is in SUN patch
> >>>> 113277 and the current revision is -26. Native support for your IBM
> >>>> Ultrium-TD2 drives was introduced in the -10 release. So, as long
> >>>> as
> >>>> you are at -10 or above, you shouldn't be using any st.conf entries.
> >>>> However, I don't work for Sun support, so you really should verify
> >>>> it
> >>>> with them.
> >>>>
> >>>> I'd also suspect any HBA drivers/firmware or any other fiber devices
> >>>> too.
> >>>>
> >>>> -Tim
> >>>>
> >>>> On Jan 12, 2005, at 8:12 AM, Kathryn Hemness wrote:
> >>>>
> >>>>> I've also googled and found that article. I'm definitely using the
> >>>>> veritas drivers.
> >>>>>
> >>>>>
> >>>>> On Tue, 11 Jan 2005, Chapman, Scott wrote:
> >>>>>
> >>>>>> Date: Tue, 11 Jan 2005 20:07:10 -0800
> >>>>>> From: "Chapman, Scott" <Scott.Chapman AT icbc DOT com>
> >>>>>> To: 'Kathryn Hemness' <kfhemness AT ucdavis DOT edu>
> >>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>>>> 2004)
> >>>>>>
> >>>>>> Kathryn, I did a google search with "External event caused rewind
> >>>>>> during"
> >>>>>> and the first hit mentions something from your logs:
> >>>>>>> From the 5.0mp2 patch release . . .
> >>>>>> "For a checkpoint restart backup, in the rare case where bptm
> >>>>>> detects
> >>>>>> that a position check problem occurred following a checkpoint
> >>>>>> because of
> >>>>>> a
> >>>>>> misconfigured drive or a rewind from an external source, and
> >>>>>> the
> >>>>>> backup
> >>>>>> is
> >>>>>> later resumed, the information on the tape prior to the
> >>>>>> checkpoint
> >>>>>> may be
> >>>>>>
> >>>>>> invalid.
> >>>>>>
> >>>>>> The bptm log would indicate the position check problem with one
> >>>>>> of
> >>>>>> the
> >>>>>> following logs after a checkpoint:
> >>>>>>
> >>>>>> 08:39:57.969 [4393] <16> write_data: FREEZING media id 00011,
> >>>>>> too
> >>>>>> many
> >>>>>> data blocks written, check tape/driver block size configuration
> >>>>>>
> >>>>>> OR
> >>>>>>
> >>>>>> log.041204:14:39:12.373 [6416] <16> write_data: FREEZING media
> >>>>>> id
> >>>>>> 00005,
> >>>>>> <<<< here is what your logs reflect also
> >>>>>> External event caused rewind during write, all data on media is
> >>>>>> lost
> >>>>>>
> >>>>>> The problem would occur if the same backup were resumed and
> >>>>>> completed
> >>>>>> with
> >>>>>> a successful status."
> >>>>>>
> >>>>>> The one thing is this piece of information is that they mention a
> >>>>>> misconfigured drive.
> >>>>>>
> >>>>>> Question 1) Do you have the latest st driver patch installed on
> >>>>>> the
> >>>>>> backup
> >>>>>> server?
> >>>>>> Question 2) You are using the Veritas tape drivers right? This is
> >>>>>> very
> >>>>>> important, as there doesn't seem to be many people having luck
> >>>>>> with
> >>>>>> non-veritas drivers.
> >>>>>>
> >>>>>> Here is the google search
> >>>>>> http://www.google.ca/search?
> >>>>>> hl=en&q=%22External+event+caused+rewind+during%2
> >>>>>> 2&meta=
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Kathryn Hemness [mailto:kfhemness AT ucdavis DOT edu]
> >>>>>> Sent: Tuesday, January 11, 2005 4:36 PM
> >>>>>> To: veritas-bu AT mailman.eng.auburn DOT edu
> >>>>>> Cc: scott.chapman AT icbc DOT com; song_1977 AT yahoo DOT com
> >>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>>>> 2004)
> >>>>>>
> >>>>>>
> >>>>>> Greetings --
> >>>>>>
> >>>>>> I ran a successful backup using the
> >>>>>> /opt/openv/netbackup/db/config/NO_POSITION_CHECK
> >>>>>> setting suggested by Scott Chapman.
> >>>>>>
> >>>>>> Then I google'd for NO_POSITION_CHECK and found the following
> >>>>>> Veritas
> >>>>>> Support patch readme which had a good explanation for the behavior
> >>>>>> I'm
> >>>>>> seeing:
> >>>>>>
> >>>>>> http://seer.support.veritas.com/docs/246368.htm
> >>>>>>
> >>>>>> What's really funny is that this readme is for NB3.4 in 2002.
> >>>>>>
> >>>>>> Now that I know the cause of the problem, I need to determine a
> >>>>>> solution
> >>>>>> which will enable me to use the checkpoint restart feature of
> >>>>>> NetBackup 5.1.
> >>>>>>
> >>>>>> I welcome any suggestions. I'm hoping there are easy Solaris or
> >>>>>> LSI
> >>>>>> Logic
> >>>>>> HBA commands for the final solution.
> >>>>>>
> >>>>>> On Tue, 11 Jan 2005, Chapman, Scott wrote:
> >>>>>>
> >>>>>>> Date: Tue, 11 Jan 2005 15:37:29 -0800
> >>>>>>> From: "Chapman, Scott" <Scott.Chapman AT icbc DOT com>
> >>>>>>> To: 'Kathryn Hemness' <kfhemness AT ucdavis DOT edu>
> >>>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>>>>> 2004)
> >>>>>>>
> >>>>>>>
> >>>>>>> I am running 4.5fp5 and 5.0 at a different site. You aren't
> >>>>>>> running
> >>>>>>> the IBM driver for the tape drives are you? I know that has
> >>>>>>> caused
> >>>>>>> some problems for people.
> >>>>>>>
> >>>>>>> What does "sgscan -v conf" show? When I run that it confirms
> >>>>>>> that
> >>>>>>> the
> >>>>>>> drive config does not come from the st.conf by putting
> >>>>>>> "NOT-IN-ST-CONFIG-FILE" at the end of each tape drive line . . .
> >>>>>>>
> >>>>>>> Scott Chapman
> >>>>>>> ICBC - Victoria, Government St.
> >>>>>>> Phone: 250.414.7650 Cell: 250.213.9295
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Kathryn Hemness [mailto:kfhemness AT ucdavis DOT edu]
> >>>>>>> Sent: Tuesday, January 11, 2005 3:13 PM
> >>>>>>> To: K Chapman
> >>>>>>> Cc: Chapman, Scott; veritas-bu AT mailman.eng.auburn DOT edu;
> >>>>>>> song_1977 AT yahoo DOT com
> >>>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>>>>> 2004)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Turning off checkpoints was something I did early in my
> >>>>>>> troubleshooting attempts.
> >>>>>>>
> >>>>>>> I've just turned off a couple of Solaris storage managment
> >>>>>>> daemons
> >>>>>>> (ssdgrptd and
> >>>>>>> ssagent) on my server and am running another test backup now. It
> >>>>>>> should finish in about 15 more minutes.
> >>>>>>>
> >>>>>>> I'll try the NO_POSITION_CHECK after this test finishes and let
> >>>>>>> you
> >>>>>>> know what happens.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, 11 Jan 2005, K Chapman wrote:
> >>>>>>>
> >>>>>>>> Date: Tue, 11 Jan 2005 14:54:31 -0800 (PST)
> >>>>>>>> From: K Chapman <tech2187 AT yahoo DOT com>
> >>>>>>>> To: Kathryn Hemness <kfhemness AT ucdavis DOT edu>,
> >>>>>>>> "Chapman, Scott" <Scott.Chapman AT icbc DOT com>
> >>>>>>>> Cc: veritas-bu AT mailman.eng.auburn DOT edu, song_1977 AT yahoo DOT
> >>>>>>>> com
> >>>>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of Tape (from Nov.
> >>>>>>>> 2004)
> >>>>>>>>
> >>>>>>>> as a test, can you try with the position check turned
> >>>>>>>> off?
> >>>>>>>>
> >>>>>>>> touch /opt/openv/netbackup/db/config/NO_POSITION_CHECK
> >>>>>>>>
> >>>>>>>> --- Kathryn Hemness <kfhemness AT ucdavis DOT edu> wrote:
> >>>>>>>>
> >>>>>>>>> Hi, Scott -
> >>>>>>>>>
> >>>>>>>>> Here's the output of my sgscan -v:
> >>>>>>>>>
> >>>>>>>>> /dev/sg/c0t3l2: Tape (/dev/rmt/0): "IBM
> >>>>>>>>> ULTRIUM-TD2 4770"
> >>>>>>>>> /dev/sg/c0t3l3: Tape (/dev/rmt/1): "IBM
> >>>>>>>>> ULTRIUM-TD2 4770"
> >>>>>>>>> /dev/sg/c0t3l4: Tape (/dev/rmt/2): "IBM
> >>>>>>>>> ULTRIUM-TD2 4770"
> >>>>>>>>>
> >>>>>>>>> We got the library in October. The drives should be
> >>>>>>>>> at the current FW level.
> >>>>>>>>>
> >>>>>>>>> Are you using NB51?
> >>>>>>>>>
> >>>>>>>>> On Tue, 11 Jan 2005, Chapman, Scott wrote:
> >>>>>>>>>
> >>>>>>>>>> Date: Tue, 11 Jan 2005 10:42:36 -0800
> >>>>>>>>>> From: "Chapman, Scott" <Scott.Chapman AT icbc DOT com>
> >>>>>>>>>> To: 'Kathryn Hemness' <kfhemness AT ucdavis DOT edu>,
> >>>>>>>>>> veritas-bu AT mailman.eng.auburn DOT edu
> >>>>>>>>>> Cc: song_1977 AT yahoo DOT com
> >>>>>>>>>> Subject: RE: [Veritas-bu] RE:Veritas-bu] End of
> >>>>>>>>> Tape (from Nov. 2004)
> >>>>>>>>>>
> >>>>>>>>>> Kathryn are you running current firmware on the
> >>>>>>>>> LTO2 drives? I seem to
> >>>>>>>>>> remember something about old firmware doing
> >>>>>>>>> rewinds before netbackup was
> >>>>>>>>>> done with the drive . . .
> >>>>>>>>>>> From your logs:
> >>>>>>>>>> 01/10/2005 13:48:50 albus.ucdavis.edu
> >>>>>>>>> albus.ucdavis.edu FREEZING media id
> >>>>>>>>>> 040004, External event caused rewind during write,
> >>>>>>>>> all data on media is lost
> >>>>>>>>>>
> >>>>>>>>>> I am running IBM drives (we don't use the LSI
> >>>>>>>>> logic HBA's) and here is some
> >>>>>>>>>> output from sgscan -v conf:
> >>>>>>>>>> /dev/sg/c2t0l0: Tape (/dev/rmt/0): "IBM
> >>>>>>>>> ULTRIUM-TD2 38D0" :
> >>>>>>>>>> NOT-IN-ST-CONFIG-FILE
> >>>>>>>>>> /dev/sg/c2t1l0: Tape (/dev/rmt/1): "IBM
> >>>>>>>>> ULTRIUM-TD2 38D0" :
> >>>>>>>>>> NOT-IN-ST-CONFIG-FILE
> >>>>>>>>>> /dev/sg/c2t2l0: Tape (/dev/rmt/2): "IBM
> >>>>>>>>> ULTRIUM-TD2 38D0" :
> >>>>>>>>>> NOT-IN-ST-CONFIG-FILE
> >>>>>>>>>> ...
> >>>>>>>>>>
> >>>>>>>>>> I don't have anything in the st.conf for the
> >>>>>>>>> drives as they have been added
> >>>>>>>>>> to the st several patches ago. You might check
> >>>>>>>>> you st patch level as well .
> >>>>>>>>>> . .
> >>>>>>>>>>
> >>>>>>>>>> Hope this helps.
> >>>>>>>>>>
> >>>>>>>>>> Scott Chapman
> >>>>>>>>>> ICBC - Victoria, Government St.
> >>>>>>>>>> Phone: 250.414.7650 Cell: 250.213.9295
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Kathryn Hemness
> >>>>>>>>> [mailto:kfhemness AT ucdavis DOT edu]
> >>>>>>>>>> Sent: Tuesday, January 11, 2005 10:02 AM
> >>>>>>>>>> To: veritas-bu AT mailman.eng.auburn DOT edu
> >>>>>>>>>> Cc: song_1977 AT yahoo DOT com
> >>>>>>>>>> Subject: [Veritas-bu] RE:Veritas-bu] End of Tape
> >>>>>>>>> (from Nov. 2004)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Good Morning --
> >>>>>>>>>>
> >>>>>>>>>> Was there ever a resolution to your NB5.0MP2/LTO
> >>>>>>>>> end of tape problem?
> >>>>>>>>>>
> >>>>>>>>>> I'm currently fighting with a new installation
> >>>>>>>>> NB5.1 on a Solaris 9 system
> >>>>>>>>>> using
> >>>>>>>>>> LTO2 tape drives. My backups ALWAYS fail either
> >>>>>>>>> at a checkpoint-restart
> >>>>>>>>>> WRITE or
> >>>>>>>>>> at the very last WRITE of the backup, regardless
> >>>>>>>>> of how big the backup is.
> >>>>>>>>>>
> >>>>>>>>>> I've been told by my NetBackup tech support (via
> >>>>>>>>> Sun) that it was a hardware
> >>>>>>>>>> configuration problem.
> >>>>>>>>>>
> >>>>>>>>>> The backups always fail, regardless of any st.conf
> >>>>>>>>> modifications and I've
> >>>>>>>>>> even
> >>>>>>>>>> taken the fiber switch out of the mix. Here's a
> >>>>>>>>> summary of my hardware and
> >>>>>>>>>> the
> >>>>>>>>>> types of errors I'm seeing (by the way, ufsdump
> >>>>>>>>> works just fine....).
> >>>>>>>>>>
> >>>>>>>>>> Master: Solaris 9 version 4/04 on a Sun V240 with
> >>>>>>>>> 2 LSI Logic FC919X HBAs
> >>>>>>>>>> running
> >>>>>>>>>> NB5.1 Enterprise Server. One LSI Logic HBA is
> >>>>>>>>> connected directly to the
> >>>>>>>>>> fiber/scsi
> >>>>>>>>>> bridge of a Qualstar 88264 LTO2 library, the other
> >>>>>>>>> to a Brocade 32-port
> >>>>>>>>>> fiber
> >>>>>>>>>> switch attached to a Sun 3511 storage array.
> >>>>>>>>>>
> >>>>>>>>>> I have tried at least 4 different st.conf LTO2
> >>>>>>>>> configurations with same
> >>>>>>>>>> failing
> >>>>>>>>>> results and am now not using any special LTO2
> >>>>>>>>> definitions.
> >>>>>>>>>>
> >>>>>>>>>> Here are the failure errors from both the
> >>>>>>>>> NetBackup reports and from the
> >>>>>>>>>> bptm logs:
> >>>>>>>>>>
> >>>>>>>>>> 01/10/2005 13:48:50 albus.ucdavis.edu
> >>>>>>>>> albus.ucdavis.edu FREEZING media id
> >>>>>>>>>> 040004, External event caused rewind during write,
> >>>>>>>>> all data on media is lost
> >>>>>>>>>> 01/10/2005 13:48:54 albus.ucdavis.edu
> >>>>>>>>> albus.ucdavis.edu CLIENT
> >>>>>>>>>> albus.ucdavis.edu POLICY IR-ISM_02 SCHED
> >>>>>>>>> WeeklyFull EXIT STATUS 84 (media
> >>>>>>>>>> write error)
> >>>>>>>>>> 01/10/2005 13:48:54 albus.ucdavis.edu
> >>>>>>>>> albus.ucdavis.edu backup of client
> >>>>>>>>>> albus.ucdavis.edu exited with status 84 (media
> >>>>>>>>> write error)
> >>>>>>>>>>
> >>>>>>>>>> Here's the bptm log entry for the above error:
> >>>>>>>>>>
> >>>>>>>>>> 13:48:48.032 [1297] <2> write_backup: tp.tv_sec =
> >>>>>>>>> 1105393728, stp.tv_sec =
> >>>>>>>>>> 1105391634, tp.tv_usec = 27455, stp.tv_usec =
> >>>>>>>>> 544901, et = 2093483,
> >>>>>>>>>> mpx_total_kbytes[TWIN_INDEX = 0] = 21261376 13:48:48.075
> >>>>>>>>>> [1297]
> >>>>>>>>>> <2> io_terminate_tape: writing
> >>>>>>>>> empty backup header,
> >>>>>>>>>> drive index 0, copy 1
> >>>>>>>>>> 13:48:48.091 [1297] <2> io_ioctl: command
> >>>>>>>>> (0)MTWEOF 1 from (bptm.c.7919) on
> >>>>>>>>>> drive index 0
> >>>>>>>>>> 13:48:48.645 [1297] <2> io_write_back_header:
> >>>>>>>>> drive index 0, empty_file,
> >>>>>>>>>> file num = 2, mpx_headers = 0, copy 1
> >>>>>>>>>> 13:48:48.650 [1297] <2> io_close: closing
> >>>>>>>>>> /usr/openv/netbackup/db/media/tpreq/040004, from
> >>>>>>>>> bptm.c.8046
> >>>>>>>>>> 13:48:50.848 [1297] <2> io_terminate_tape:
> >>>>>>>>> absolute block position prior to
> >>>>>>>>>> writing empty header is 332201, copy 1
> >>>>>>>>>> 13:48:50.848 [1297] <2> io_terminate_tape: block
> >>>>>>>>> position check: actual
> >>>>>>>>>> 332201, expected 332213
> >>>>>>>>>> 13:48:50.848 [1297] <2> set_job_details: Sending
> >>>>>>>>> Tfile jobid (907)
> >>>>>>>>>> 13:48:50.848 [1297] <2> set_job_details: LOG
> >>>>>>>>> 1105393730 16 bptm 1297
> >>>>>>>>>> FREEZING media id 040004, External event caused
> >>>>>>>>> rewind during write, all
> >>>>>>>>>> data on media is lost
> >>>>>>>>>>
> >>>>>>>>>> 13:48:50.848 [1297] <2> set_job_details: Done 13:48:50.880
> >>>>>>>>>> [1297] <16> io_terminate_tape:
> >>>>>>>>> FREEZING media id 040004,
> >>>>>>>>>> External event caused rewind during write, all
> >>>>>>>>> data on media is lost
> >>>>>>>>>> 13:48:50.898 [1297] <2> log_media_error:
> >>>>>>>>> successfully wrote to error file -
> >>>>>>>>>> 01/10/05 13:48:50 040004 0 WRITE_ERROR
> >>>>>>>>>> 13:48:50.910 [1297] <2> check_error_history:
> >>>>>>>>> called from bptm line 17870,
> >>>>>>>>>> EXIT_Status = 84
> >>>>>>>>>> 13:48:50.911 [1297] <2> check_error_history: drive
> >>>>>>>>> index = 0, media id =
> >>>>>>>>>> 040004, time = 01/10/05 13:48:50, both_match = 0,
> >>>>>>>>> media_match = 0,
> >>>>>>>>>> drive_match = 0
> >>>>>>>>>> 13:48:50.911 [1297] <2> tpunmount:
> >>>>>>>>> Check_for_waiting = 0,
> >>>>>>>>>> No_tpunmount_after_restore = 0,
> >>>>>>>>> Media_Unmount_Delay = 0, MediaOffset = 4
> >>>>>>>>>> 13:48:50.911 [1297] <2> tpunmount: tpunmount'ing
> >>>>>>>>>> /usr/openv/netbackup/db/media/tpreq/040004
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Since ufsdump works, this is indicating a
> >>>>>>>>> NetBackup 5.1 problem. Anyway, I
> >>>>>>>>>> notice
> >>>>>>>>>> in your post-November posts, you referred to NB4.5
> >>>>>>>>> servers. Did you have to
> >>>>>>>>>> downgrade NetBackup in order to get your LTO
> >>>>>>>>> drives to work properly?
> >>>>>>>>>
> >>>>>>>> === message truncated ===
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> =====
> >>>>>>>> aaarrrggghhh!!!!
> >>>>>>>> FreeBSD rocks
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> __________________________________
> >>>>>>>> Do you Yahoo!?
> >>>>>>>> Yahoo! Mail - Find what you need with new enhanced search.
> >>>>>>>> http://info.mail.yahoo.com/mail_250
> >>>>>>>>
> >>>>>>>
> >>>>>>> --kathy
> >>>>>>>
> >>>>>>> =================================================================
> >>>>>>> ==
> >>>>>>> ==
> >>>>>>> =
> >>>>>>> ======
> >>>>>>> ===
> >>>>>>> Kathryn Hemness kfhemness AT ucdavis DOT edu
> >>>>>>> System Administrator phone: 530.752.6547
> >>>>>>> Campus Data Center & Client Services fax: 530.752.9154
> >>>>>>>
> >>>>>>
> >>>>>> --kathy
> >>>>>>
> >>>>>> ==================================================================
> >>>>>> ==
> >>>>>> ==
> >>>>>> ======
> >>>>>> ===
> >>>>>> Kathryn Hemness kfhemness AT ucdavis DOT edu
> >>>>>> System Administrator phone: 530.752.6547
> >>>>>> Campus Data Center & Client Services fax: 530.752.9154
> >>>>>>
> >>>>>
> >>>>> --kathy
> >>>>>
> >>>>> ===================================================================
> >>>>> ==
> >>>>> ==
> >>>>> ========
> >>>>> Kathryn Hemness kfhemness AT ucdavis DOT edu
> >>>>> System Administrator phone: 530.752.6547
> >>>>> Campus Data Center & Client Services fax: 530.752.9154
> >>>>> _______________________________________________
> >>>>> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> >>>>> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >>>>>
> >>>>
> >>>
> >>> --kathy
> >>>
> >>> =====================================================================
> >>> ==
> >>> ========
> >>> Kathryn Hemness kfhemness AT ucdavis DOT edu
> >>> System Administrator phone: 530.752.6547
> >>> Campus Data Center & Client Services fax: 530.752.9154
> >>>
> >>
> >
> > --kathy
> >
> > =======================================================================
> > ========
> > Kathryn Hemness kfhemness AT ucdavis DOT edu
> > System Administrator phone: 530.752.6547
> > Campus Data Center & Client Services fax: 530.752.9154
> >
>
--kathy
===============================================================================
Kathryn Hemness kfhemness AT ucdavis DOT edu
System Administrator phone: 530.752.6547
Campus Data Center & Client Services fax: 530.752.9154
|
