Re: [Networker] Linux Tape Drive Re-Ordering Issue

You nailed it  Frank. 

I have a meeting tomorrow with my Networker DSE and will show him this
email. My Linux guy also says this is coming from Networker as it is not
handling bad drives very well when it puts them in service mode.


Raymond Davis

Quest Diagnostics | Sr. Storage Engineer | 1200 Wall St West |
Lyndhurst, NJ 07071  USA | phone +1.201.729.7882 |  mobile
+1.201.841.2335 | Raymond.P.Davis AT QuestDiagnostics DOT com |
www.QuestDiagnostics.com 

Please think about resource conservation before you print this message

 


-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Francis Swasey
Sent: Tuesday, November 09, 2010 07:17 AM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: Re: [Networker] Linux Tape Drive Re-Ordering Issue

Hi Jee,

Sadly, it has everything to do with NetWorker these days.  Starting with
7.5 NetWorker now records the serial numbers from the drives in the
jukebox and if it finds a mismatch (as it will after you physically
replace a drive that has worn out), it refuses to use that drive.

You have to perform a scan for new devices from the NMC on the system
that sees the robotics of the jukebox and then wait about 5 minutes for
NetWorker to figure out that the serial number of the drive has changed.

If you do not perform that scan from the NMC, NetWorker will load a tape
into the drive and then yell about the serial number mismatch and refuse
to do anything further (so you have a tape in a drive where it belongs
and networker will not even unload the tape because the serial number is
wrong on the drive and NetWorker thinks you've got a drive ordering
problem).

As I've recently had a run on physical drive replacements, I've become
intimately familiar with this new addition to the drive replacement
procedure.

Therefore, NetWorker has to document this process because NetWorker is
now integrally involved in it.

EMC also owes us a way to perform the scan from the command line.  There
are times (like when you are using a VPN client) that the NMC java
applet will not actually work, which is a large pain in the backside
when it happens...  My experience with this is using the Cisco VPN
clients (including the SSL AnyConnect client which UVM just rolled out).
However, I assume that any daemon on my workstation/laptop that somehow
inserts itself between the OS and the Hardware NIC would cause the same
problem.  And the only way I can figure this is a problem is that the
NMC java applet was written to tickle the Hardware directly instead of
going through the OS for its network traffic -- which, as far as I'm
concerned is violating programming 101.

Frank

On 11/8/10 2:42 PM, jee wrote:
> Hi Frank,
> 
> If by "replace" you mean "physically replace", then don't look for 
> that in the netwoker documentation. It won't cover that because that 
> has nothing  to do with networker troubleshooting and/or
administration.
> 
> You should be able to find that in formation in the corresponding 
> service
> (hardware) manuals.
> 
> 
> jee
> 
> 
> 
> On Monday 08 November 2010 13:31:29 Francis Swasey wrote:
>> Brian,
>>
>> Thanks for that pointer.  Sadly, it doesn't cover how to replace a 
>> failed tape drive in a library at all (at least there is not one 
>> single use of the word "replace" in the entire document).  I'll
peruse it in depth later...
>>
>> Frank
>>
>> On 11/5/10 3:19 PM, Brian Narkinsky wrote:
>>> You have to dig through PowerLink a bit to find this but there is a 
>>> pretty good paper on how to configure Tape Drives in Networker.  Lot

>>> of good info in here.
>>>
>>> EMC(r) NetWorker(r)
>>> Configuring Tape Devices for EMC NetWorker Technical Note P/N 
>>> 300-008-352 REV A04
>>>
>>> On Fri, Nov 5, 2010 at 9:24 AM, Francis Swasey 
>>> <Frank.Swasey AT uvm DOT edu>
> wrote:
>>>> Have you tested what happens when a tape drive fails and you have 
>>>> to replace it with a new one (which has a different serial number) 
>>>> doing it that way?
>>>>
>>>> For those of us that created our jukebox back in the 7.2 (or 
>>>> earlier) days and have the jukebox defined using the 
>>>> rd=<storagenode>:/dev/nstX devices, I strongly urge you to consider

>>>> coercing udev into picking which /dev/nstX device the drive is 
>>>> based on the serial number (then remembering to update that rule 
>>>> when you replace a drive).
>>>>
>>>> I have set it up as /etc/udev/rules.d/20-local.rules -- which is a 
>>>> copy of the 50-udev.rules file with a line added for each tape 
>>>> drive like:
>>>>
>>>> KERNEL=="nst[0-9]*", SUBSYSTEM=="scsi_tape", 
>>>> ENV{ID_SERIAL}=="3500507631203e5d4", NAME="nst0",
>>>> SYMLINK+="tape/by-id/$env{ID_BUS}-$env{ID_SERIAL}-nst",
SYMLINK+="tape0"
>>>>
>>>> just in front of the
>>>>
>>>> KERNEL=="nst[0-9]*", SUBSYSTEM=="scsi_tape", ENV{ID_SERIAL}=="?*",
>>>> SYMLINK+="tape/by-id/$env{ID_BUS}-$env{ID_SERIAL}-nst"
>>>>
>>>> line.
>>>>
>>>> Why would I do something this convoluted?  Because I don't like the

>>>> risk of having to perform a
>>>> 25 hour jukebox inventory just because a tape drive failed and 
>>>> suddenly I have to re-define the jukebox because EMC's code messed 
>>>> up and can't figure out what happened to that tape drive that broke

>>>> and was removed.
>>>>
>>>> Frank
>>>>
>>>> On 11/5/10 8:43 AM, Riku Valli wrote:
>>>>> Matthew Huff wrote:
>>>>>> I'm not sure you understand persistent binding. With persistent 
>>>>>> binding,
>>>>
>>>> no changes
>>>>
>>>>>> (including SCSI bus resets) will change the names (and therefore 
>>>>>> the
>>>>
>>>> drive order).
>>>>
>>>>> Yes, tape drives uses serialnumbers for devicenames at RHEL 5 for
>>>>
>>>> persistent bindings. So them
>>>>
>>>>> cannot change.
>>>>> /dev/tape/by-id/scsi-3500308c09fb49000-nst
>>>>>
>>>>> When you made jukebox, you should select use persistent names and 
>>>>> NMC:s
>>>>
>>>> graphical gui uses
>>>>
>>>>> automatic this by-id name convention or manual with command 
>>>>> jbconfig -p,
>>>>
>>>> but jbconfig is'nt
>>>>
>>>>> recommed to use at newer Networker versions.
>>>>
>>>> To sign off this list, send email to listserv AT listserv.temple DOT edu 
>>>> and type "signoff networker" in the body of the email. Please write

>>>> to networker-request AT listserv.temple DOT edu if you have any problems 
>>>> with this list. You can access the archives at 
>>>> http://listserv.temple.edu/archives/networker.html or via RSS at 
>>>> http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
>>>
>>> To sign off this list, send email to listserv AT listserv.temple DOT edu 
>>> and type "signoff networker" in the body of the email. Please write 
>>> to networker-request AT listserv.temple DOT edu if you have any problems 
>>> with this list. You can access the archives at 
>>> http://listserv.temple.edu/archives/networker.html or via RSS at 
>>> http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
> 
> 

-- 
Frank Swasey                    | http://www.uvm.edu/~fcs
Sr Systems Administrator        | Always remember: You are UNIQUE,
University of Vermont           |    just like everyone else.
  "I am not young enough to know everything." - Oscar Wilde (1854-1900)

To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or via RSS at
http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
------------------------------------------
The contents of this message, together with any attachments, are
intended only for the use of the person(s) to which they are
addressed and may contain confidential and/or privileged
information. Further, any medical information herein is
confidential and protected by law. It is unlawful for unauthorized
persons to use, review, copy, disclose, or disseminate confidential
medical information. If you are not the intended recipient,
immediately advise the sender and delete this message and any
attachments. Any distribution, or copying of this message, or any
attachment, is prohibited.

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER