IBMtape Stopped Working

[GregE]

Solaris 10, IBMtape 4.1.9.5

I installed IBMtape on a test server and it was working fine. Two weeks later (yesterday) I went back to try a configuration for alternate pathing (just made additions to lpfc.com, devlink.tab, and IBMtape.conf) and it didn't work, so I simply removed my additions, rebooted and was back the way it was.......except that IBMtape doesn't work anymore. I've compared everything to another server to be sure I put things back the way they were.

Running this...
add_drv -m '* 0666 bin bin' IBMtape

...Results in this...
devfsadm: driver failed to attach: IBMtape
Warning: Driver (IBMtape) successfully added to system but failed to attach

Tracking down some info in /var/adm/messages. It needs directories that don't exist but should...
/dev/ibmcontrol
/dev/ibmdaemon

So I uninstalled the IBMtape driver and reinstalled, without success.

Anyone have any ideas as to why this would disappear? And how to get it back?

 

[Eldoraan]

silly question, but sometimes they help... that host still zoned to the tapes?

 

[Mikey D]

Yeah failed to attach usually means it can't find the target for whatever reason, could be zoning issues or something other funky going on with the wwpn's.

Might want to go in to /usr/sbin/lpfc/lputil and check the persistent bindings for each adapater and see if any targets are missing.

I have seen sometimes where the lpfc adapaters will reorder themselves causing issues like this. Usually only happens after some pretty specific circumstances though.

 

[GregE]

The mappings in lputil are there. The drives are also still zoned properly. But with /dev/ibmcontrol and /dev/ibmdaemon missing. I'm thinking the "failed to attach" in this case may mean failed to attach to those logical device mappings, not physical. Just a guess.

BUT......Are those created at boot, or created at installation of IBMtape? I've reinstalled IBMtape so contrary to my above guess, perhaps those get created only after physical contact is made?

 

[Mikey D]

I think I may know what's up.

Make sure these two entries are in your /usr/kernel/drv/IBMtape.conf and then reconfig reboot.

name="IBMtape" parent="pseudo" instance=16382;
name="IBMtape" parent="pseudo" instance=16383;


I checked those paths and they come back to those lines missing.

$ ls -ltra /dev/ibmcontrol
lrwxrwxrwx 1 root other 42 Feb 17 15:24 /dev/ibmcontrol -> ../devices/pseudo/IBMtape@16383:ibmcontrol
$ ls -ltra /dev/ibmdaemon
lrwxrwxrwx 1 root other 41 Feb 17 15:24 /dev/ibmdaemon -> ../devices/pseudo/IBMtape@16382:ibmdaemon

 

[GregE]

I knew about that and should have included it my email. I'd looked at the paths on other servers that I've setup to see what should be the path that's missing. Also, those lines are in the IBMtape.conf and have always been there. And that file is owned by root. This is strange.

 

[Mikey D]

You may have already but make sure you reboot after installing or reinstalling IBMtape

What's the lpfc.conf, IBMtape.conf, and devlink.tab file all look like now?

 

[GregE]

It's been rebooted.

.../etc/devlink.tab...(with tabs between right-most ';' and rmt)
type=ddi_pseudo;name=IBMtape \M0
# /dev/rmt/smc (medium changer, ie. tape library) #
type=ddi_byte:tape;addr=1,1; rmt/0\M0
# /dev/rmt/st (tape drives) #
type=ddi_byte:tape;addr=1,0; rmt/1\M0
type=ddi_byte:tape;addr=2,0; rmt/2\M0
type=ddi_byte:tape;addr=3,0; rmt/3\M0
type=ddi_byte:tape;addr=4,0; rmt/4\M0
type=ddi_byte:tape;addr=5,0; rmt/5\M0
type=ddi_byte:tape;addr=7,0; rmt/7\M0
type=ddi_byte:tape;addr=8,0; rmt/8\M0
type=ddi_byte:tape;addr=9,0; rmt/9\M0
type=ddi_byte:tape;addr=a,0; rmt/10\M0
type=ddi_byte:tape;addr=b,0; rmt/11\M0
...
.../opt/IBMtape/IBMtape.conf (symbolically linked to /usr/kernel/drv/IBMtape.conf)...
name="IBMtape" parent="lpfc" target=1 lun=0;
name="IBMtape" parent="lpfc" target=1 lun=1;

name="IBMtape" parent="lpfc" target=2 lun=0;
name="IBMtape" parent="lpfc" target=3 lun=0;
name="IBMtape" parent="lpfc" target=4 lun=0;
name="IBMtape" parent="lpfc" target=5 lun=0;
name="IBMtape" parent="lpfc" target=7 lun=0;
name="IBMtape" parent="lpfc" target=8 lun=0;
name="IBMtape" parent="lpfc" target=9 lun=0;
name="IBMtape" parent="lpfc" target=10 lun=0;
name="IBMtape" parent="lpfc" target=11 lun=0;
name="IBMtape" parent="pseudo" instance=16382;
name="IBMtape" parent="pseudo" instance=16383;
name="IBMtape" class="scsi" target=1 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=1 lun=1 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=2 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=3 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=4 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=5 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=7 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=8 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=9 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=10 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
name="IBMtape" class="scsi" target=11 lun=0 block_size=0 buffering=1 immediate=0 trailer=0 sili=0;
...
.../kernel/drv/lpfc.conf....
fcp-bind-method=2;

# BEGIN: LPUTIL-managed Persistent Bindings
fcp-bind-WWPN="500601693060359f:lpfc0t0",
"500601613060359f:lpfc0t1",
"50050763004c6c11:lpfc1t1",
"50050763004c6c12:lpfc1t2",
"50050763004c6c13:lpfc1t3",
"50050763004c6c14:lpfc1t4",
"50050763004c6c15:lpfc1t5",
"50050763004c6c17:lpfc1t7",
"50050763004c6c18:lpfc1t8",
"50050763004c6c19:lpfc1t9",
"50050763004c6c1a:lpfc1t10",
"50050763004c6c1b:lpfc1t11";

automap=0;

 

[GregE]

$ ls -l devices/pci@8,700000/lpfc@4 is empty. This is where /dev/rmt would link to.

 

[Mikey D]

Do you have the Emulex FCA utilities installed?

If so run /opt/EMLXemlxu/bin/emlxdrv and verity that the lpfc driver is still being used for the HBA cards and that it somehow didn't switch over to the leadville drivers. Should show lpfc as the driver for Emulex branded cards. If you don't have the FCA utilities installed you can get them here.

http://www.emulex.com/downloads/emulex/cnas-and-hbas/drivers/solaris/630g.html

You'll have to look at the links on the right for the specific LPFC driver you're using, which btw what version of LPFC is installed? (pkginfo -l lpfc)

EMLXDRV Driver Management Utility, Version 1.00k
COPYRIGHT © 2004-2006 Emulex. All rights reserved.

Driver Alias Present Boot Sun Models
-------------------------------------------------------------------------------
lpfc lpfs no no no LP8000S and LP9002S (SBUS)
lpfc f800 no no no LP8000 and LP8000DC
lpfc f900 no no no LP9002, LP9002C, LP9002DC, and LP9402DC
lpfc f980 no no no LP9802 and LP9802DC
lpfc fa00 no no no LP10000, LP10000DC and LP10000ExDC
lpfc fd00 no no no LP11000 and LP11002
lpfc fe00 yes no no LPe11000 and LPe11002
emlxs f0a5 no no no 2G Blade Adapter (emlxs only)
emlxs fc00 no no yes LP10000-S and LP10000DC-S (emlxs only)
emlxs fc10 no no yes LP11000-S and LP11002-S (emlxs only)
emlxs fc20 no no yes LPe11000-S and LPe11002-S (emlxs only)

 

[GregE]

LP10000-E EMC 2Gb 133Mhz PCI-X Single Emulex Lightpulse
Version is 6.02f. I know that's pretty old, and is the installed version on all of our servers, including the TSM server. And this was working fine until I tried to configure the .conf files for alternate pathing. Then took everything out and set it back the way it was, but it's never worked since before that.

Driver Alias Present Boot Sun Models
-------------------------------------------------------------------------------
lpfc lpfs no no no LP8000S and LP9002S (SBUS)
lpfc f800 no no no LP8000 and LP8000DC
lpfc f900 no no no LP9002, LP9002C, LP9002DC, and LP9402DC
lpfc f980 no no no LP9802 and LP9802DC
lpfc fa00 yes no no LP10000, LP10000DC and LP10000ExDC
lpfc fd00 no no no LP11000 and LP11002
lpfc fe00 no no no LPe11000 and LPe11002
emlxs f0a5 no no no 2G Blade Adapter (emlxs only)
emlxs fc00 no no yes LP10000-S and LP10000DC-S
emlxs fc20 no no yes LPe11000-S and LPe11002-S

 

[Mikey D]

Greg was wondering if you made it any further on this problem? I'd be curious to hear the solution if you do find it out.

Probably and IBM PMR needed at this point.

 

[GregE]

Thanks. No, I've pulled in the UNIX admins to look at it and we'll then call SUN as that likely is going to be the direction. I called IBM, mentioned IBMtape driver, and that threw them all for a loop. Since it's software I chose the option to go to software. It's not on their "list" so they sent me to hardware, who then wanted serial numbers of hardware. They've never called back after issuing a PMR and we decided that this is going to end up at SUN anyway because what's happening is devices are not being created. This is really puzzling. I'll definitely post the findings.

 

[GregE]

For anyone interested, it turned out to be what I was afraid (and glad) it would be..............SOMETHING STUPIDLY SIMPLE.

Simple, yet I'm not sure why not documented. In every manual it says IBMtape.conf should be in /usr/kernel/drv. We had one in /kernel/drv, and it was an empty file, and I figured it wasn't paying attention to it there anyway. WRONG! How did it get there? I don't know. It had a date on it from 8 months ago, WELL BEFORE I ever began to work on this tape project.

So, all IBM documentation says IBMtape.conf file should be in /usr/kernel/drv, but I've not seen anywhere where it specifies that it will look in /kernel/drv first and use that one, but that is exactly what happened.

(IBM was of no help on IBMtape driver. Support didn't even know how to route my call and I never have talked to anyone at IBM with specific knowledge of the tape driver itself. I had a couple of people try to help me, and I definitely appreciated that, but the support channel itself had no clue of how to handle my call.)

 

[Mikey D]

Ha good deal at least it was an easy fix or else it probably would have taken IBM and SUN a lot longer to figure out.

I really which they'd just make one location for IBMtape.conf instead of the two they even have now.

 

[GregE]

I really which they'd just make one location for IBMtape.conf instead of the two they even have now.

Yea, or call one /opt/IBMtape/IBMtape.conf.smp or something since it's only the reference version. I like to disregard the /opt/IBMtape one, and create a symbolic link from /kernel/drv to /opt/IBMtape and work it there. Just keeps it a bit organized for me.

Sun checked it thru the OS, but once they saw that it wasn't an OS issue, they're hands are out of it because its an EMC branded HBA. Same one they ship with, but since it came from a VAR, they won't touch it.