[Networker] Problems cloning on a storage node after adding DDS license

I've had case 3077121 open for 36 hours, just wondering if anyone else has
any more helpful tips.

Until this weekend, we had all of our clients backing themselves up to our
main networker server (Lazarus) and tape-to-tape cloning happening on a
remote storage node (Noah). All of the cloning was able to happen
off-network without shuttling tapes because Noah has fiber channel
connectivity to Lazarus's jukebox.

Then we added 2 DDS licenses and 1 NDMP license, and it stopped working.
Now the tapes are no longer loaded into the drive on Noah as desired. They
are loaded into Lazarus and the data flows, more slowly, over the Ethernet
link between the two systems.

To illustrate:

[Main jukebox on Lazarus shared via fiber channel, 2 DDS licenses]
  drive 1 (/dev/rmt/0cbn) slot   :
          (rd=noah:/dev/rmt/2cbn) slot   :
          (rd=pod.unet.brandeis.edu:nrst7a) slot   :
          (rd=dop.unet.brandeis.edu:nrst0a) slot   :
  drive 2 (/dev/rmt/1cbn) slot   :
          (rd=noah:/dev/rmt/3cbn) slot   :
          (rd=pod.unet.brandeis.edu:nrst4a) slot   :
          (rd=dop.unet.brandeis.edu:nrst5a) slot   :
  drive 3 (/dev/rmt/2cbn) slot   :
  drive 4 (/dev/rmt/3cbn) slot   :
  drive 5 (/dev/rmt/4cbn) slot   :
  drive 6 (/dev/rmt/5cbn) slot   :

In addition to those 6 fiber channel-attached drives, the remote Noah
storage node also has its own little SCSI-attached jukebox:

[Stub jukebox SCSI-attached to noah only]
  drive 1 (rd=noah:/dev/rmt/1cbn) slot   :
  drive 2 (rd=noah:/dev/rmt/0cbn) slot   :

Until the addition of the second NDMP license and the two DDS licenses
last Saturday, our configuration for the main jukebox looked like this:

  drive 1 (rd=pod.unet.brandeis.edu:nrst7a) slot   :
  drive 2 (rd=noah:/dev/rmt/3cbn) slot   :
  drive 3 (/dev/rmt/2cbn) slot   :
  drive 4 (/dev/rmt/3cbn) slot   :
  drive 5 (/dev/rmt/4cbn) slot   :
  drive 6 (/dev/rmt/5cbn) slot   :

Under the above configuration, with our Default pool configured to use any
of the 5 drives noah:3cbn or lazarus:3cbn-5cbn and clients configured with
"storage node: lazarus" and "clone storage node: noah", we were getting the
preferred behavior: clients would back up to the drives in Lazarus, then
we would eject all the tapes, then we would run an nsrclone -S -f on the
list of recent Default pool ssids and the tapes would be loaded into and
read from noah:/dev/rmt/3cbn only.

Under the current more complex DDS configuration, with out Default pool
configured to use any of the 6 logical drives noah:2cbn-3cbn or
lazarus:2cbn-5cbn and clients configured with "storage node: lazarus" and
"clone storage node: noah", we are seeing the cloning source tapes loaded
into lazarus:2cbn-5cbn, read onto Lazarus, and the data transferred over
the Ethernet network to Noah for writing on Noah's locally attached
0cbn-1cbn.

Performance is less than ideal and our Ethernet network is unnecessarily
loaded.

We would like to return to the previous preferred situation, with all the
cloning action happening on Noah.

I have scrutinized nsr[jb].res and don't believe anything has changed other
than the jukebox config. For all clients, the "storage nodes" attribute is
set to lazarus and the "clone storage nodes" attribute is set to noah.

I've tried setting "access weight" and changing to noah's fully qualified
domain name to no effect.

If I manually load and mount a tape in noah:/dev/rmt/2cbn or
noah:/dev/rmt/3cbn then the data is read fine and all action happens
entirely on noah. However, if I put all the drives owned exclusively by
Lazarus into "Service Mode," so that the only possible choice for mounting
the tapes is the slots on noah, I get this error:

# nsrclone -v -S 3668789250
Automatically copying save sets(s) to other volume(s)

Starting cloning operation...
  ...from storage node: lazarus
nsrclone: error, no matching devices on `lazarus'
nsrclone: Cannot open nsrclone session with lazarus4
nsrclone: Failed to clone any save sets

Why is it insisting on loading the tapes into Lazarus when they're
perfectly welcome on Noah?

Why did this work before Saturday?
--
Rich Graves <rcgraves AT brandeis DOT edu>
UNet Systems Administrator

--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=