Hot Diggety! Murthy V Gongala was rumored to have written:
>
> Level indicated by  "lslpp -l bos.rte.libc" - 5.1.0.25

That sounds like 5.1 ML02 -- latest major patch set collection, very good.

> TSM shows the volumes as  - Private.

Ouch. Ok, the basic problem you're running in is that the TSM server
*wants* at least one more scratch tape, but all the tapes you have
right now is marked as Private.

I think you haven't ruled out a tape communication problem yet, because
TSM marks all unreadable or inaccessible tapes as Private. Most likely
they are in the Private state because they already had existing backups,
but I can't rule out a tape drive communication issue of some sort.

Not running a FC-AL setup rules out a FC-AL switch failure issue, and
since you say the setup's worked fine daily until now AND the 'lsdev -Cc
tape' output shows all devices as 'Available' state, unlikely to be a
drive communication issue, I think.

I'm far from a TSM expert (many folks here are!) but I'm guessing
you'd want to "compact" the data on the existing tapes to create enough
free space for at least one or more scratch tape.

May want to check tape utilization of each volume with:

tsm> q vol

Are they at 100% of about 190 GB per tape? Or are there some tapes that
have like 50%, 60%, something below 100%?

You may need to look into several things:

1. 'expire inventory' TSM command that will mark expired files as
being 'free' (reclaimable space). Run it daily after all of your backups
has completed - can be done via a server-scheduled command that you enter
only once.

Then do another 'q vol' to see if that helped clear up more free space,
enough for a 'move data' command to work.

2. look to keep a tight control on retention of data through adjusting
management classes, policy domains, storage pools, etc...  or by limiting
what data you back up through inclexcl include/excludes in client's dsm.opt
file.

3. do you have an "off-site copygroup"? (It can be an on-site copygroup
or off-site copygroup or whatever you want to call it, but it has to be
something in addition to your primary tape storage pool)

If you do, then you will want to make a copy of your existing tape data
into the offsite copygroup regularly -- once a week or whatever frequency
works for you. But you need to do it regularly because it requires scratch
tapes to do that, so you can't do that right now without clearing up some
room first.

Other folks will probably have better suggestions on how to get you out
of this dug hole. My comments may help more with maintaining the setup
to avoid this in future, once you're back to normal. I'm afraid I'm not
experienced enough to give you the best answer for how to recover, but
we've got a lot of good folks on this list who can give pointers.

-Dan