I received a lot of suggestions, thanks.
So far we have:
- checked log files for errors
- brought the master down (e450) to do a probe-scsi
- checked st.conf, sg.conf
- deleted all drives, storage unit, robot, and readded

The system now sees all the drives correctly, but the problem is still there, 
the drives are being marked down.  We are able to go to disk successfully.

We think it's a NBU problem at this point.  We are running 3.4.1 and very 
tempted to throw the 4.5 upgrade on to see what happens, or reinstall and 
go back to 3.4.  I think either will probably introduce other problems and
may not help with the existing problem, but I'm open to suggestions.

Warning - long log follows.


Someone asked for bptm log info.
In /opt/openv/netbackup/db/error I get:
grep bptm log_1021867200
1021903311 1 132 16 backup02 111846 0 0 wapcuwg01bb0009 bptm media manager 
terminated during mount of media id 000010, possible media mount timeout
1021903313 1 4 16 backup02 111846 0 0 wapcuwg01bb0009 bptm media manager 
terminated by parent process
1021903403 1 130 4 backup02 0 0 0 *NULL* bptm media id 000010 removed from 
media manager database (expired)
1021915572 1 130 4 backup02 0 0 0 *NULL* bptm media id 000186 removed from 
media manager database (expired)
1021915625 1 130 4 backup02 0 0 0 *NULL* bptm media id 000081 removed from 
media manager database (expired)
1021915630 1 130 4 backup02 0 0 0 *NULL* bptm media id 000140 removed from 
media manager database (expired)
1021915783 1 130 4 backup02 0 0 0 *NULL* bptm media id 000161 removed from 
media manager database (expired)
1021915989 1 4 4 backup02 113073 113073 0 backup02 bptm begin writing backup id 
backup02_1021915676, copy 1, fragment 1, to media id 000181 on drive index 4
1021916498 1 4 4 backup02 113073 113073 0 backup02 bptm successfully wrote 
backup id backup02_1021915676, copy 1, fragment 1, 726464 Kbytes at 1436.901 
Kbytes/sec
1021916529 1 132 8 backup02 113074 113074 0 backup02 bptm media id 000140 is in 
a DOWN drive, misplaced, write protected or unmountable; attempting retry with 
a different media id
1021916705 1 132 16 backup02 113074 113074 0 backup02 bptm attempted to use new 
media (000161), but the media id found (000145) is already in use, FREEZING 
000161
1021916713 1 132 16 backup02 113076 0 0 wapcuwg01aa0008 bptm incorrect media 
found in drive index 3, expected 000153, found 000161, FREEZING 000153
1021916741 1 132 16 backup02 113075 0 0 wapcuwg01bb0005 bptm read error on 
media id 000130, drive index 5, reading header block, I/O error
1021916743 1 4 4 backup02 113073 113073 0 backup02 bptm begin writing backup id 
backup02_1021915676, copy 1, fragment 2, to media id 000183 on drive index 6
1021916891 1 132 16 backup02 113076 0 0 wapcuwg01aa0008 bptm incorrect media 
found in drive index 3, expected 000145, found 000148, FREEZING 000145
1021916913 1 132 16 backup02 113074 113074 0 backup02 bptm attempted to use new 
media (000081), but the media id found (000153) is already in use, FREEZING 
000081
1021916983 1 130 4 backup02 0 0 0 *NULL* bptm media id 000140 removed from 
media manager database (expired)
1021917080 1 132 16 backup02 113075 0 0 wapcuwg01bb0005 bptm read error on 
media id 000130, drive index 7, reading header block, I/O error
1021917411 1 132 16 backup02 113075 0 0 wapcuwg01bb0005 bptm read error on 
media id 000130, drive index 8, reading header block, I/O error
1021917412 1 132 8 backup02 113075 0 0 wapcuwg01bb0005 bptm FREEZING media id 
000130, it has had at least 3 errors in the last 12 hour(s)
1021917460 1 132 16 backup02 113074 113074 0 backup02 bptm media manager 
terminated during mount of media id 000019, possible media mount timeout
1021917462 1 4 16 backup02 113074 113074 0 backup02 bptm media manager 
terminated by parent process
1021917468 1 132 16 backup02 113075 0 0 wapcuwg01bb0005 bptm media manager 
terminated during mount of media id 000143, possible media mount timeout
1021917468 1 132 16 backup02 113076 0 0 wapcuwg01aa0008 bptm media manager 
terminated during mount of media id 000160, possible media mount timeout
1021917470 1 4 16 backup02 113075 0 0 wapcuwg01bb0005 bptm media manager 
terminated by parent process
1021917470 1 4 16 backup02 113076 0 0 wapcuwg01aa0008 bptm media manager 
terminated by parent process
1021917583 1 130 4 backup02 0 0 0 *NULL* bptm media id 000019 removed from 
media manager database (expired)
1021918400 1 132 8 backup02 113077 0 0 wapcuwg01aa0008 bptm cannot locate on 
drive index 4, locate scsi command failed, key = 0x3, asc = 0x11, ascq = 0x0
1021918691 1 4 16 backup02 113073 113073 0 backup02 bptm media manager 
terminated by parent process