[ADSM-L] Celerra NDMP backup failure

After years (at least 5) of working with no issues our Celerra NDMP backups 
suddenly started failing.  No known changes at the time the problem started.

IBM says the Celerra is reporting a media error.  EMC says the Celerra is 
timing out.

TSM on AIX, 6.2.3.0
NDMP backups are LANFree.

Original config:
Celerra to NDMP pool of format "EMC Celerra Dump"  Last change to the pool was 
2 years ago.
Device class, type NAS.  Target tape drives 3592-E05.

Current config: (EMC wanted to try a DD target)
The target device is now an LTO-1 in a DD990.

A week after the backups started failing we upgraded the code on the SAN 
switches, a pre-planned changed.

The week after that the Celerra code was upgraded, a pre-planned change.

The only backups that complete run under 120 minutes.  We cannot find any 
setting that would control this.

The backups that fail, fail in about 60-120 minutes.

We have tried and looked at many things, but obviously the switch to make this 
work is still off and hidden.

Any suggestions on where to look will be appreciated. 


tsm: TSMSERVER>q act s=15139 begint=11:30

Date/Time            Message
-------------------- ----------------------------------------------------------
06/23/2014 13:19:42  ANR0984I Process 15139 for BACKUP NAS (FULL) started in
                      the BACKGROUND at 13:19:42. (SESSION: 52993, PROCESS:
                      15139)
06/23/2014 13:19:42  ANR1063I Full backup of NAS node NASNODE1, file system
                      /root_vdm_4/NAS01_1, started as process 15139 by
                      administrator ADMIN. (SESSION: 52993, PROCESS: 15139)
06/23/2014 13:19:42  ANR8337I NAS volume DD0008 mounted in drive TSMSERVER_N_501
                      (c80t0l0). (SESSION: 52993, PROCESS: 15139)
06/23/2014 13:19:42  ANR1340I Scratch volume DD0008 is now defined in storage
                      pool NAS_V_TAPE_02. (SESSION: 52993, PROCESS: 15139)
06/23/2014 13:19:42  ANR0513I Process 15139 opened output volume DD0008.
                      (SESSION: 52993, PROCESS: 15139)
06/23/2014 14:17:44  ANR0515I Process 15139 closed volume DD0008. (SESSION:
                      52993, PROCESS: 15139)
06/23/2014 14:17:44  ANR8336I Verifying label of NAS volume DD0008 in drive
                      TSMSERVER_N_501 (c80t0l0). (SESSION: 52993, PROCESS: 
15139)
06/23/2014 14:17:45  ANR8468I NAS volume DD0008 dismounted from drive
                      TSMSERVER_N_501 (c80t0l0) in library VLIBTSMSERVER_N. 
(SESSION:
                      52993, PROCESS: 15139)
06/23/2014 14:17:45  ANR1104E NAS Backup process 15139 terminated - NDMP
                      session errors encountered. (SESSION: 52993, PROCESS:
                      15139)
06/23/2014 14:17:45  ANR0988I Process 15139 for BACKUP NAS (FULL) running in
                      the BACKGROUND processed 140,172,591,104 bytes with a
                      completion state of FAILURE at 14:17:45. (SESSION: 52993,
                      PROCESS: 15139)
06/23/2014 14:17:45  ANR1893E Process 15139 for BACKUP NAS (FULL)  completed
                      with a completion state of FAILURE. (SESSION: 52993,
                      PROCESS: 15139)






Andy Huebner