[ADSM-L] Problems archiving Linux symbolic links on NFS

CentOS (RH lite) 5 with TSM 7.1.4.4 client

User called about an archive that keeps aborting/failing with no real
details.  Sometimes the archive will run for a while and sometimes fail
pretty quickly.

04/28/2016 17:39:50 ANS4023E Error processing '/home1': file input/output
error
04/28/2016 17:41:30 ANS1999E Archive processing of '/home1/ngandhi/*'
stopped.

04/28/2016 17:41:30 ANS4023E Error processing '/home1': file input/output
error
04/28/2016 17:56:36 ANS2820E An interrupt has occurred. The current
operation will end and the
client will shut down.
04/28/2016 17:56:44 ANS0361I DIAG: linux/psunxthr.cpp  ( 207): unknown
thread, fatal error, signal 11
04/28/2016 17:57:22 ANS2820E An interrupt has occurred. The current
operation will end and the
client will shut down.
04/28/2016 17:57:30 ANS0361I DIAG: linux/psunxthr.cpp  ( 207): Unknown
thread, fatal error, signal 11
04/28/2016 17:58:36 ANS1999E Archive processing of '/home1/ngandhi/*'
stopped.

04/28/2016 17:58:36 ANS4023E Error processing '/home1': file input/output
error

Turned on tracing and after lots of different Googling, I guessed it has
something to do with symbolic links (this users has lots of them) and the
filesystem is NFS.  The user confirmed this.  They also did all kinds of
verification that there aren't any problems with the symbolic links or the
filesystem.  They can't use archsyml=no since these are end-of-semester
archiving and if needed, the users would have to have it back the way they
left it.

Problem is, all hits that describe this issue are from 6.x days and say
they are resolved.

04/29/2016 11:34:40.318 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home1/ngandhi/AT/at_tetra.qsu
b.69111/all_tetrasacc_m203/gold_all_tetrasacc_m203.mol2.gz
04/29/2016 11:34:40.318 : unxfilio.cpp        (2796): fioGetAttrib: type:
1111 size: 3148 for /home1/ngandhi/AT/
at_tetra.qsub.69111/all_tetrasacc_m203/gold_all_tetrasacc_m203.mol2.gz
04/29/2016 11:34:40.318 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home1/ngandhi/AT/at_tetra.qsu
b.69111/all_tetrasacc_m203/ranked_all_tetrasacc_m203_1.mol2
04/29/2016 11:34:40.453 : unxfilio.cpp        (2348): fioGetAttrib: stat
error for /home1/ngandhi/AT/at_tetra.qs
ub.69111/all_tetrasacc_m203/ranked_all_tetrasacc_m203_1.mol2, errno = 5
04/29/2016 11:34:40.453 : linux/pserrno.cpp   ( 309): TransErrno: Received
error from stat, errno = 5, rc = 164
04/29/2016 11:34:40.453 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /
04/29/2016 11:34:40.453 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /proc
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /proc
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /sys
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /sys
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /tmp
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /tmp
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /usr/global
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /usr/global
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /home
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /export/global
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /export/global
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /export/home
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /export/home
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /proc/sys/fs/binfmt_misc
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /proc/sys/fs/binfmt_misc
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /ipathfs
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /ipathfs
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /var/lib/nfs/rpc_pipefs
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /var/lib/nfs/rpc_pipefs
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /proc/fs/nfsd
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /proc/fs/nfsd
04/29/2016 11:34:40.454 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /home1
04/29/2016 11:34:40.456 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home1
04/29/2016 11:34:40.456 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /home2
04/29/2016 11:34:40.456 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home2
04/29/2016 11:34:40.457 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /home3
04/29/2016 11:34:40.457 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home3
04/29/2016 11:34:40.457 : virtstat.cpp        ( 622): dsmNFSsyscall statfs
for path /home1
04/29/2016 11:34:40.458 : virtstat.cpp        ( 622): dsmNFSsyscall stat
for path /home1

Another odd thing in the trace (above) is when the failure occurs, the
client seems to randomly start scanning all kinds of filesystems, not just
the ones it is archiving.

Thoughts or suggestions ?

--
*Zoltan Forray*
TSM Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator (in training)
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://infosecurity.vcu.edu/phishing.html