Cannot make connections to TSM server once backups start

[rowl]

I ran a test on 6.3.1.0 today (SLES 11 SP1). Two of us connected to the TSM server with dsmadmc just before the backups started. Everything looked good on the server side, but we soon found that no more admin sessions could connect. Then found that no more clients could connect either. Of the 160 clients scheduled to backup, only 96 managed to get a connection. Maxsessions is set to 1,000.

ping server would not respond, and I could not use command redirection from another instance either. From the two consoles that we had open we could still run commands and monitor activity. My coworkers session timed out and he could not get back in. I ran a can se all to see if that would make any difference. Once all sessions were gone except my admin connection others tried to connect and could not get in. Eventually my session timed out and we couldn't log into the server at all. Ended up having to kill dsmserv process.

We are using a Data Domain array with nfs for the storage pool. Do we look at TSM, Linux tuning, or could this be an nfs related issue?

Any ideas?

Thanks,
-Rowl

[moon-buddy]

Have you managed to bring up dsmserv again?

If you can't then it maybe an OS issue - try a reboot.

If you try bringing up dsmserv at the console level, what errors do you see? How is NFS used? Is NFS used for everything include DB and LOG volumes? If it is, check NFS mounts and Data Domain.

[rowl]

I was able to restart the server. NFS is only used for the storage pool, the DB and Logs are on an EMC VMAX array.

I did find one error int he /var/log/messages file that is interesting.

Description:
TOO MANY OPEN FILES

Associated information:
Sqlcode -958
SQL0958C The maximum number of open files has been reached.

I see from sysctl -a fs.file-max = 7312131 so I don't expect that is that is the setting I need to increase. Going to keep digging......

[moon-buddy]

I believe you need to tweak the max open file handler which has memory dependencies.