Error log filling up on datamover quiescing errors

[sittnick]

We have a TSM server that handles VMs exclusively. There is the TSM server, TSM1, and another server for 15 Datamovers, LIN3.

Everything has been stable for some time, but starting last week, quiescing errors (115) have started to fill up the error log of different data movers at an alarming rate. Yesterday evening, during backups, it got stuck on db169/disk 4 and filled up over 200GB. (other times the problem server was different as well as the datamover) I wound up having to delete the error log after copying a portion of it. Eventually I had to reboot the datamover box to make the memory available to the OS.

My co-worker tried adding the ERRORLOGMAX statement to the dsm.sys file for the datamover that was acting up. (I don't have write rights to that particular server) She told me it returned an error and wouldn't accept it.

The first time this happened I put in a call to IBM, and I got an e-mail from a first level fellow who requested an INSANE amount of traces, config files, etc. much of which was clearly irrelevant.

We are on TSM 7.1.1 on Red Hat Linux Enterprise 6.7.

Following is the repeated snippet in the error log from LIN3 (datamover 0, file name dsmerror0.log)(server names changed for clarity):

TSM return code : 115
TSM file : vmbackvcm.cpp (218)
10/05/2015 22:23:29 ANS5250E An unexpected error was encountered.
TSM function name : vcmLogger::trace
TSM function : CacheManager::swap_out: Can't write object (id='Path: '/tmp/tsmvmbackup/fullvm/CDF_Local/DB169/Hard Disk 4' Job: 43004 MB Number: 1431')

TSM return code : 115
TSM file : vmbackvcm.cpp (218)
10/05/2015 22:23:29 ANS5250E An unexpected error was encountered.
TSM function name : vcmLogger::trace
TSM function : CacheManager::swap_out: Can't write object (id='Path: '/tmp/tsmvmbackup/fullvm/CDF_Local/DB169/Hard Disk 4' Job: 43004 MB Number: 1432')

 

[marclant]

I found this APAR for a similar issue with Fastback: http://www-01.ibm.com/support/docview.wss?uid=swg1IC78188

Could be something similar with VE, might want to open a problem ticket with IBM.

 

[Inuitten]

Did you find out what happened? I have similar errors filling 1 of my datamover logs.

 

[sittnick]

Did you find out what happened? I have similar errors filling 1 of my datamover logs.

The problem has not repeated, and I have been unable to determine what happened. The ticket I opened with IBM was a awaste of time. IBM support told me that I had the errors because the partition wasn't large enough. They ignored the fact that the partition filled because of the error, no matter how many times I tried to spell it out for him. It is driving us to consider another solution.

I get the feeling that TSM is in trouble. It is a bad sign when a company tries to freshen the image of a product by changing the name for a 7.1.3 release. Spectrum Protect sounds like a name that a deadlocked commitee took as a compromise.

 

[Inuitten]

The error indicates that it tries to write the entire disk to cache. And since there obviously is not room for it it throws the error filling the log. I have seen in twice now.
We are looking into Veeam. the 2 TB disk limit and Vcloud implementation in TSM(or Tivoli Spectrum protect) is just not good enough.