SystemState Major Errors

CharlesTSM

ADSM.ORG Member
Joined
Jun 10, 2015
Messages
16
Reaction score
0
Points
0
PREDATAR Control23

Hey,

I've got a repeating issue with a Server 2012 R2 box that refuses to complete a SystemState Backup...

I get the following errors:

06/29/2015 03:29:44 ANS5250E An unexpected error was encountered.
TSM function name : vssCloseGroups
TSM function : baCloseGroupLeader() for file 'xxxxxx\SystemState\NULL\System State\SystemState\TSM\FULL\00000000000000\Top\SystemState\System State\0\SystemState' returned rc=990
TSM return code : 990
TSM file : ..\..\common\winnt\vssback.cpp (2683)
06/29/2015 03:29:44 ANS0361I DIAG: cuEndTxnEnhanced: Out of sequence verb: verb: 13
06/29/2015 03:29:44 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'xxxxx\SystemState\NULL\System State\SystemState\BACKUP_STATE\TSM\00000000000000' returned rc=136
TSM return code : 136
TSM file : ..\..\common\winnt\vssback.cpp (3135)
06/29/2015 03:29:44 cuGetBackQryResp: Out of sequence verb received; EndTxn wanted
06/29/2015 03:29:44 ANS0361I DIAG: cuPing(): Out of sequence verb: verb: 13
06/29/2015 03:29:44 sessOpen: Session state transition error, sessState: sSignedOn.
06/29/2015 03:29:44 sessOpen: Transitioning: sSignedOn state ===> sTRANSERR state
06/29/2015 03:29:44 cuFSUpd: Received rc: 136 from cuBeginTxn
06/29/2015 03:29:44 fsIncrDateUpdate: received error from cuFSUpd
06/29/2015 03:29:49 ANS1026E The session is rejected: There was a communications protocol error.


I found something similar on here that advised me to recreate the adsm.sys folder contents (stopping all services, deleting content and restarting services)....

I performed this and it appeared to work for one backup then started to error again.

I also read that it is a version mismatch from client to server however we have other servers on this version running on 2012 r2 and they work fine...

Any help would be appreciated unsure where to go from here...

Thanks,

Charles
 
PREDATAR Control23

From the TSM Client command line issue backup systemstate.
Does the TSM Client produce the same messages?

From a Windows Command Prompt, issue:
vssadmin list writers > c:\writers.txt
Confirm that all the writers are stable.

Even though all the writers are stable, there still be an issue with the system state.
If the writer is not stable, a system reboot will resolve the issue.
There are times that a system reboot is not a permanent fix.

To verify or deny this we need discount TSM as a possible suspect.
Please take to following step.

NOTE: You must be logged in with a user that is part of the Local Administrators group.
1) Start the ntbackup utility. This can be done by going through the start menu, or by going to run and typing in ntbackup.exe
2) If wizard starts cancel it. Go to the Tools menu, choose Options. Select the Backup Log tab, and make sure that the Detailed item is selected. Click OK
3) From the main window, choose the Backup tab.
4) Select the System Object / System State / System Services from the list of items to backup.
5) Alter the filename in the "Backup Media or Filename" to C:\Backup.bkf
6) Start the backup.
7) There will be a report Button once the backup completes.
8) Close the ntbackup utility and delete the file it created for this test : C:\Backup.bkf

For the Windows challenged - How to Perform a System State Backup
http://www.lwcomputing.com/tips/static/sysstate.asp#how

If we are not able to backup the system state via the NTBackup utility.
Windows 2003 - NTBackup
Windows 2008/2012 - Windows Server Backup
Windows Vista - Windows Backup and Restore Center

Bad news, need to contact MS Support to have the system state checked out.

If all the writers are stable and we are able to backup the system state via the NTBackup utility.
Need to stress the system state to confirm that there are no issue.

Vshadow is a Microsoft utility used to test VSS. This is helpful in determining any issues with VSS outside of Tivoli Storage Manager.

Vshadow can be downloaded here:
Windows 2003 and Windows 2003 R2 VSS SDK
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23490

Windows 7
http://www.microsoft.com/download/en/details.aspx?id=8279

From a Windows Command Prompt, issue the following commands using the vshadow utility (contact Microsoft for a version appropriate for the version of Windows being used):
vshadow –p c: > vshadow_p.out
vshadow –wm2 > vshadow_wm2.out
vshadow -ws > vshadow_ws.out
vshadow -nw c: > vs_nw.out
vshadow -nw -p c: > vs_p.out

For Windows 8/2008 (R2)/2012 (R2) users...
http://www-01.ibm.com/support/docview.wss?uid=swg21699984

If you are able to backup the system state via the NTBackup utility and not via the TSM Client, and don't want to do all the above.
Exclude the system state from the TSM Client.
Backup the system state via the NTBackup utility to a file.
Then use the TSM Client to backup the file.

Good Luck,
Sias
 
  • Like
Reactions: Art
PREDATAR Control23

Sias,

That has been most helpful especially the diskshadow part as I never knew this tool existed, great in-depth analysis, 'sadly' everything comes back looking fine so no help in this situation.

One thing I wanted to run past you was regarding running a manual backup of system state; obviously with NTbackup this was easy however with 2012 R2 you need to install the Server feature "Windows Backup and Restore", does installing this feature have any negative knock on affect with TSM as I'm a bit wary?

I know that in the past installing two different backup software on one machine could cause issues.

Thanks,

C
 
PREDATAR Control23

You can have as many backup application install on the system, but they can not run at the same time.

And if the two backup application have to run at the same time, they should not be backing up the same data.

Good Luck,
Sias
 
PREDATAR Control23

Hey Sias,

So I've ran through the above and have troubleshoot the issue some more.

Windows Backup backups up the system state with no errors whatsoever all works fine, all the VSS reports look good too.

However I still get errors in TSM when it is trying to backup the system state:

07/12/2015 03:26:42 ANS5250E An unexpected error was encountered.
TSM function name : vssCloseGroups
TSM function : baCloseGroupLeader() for file 'xxxxxx\SystemState\NULL\System State\SystemState\TSM\FULL\00000000000000\Top\SystemState\System State\0\SystemState' returned rc=990
TSM return code : 990
TSM file : ..\..\common\winnt\vssback.cpp (2683)
07/12/2015 03:26:42 ANS0361I DIAG: cuEndTxnEnhanced: Out of sequence verb: verb: 13
07/12/2015 03:26:42 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'xxxxxxx\SystemState\NULL\System State\SystemState\BACKUP_STATE\TSM\00000000000000' returned rc=136
TSM return code : 136
TSM file : ..\..\common\winnt\vssback.cpp (3135)


07/13/2015 03:31:32 ANS5250E An unexpected error was encountered.
TSM function name : vssCloseGroups
TSM function : baCloseGroupLeader() for file 'xxxxxxx\SystemState\NULL\System State\SystemState\TSM\FULL\00000000000000\Top\SystemState\System State\0\SystemState' returned rc=2
TSM return code : 2
TSM file : ..\..\common\winnt\vssback.cpp (2683)


Anything else I can try?

Thanks,

C
 
PREDATAR Control23

Hi,

Your problem with the systemstate backup is not due to VSS, but rather a communication problem between the client and the server. You could just easily get similar problems during the backup of files. In the part of the backup you are getting the failure, the VSS snapshot has already been taken and TSM is reading from it. The actual action that the TSM Client was doing is grouping all the systemstate objects for this particular backup as one restore point. This group includes all the new/modified system state objects backed up today and all the previously active objects backed up.

Here's the details:

07/12/2015 03:26:42 ANS0361I DIAG: cuEndTxnEnhanced: Out of sequence verb: verb: 13
07/12/2015 03:26:42 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'xxxxxxx\SystemState\NULL\System State\SystemState\BACKUP_STATE\TSM\00000000000000' returned rc=136
TSM return code : 136

When the TSM Client and TSM Server talk to each other, they exchange verbs and there is a normal sequence, login, acknowledge login, open session, prepare to send data, send data, close session. My example is simple, but the point is that for example, the client can't do a send data verb before letting the server know first that it is about to send data. So if a few packets are dropped, and a verb doesn't make it from the client to the server or makes it later, when the server receives the next verb, it will report back to the client that the verb is out of sequence, because it's not the expected order.

RC=136 is a TSM return code, it means:
0136 E
DSM_RC_COMM_PROTOCOL_ERROR The session is rejected: There was a communications protocol error.

Explanation
An unexpected network message was received by the client. This could be caused by network problems or a programming error.

System action
TSM canceled the current operation.

User response
Verify that your communication path is functioning properly and try the operation again. If the problem persists, contact your TSM administrator for further assistance.

Parent topic: API return codes
source: http://www-01.ibm.com/support/knowl.../com.ibm.itsm.msgs.client.doc/msg-0136_E.html
 
PREDATAR Control23

Hi Marclant

That's really interesting many thanks for replying.

The information you posted is actually ringing alarms as we do have a known networking issue on the TSM Server, however it has only ever affected the ability for TSM Server to Login to itself (TCP Stack issue). We have a few hundred backups running and this problem is only affecting this one machine so little bit odd but we think we have a fix for the networking issue so I'll roll that out and see if this problem disappears.

Thanks again, C
 
Top