64 bit Windows 2008 client backup issue

[BrianP]

All,
We are having an issue backing up a MS Windows 2008 64-bit server. We contacted IBM and they pointed to a Microsoft VSS issue. Microsoft has nothing on this for us.
In the activity log I see the following:

12/03/08 00:02:56 ANR0444W Protocol error on session 37690 for node ServerX (WinNT) - out-of-sequence verb (type BeginTxn) received. (SESSION: 37690)
12/03/08 00:02:56 ANR0484W Session 37690 for node ServerX (WinNT) terminated - protocol violation detected. (SESSION: 37690)

I see the following in the dsmsched.log

12/03/2008 00:05:19 ANS1999E Incremental processing of 'SCOMSQL\SystemState\NULL\System State\SystemState' stopped.
12/03/2008 00:05:33 ANS5280E Object enumeration from a file set or file list failed.


In the dsmerror log, we get the following:

12/02/2008 23:25:15 ANS1959W Removing previous incomplete group '\System State\0\SystemState' Id:0-144544555
12/03/2008 00:05:18 ANS5279E Error processing '\\SERVERX\c$\Windows\System32\drivers\tmtdi.sys': file not found.
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : VssRequestor::VssBackupComplete
TSM function : BackupComplete() returned VSS_E_BAD_STATE
TSM return code : 4345
TSM file : vssreq.cpp (1958)
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : CompleteVssSnapshot
TSM function : psVssBackupComplete() failed
TSM return code : 4345
TSM file : txncon.cpp (6389)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackQry: Received rc: -50 trying to send BackQry verb
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssCloseGroups
TSM function : baCloseGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\00000000000000\Top\SystemState\System State\0\SystemState' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2220)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\BACKUP_STATE\TSM\00000000000000' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\00000000000000\Top\SystemState\Bootable\Bootable System State\0\Bootable' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\Bootable\COM+ REGDB\COM+ REGDB\1\COM+ REGDB Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\Bootable\Registry\Registry\1\Registry Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\Bootable\System Files\System Files\1\System Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\IISCONFIG\null\1\IIS Config Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:18 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:18 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\IISMETABASE\null\1\IIS Metabase Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:19 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:19 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:19 cuBackUpd: Received rc: -50 trying to send BackUpd verb
12/03/2008 00:05:19 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:19 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : baUpdateGroupLeader() for file 'SERVERX\SystemState\NULL\System State\SystemState\TSM\FULL\20081202232457\Top\SystemState\WMI\Windows Managment Instrumentation\1\WMI Writer' returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2633)
12/03/2008 00:05:19 sessSendVerb: Error sending Verb, rc: -50
12/03/2008 00:05:19 ANS5250E An unexpected error was encountered.
TSM function name : deleteServerObj
TSM function : Cleaning up TSM group on server failed
TSM return code : -50
TSM file : vssback.cpp (6716)
12/03/2008 00:05:19 ANS5250E An unexpected error was encountered.
TSM function name : vssUpdateGroups
TSM function : updateDCAfterTSMBkup() returned rc=-50
TSM return code : -50
TSM file : vssback.cpp (2666)
12/03/2008 00:05:19 ANS1999E Incremental processing of 'SERVERX\SystemState\NULL\System State\SystemState' stopped.
12/03/2008 00:05:33 ANS5280E Object enumeration from a file set or file list failed.
12/03/2008 00:05:37 ANS1512E Scheduled event 'BKUP_A_2322_SERVERX' failed. Return code = 12.

Any ideas?

 

[rdeselle]

Brian,

Can you check and make sure that the c$\Windows\System32\drivers\tmtdi.sys' file does exist?

If not, try and create this file and take another backup of the SYstemstate.

Robert
TSM level 2

 

[BrianP]

The file does exist

 

[golfdork]

VSS problem

We are having the same problem with OS backups on 2008. There is a patch for 2008 for VSS -- we are waiting on an outage to patch and confirm the issue:

http://support.microsoft.com/kb/956697

HTH

Kathi

 

[Tman77]

I tried the KB article listed with no success. :-( Any more thoughts on this problem.

 

[hogmaster]

That patch is for hyper-V or did I miss something?
from the fix:
Windows Server 2008 Hyper-V VSS writer is not used during a backup job because of corrupted or invalid virtual machine configuration files

 

[gyuzik]

1. Restart COM+ Event System service.
2. Stop 'volume shadow copy' and 'microsoft sofwtare shadow copy provider'
3. issue a 'vssadmin list writers' at a command prompt and check to see if all writers are in a stable state.
4. I have found this issue is generally caused by a lack of space on the c drive of our esx guests.

 

[Tman77]

Every thing looks good here also.

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {3e0b29b0-9735-4d2d-b9a5-aeea842ce206}
State: [1] Stable
Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {c58aaeb5-bd4d-4af6-993d-0a21bcc1c043}
State: [1] Stable
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {cb24c882-37f3-4d28-94a0-2e4687ac03a6}
State: [1] Stable
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {257259ef-2791-49f1-a187-110fcc22d9ae}
State: [1] Stable
Last error: No error
Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {5179c8e1-bc5e-4d79-bcef-9c40d4f5e27b}
State: [1] Stable
Last error: No error

 

[Tman77]

ok so had chance to run a system state after doing everything gyuzik talked about and it passed. So we will see how it runs tongiht and I will let you all know tomorrow how ended up.

 

[Tman77]

The backup completed succesfully last night. Thanks to all for your help!

 

[vaibhao.wagh]

Hi Guys,
I tried the same trick as GYUZIK suggested it worked for only one scheduled backup but on next day backup is keep on failing on below error

"02/05/2010 14:05:40 ANS5250E An unexpected error was encountered.
TSM function name : isVolumeSupported
TSM function : IsVolumeSupported() returned 'VSS_E_OBJECT_NOT_FOUND' for volume 'p:\'
TSM return code : 4353
TSM file : vssreq.cpp (5542)
02/05/2010 14:05:40 ANS5250E An unexpected error was encountered.
TSM function name : vssCheckVolumeSupport
TSM function : isVolumeSupported() returned unexpected error for volume '\\*******\group$'
TSM return code : 4353
TSM file : vssback.cpp (3086)
02/05/2010 14:05:40 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log."

OS Name: Microsoftr Windows Serverr 2008 Enterprise without Hyper-V
OS Version:6.0.6002 Service Pack 2 Build 6002
TSM server Version : 5, Release 5, Level 2.0
TSM Client Version : 5, Release 5, Level 1.0

Could you please help me out to solve the issue?

Many Thanks in Advance

 

[domino]

Is it a cluster?

then look here for further informations:
http://www-01.ibm.com/support/docview.wss?uid=swg21402819

domino

 

[Tman77]

If it is not a cluster

We later had some more problems with this and finally got a reply back from IBM. They told us if we have 2008 server and sp2 that we needed to be on the 6.1 client.

Well we did that then we got.

02/03/2010 02:05:58 ANS5279E Error processing '\\server\drive$\path\file.extension: file not found.

02/03/2010 02:05:58 ANS5279E Error processing '\\tdcexchxm001\c$\Windows\System32\ibmwasr.exe': file not found.


IBM at this point started working with Microsoft and then they released 6.1.3.1 and you had to add this in the dsm.opt.

testflag SKIPMISSINGSYSWFILES
SKIPMISSINGSYSWFILES yes


Within reason the problem was fixed. In other words it will not show up as a failure in your reports. We were also told that there will be a complete fix in the 2nd half of 2010.

Let me know if you need any more information and good luck.

 

[vaibhao.wagh]

It is not a cluster server .

As per my understanding upgrading the client server is not problem solution as backup is happening after performing below steps

1. Restart COM+ Event System service.
2. Stop 'volume shadow copy' and 'microsoft sofwtare shadow copy provider'

 

[Jeff_Jeske]

^ We are seeing the same issue. I recently upgrade the problematic clients to 5.5.2.10 (YES TEN) and the issue continues unless I perform the service restarts and stop listed above.

 

[Ratgen]

4. I have found this issue is generally caused by a lack of space on the c drive of our esx guests.

Than you gyuzik
Cant understand i didnt see that one. Must have seen this problem a million times, but only on machines without VSS enabled. And the error code is quite different there...