System State Backups Failing

[Kenks]

We have 2 machines where System State Backups are failing constantly.

Manual System State backups via NT Backup work fine.

01/27/2010 10:42:53 ANS1375E The snapshot operation failed.

01/27/2010 10:42:53 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

01/27/2010 10:44:27 ANS1959W Removing previous incomplete group '\System State\0\SystemState' Id:0-68612731
01/27/2010 10:44:33 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 10:44:33 ANS5268W The Microsoft Volume Shadow Copy Services writer 'Dhcp Jet Writer' current state (VSS_WS_FAILED_AT_FREEZE) is not valid for the current operation. 
01/27/2010 10:44:33 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 10:45:03 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 10:45:03 ANS5268W The Microsoft Volume Shadow Copy Services writer 'Dhcp Jet Writer' current state (VSS_WS_FAILED_AT_FREEZE) is not valid for the current operation. 
01/27/2010 10:45:03 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 10:45:33 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 10:45:33 ANS5268W The Microsoft Volume Shadow Copy Services writer 'Dhcp Jet Writer' current state (VSS_WS_FAILED_AT_FREEZE) is not valid for the current operation. 
01/27/2010 10:45:33 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 10:46:03 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 10:46:03 ANS5268W The Microsoft Volume Shadow Copy Services writer 'Dhcp Jet Writer' current state (VSS_WS_FAILED_AT_FREEZE) is not valid for the current operation. 
01/27/2010 10:46:03 ANS5271E A Microsoft Volume Shadow Copy Services writer is in an invalid state before snapshot initialization.
01/27/2010 10:46:03 ANS1327W The snapshot operation for '<SERVERNAME>\SystemState\NULL\System State\SystemState' failed with error code: 4353.
01/27/2010 10:46:03 ANS1228E Sending of object '<SERVERNAME>\SystemState\NULL\System State\SystemState' failed
01/27/2010 10:46:03 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

01/27/2010 10:46:03 ANS1375E The snapshot operation failed.

01/27/2010 10:46:03 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

It seems like it always has to remove the incomplete systemstate group. Where would this be stored, so that I can manually delete all of it before trying to run another backup.

Can anybody help shed some light on this issue?

Thanks

 

[hyderchicagotsm]

See if the VSS hotfixes have been applied:
http://www-01.ibm.com/support/docview.wss?uid=swg21242128

Also run the 'vssadmin list writers' on the windows command line right after a failed backup to see if the writers are error-free.

 

[zsoltesz]

As I know the incomplete system state is stored the c:\adsm.sys folder.

Zoli

 

[tsmhelum]

We are starting to see VSS pop up again also.

27/2010 00:18:58 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:18:58 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:18:59 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:19:28 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:19:28 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:19:29 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:19:58 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:19:58 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:19:59 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:20:29 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:20:29 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:20:29 ANS5271E A Microsoft Volume Shadow Copy Services writer is in an invalid state before snapshot initialization.
01/27/2010 00:20:29 ANS5250E An unexpected error was encountered.
TSM function name : baHandleSnapshot
TSM function : BaStartSnapshot() failed.
TSM return code : 4353
TSM file : backsnap.cpp (3808)
01/27/2010 00:20:29 ANS1327W The snapshot operation for 'ADIBURG\SystemState\NULL\System State\SystemState' failed with error code: 4353.
01/27/2010 00:20:29 ANS5283E The operation was unsuccessful.

01/27/2010 00:20:29 ANS1512E Scheduled event 'MIDNIGHT' failed. Return code = 12.


C:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {6690ad3a-0460-4b77-bd22-2c1ce747d870}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {9b8eac19-1074-49f9-8eb5-51bec169a909}
State: [1] Stable
Last error: Retryable error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {5dfd2de2-39a4-462c-91f4-ae8756baff3d}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {b9af5b0c-a9bd-403a-8113-2336e48265c0}
State: [1] Stable
Last error: Retryable error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {15e5e128-c1eb-49fa-b4ed-2c1c0aba14f0}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {0a5e8075-1472-4d85-bd99-1db67e9fe06f}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {44416dc6-16bb-4110-bb4f-f1af3fbf109c}
State: [1] Stable
Last error: No error

This is happening on a server that was recently up graded to the TSM 6.1.0.2 client

 

[rojar_toppo]

Windows Volume Shadowcopy Services (VSS) Problem Determination

You can try following link
http://publib.boulder.ibm.com/tividd/td/TSMM/SC32-9103-01/en_US/HTML/cli_vss.html

"VSS_E_WRITERERROR_RETRYABLE" might be a transient error caused by other system activity using System resources that VSS also uses .. during the backup window.

Try manipulating these values in options file
TESTFLAG SETVSSMAXRETRY:10
TESTFLAG SETVSSDELAY:300

Also a system reboot has been known to fix the VSS issues often ,
but since you say ntbackup works at the same time the tsm fails , i would suspect TSM more .

 

[fuzzballer]

We see the same issues from time to time and we reboot our servers weekly. The following suggestion from Microsoft helps for a while on our Win2K3 servers. I put these in a batch file.
cd /d %windir%\system32
Net stop vss
Net stop swprv
regsvr32 ole32.dll
regsvr32 oleaut32.dll
regsvr32 vss_ps.dll
vssvc /register
regsvr32 /i swprv.dll
regsvr32 /i eventcls.dll
regsvr32 es.dll
regsvr32 stdprov.dll
regsvr32 vssui.dll
regsvr32 msxml.dll
regsvr32 msxml3.dll
regsvr32 msxml4.dll

Hope it helps but I know it is not a permanent fix.

 

[smitty]

We are starting to see VSS pop up again also.

27/2010 00:18:58 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:18:58 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:18:59 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:19:28 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:19:28 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:19:29 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:19:58 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:19:58 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:19:59 ANS5261W An attempt to create a snapshot has failed.
Another attempt will be made to create the snapshot in 30 seconds.
01/27/2010 00:20:29 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus failed with hr=VSS_E_WRITERERROR_RETRYABLE
01/27/2010 00:20:29 ANS5268W The Microsoft Volume Shadow Copy Services writer 'System Writer' current state (VSS_WS_STABLE) is not valid for the current operation.
01/27/2010 00:20:29 ANS5271E A Microsoft Volume Shadow Copy Services writer is in an invalid state before snapshot initialization.
01/27/2010 00:20:29 ANS5250E An unexpected error was encountered.
TSM function name : baHandleSnapshot
TSM function : BaStartSnapshot() failed.
TSM return code : 4353
TSM file : backsnap.cpp (3808)
01/27/2010 00:20:29 ANS1327W The snapshot operation for 'ADIBURG\SystemState\NULL\System State\SystemState' failed with error code: 4353.
01/27/2010 00:20:29 ANS5283E The operation was unsuccessful.

01/27/2010 00:20:29 ANS1512E Scheduled event 'MIDNIGHT' failed. Return code = 12.


C:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {6690ad3a-0460-4b77-bd22-2c1ce747d870}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {9b8eac19-1074-49f9-8eb5-51bec169a909}
State: [1] Stable
Last error: Retryable error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {5dfd2de2-39a4-462c-91f4-ae8756baff3d}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {b9af5b0c-a9bd-403a-8113-2336e48265c0}
State: [1] Stable
Last error: Retryable error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {15e5e128-c1eb-49fa-b4ed-2c1c0aba14f0}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {0a5e8075-1472-4d85-bd99-1db67e9fe06f}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {44416dc6-16bb-4110-bb4f-f1af3fbf109c}
State: [1] Stable
Last error: No error

This is happening on a server that was recently up graded to the TSM 6.1.0.2 client

We are having the same error - did you find a fix for it?

 

[JonRoenick]

I run the following in a .bat file using RunAs Administrator

cd c:\windows\system32
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.*  /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.*  /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.*  /grant "BUILTIN\Users:(RX)"

Though my problem is that the System Writer stops working. The above will fix it until the next time the server is rebooted for some reason.