vss errors after client upgrade in a clustered envirnoment

[tsmhelum]

Hello All,



I just upgraded a Windows client Cluster Configured backup environment from 5.3.0.0. to 5.5.0.2. The upgrade seemed to go ok but the active node Cluster node backup failed to complete successfully. I found the errors below in the Clients dsm.errorlog. I know VSS Errors have been mentioned here before but I can’t find one that pertains to a Cluster ( If A Cluster backup is even relevant at all when it pertains to VSS errors. Has anyone seen this type of VSS error before?

What is it with the 5.5.x.x Clients and these VSS errors anyway? I don’t believe these occurred with the earlier Clients that we had. Also are the VSS errors fixed in the latest 5.5.0.6 client? I would think/hope so If not is there something that I can put in the dsm.opt file to correct the issue? Or do I have to upgrade to 5.5.0.6? which makes me cringe (due to it being a Cluster and I just struggled a little through this first one.


DSM.Error.log

05/28/2008 00:13:34 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus() failed with hr=VSS_E_WRITERERROR_NONRETRYABLE
05/28/2008 00:13:34 ANS5269E The Microsoft Volume Shadow Copy Services writer 'Cluster Service Writer' current state (VSS_WS_STABLE) is not valid for the current operation or cannot be determined. The last error reported is '800423f4'.
05/28/2008 00:13:34 ANS5271E A Microsoft Volume Shadow Copy Services writer is in an invalid state before snapshot initialization.
05/28/2008 00:13:34 ANS1327W The snapshot operation for 'GOGICIANK1\SystemState\NULL\System State\SystemState' failed with error code: 4353.
05/28/2008 00:13:34 ANS1228E Sending of object 'GOGICIANK1\SystemState\NULL\System State\SystemState' failed
05/28/2008 00:13:34 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

05/28/2008 00:13:34 ANS1375E The snapshot operation failed.

05/28/2008 00:13:34 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

05/28/2008 00:13:34 ANS1512E Scheduled event 'MIDNIGHT' failed. Return code = 12.


05/28/2008 00:13:34 ANS1512E Scheduled event 'MIDNIGHT' failed. Return code = 12.
05/29/2008 00:09:22 ANS1959W Removing previous incomplete group '\System State\0\SystemState' Id:0-783968980
05/29/2008 00:10:11 VssRequestor::checkWriterStatus: VssRequestor::checkWriterStatus() failed with hr=VSS_E_WRITERERROR_NONRETRYABLE
05/29/2008 00:10:11 ANS5269E The Microsoft Volume Shadow Copy Services writer 'Cluster Service Writer' current state (VSS_WS_STABLE) is not valid for the current operation or cannot be determined. The last error reported is '800423f4'.
05/29/2008 00:10:11 ANS5271E A Microsoft Volume Shadow Copy Services writer is in an invalid state before snapshot initialization.
05/29/2008 00:10:11 ANS1327W The snapshot operation for 'GOGICIANK1\SystemState\NULL\System State\SystemState' failed with error code: 4353.
05/29/2008 00:10:11 ANS1228E Sending of object 'GOGICIANK1\SystemState\NULL\System State\SystemState' failed
05/29/2008 00:10:11 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

05/29/2008 00:10:11 ANS1375E The snapshot operation failed.

05/29/2008 00:10:11 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

05/29/2008 00:10:11 ANS1512E Scheduled event 'MIDNIGHT' failed. Return code = 12.



Dsmsched.log shows

05/28/2008 00:12:56 Performing a full, TSM backup of object 'COM+ REGDB Writer' component 'COM+ REGDB' using shadow copy.
05/28/2008 00:13:05 Performing a full, TSM backup of object 'Registry Writer' component 'Registry' using shadow copy.
05/28/2008 00:13:07 Performing a full, TSM backup of object 'System Writer' component 'System Files' using shadow copy.
05/28/2008 00:13:33 Performing a full, TSM backup of object 'Cluster Service Writer' component 'Cluster Database' using shadow copy.
05/28/2008 00:13:33 Performing a full, TSM backup of object 'Event Log Writer' component 'Event Logs' using shadow copy.
05/28/2008 00:13:33 Performing a full, TSM backup of object 'WMI Writer' component 'Windows Managment Instrumentation' using shadow copy.
05/28/2008 00:13:34 ANS1228E Sending of object 'GOGICIANK1\SystemState\NULL\System State\SystemState' failed
05/28/2008 00:13:34 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.


05/28/2008 00:13:34 ANS1375E The snapshot operation failed.

05/28/2008 00:13:34 ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

Thanks for any help in Advance!

TSM Configuration

TSM Server 5.5
AIX 5.3
Windows 2003 SP1/TSM Client 5.5.0.2

[maheshl_1982]

Please check for VSS writers status. Go to command prompt and do the list vss writers and check all writers status. If there is a writers which are in error status then that writer should be fixed. Some times by registering the some vss related dll files could solve the problem.

[sgabriel62]

VSS has been a continual issue for sometime now = each of us have experienced this.
Yes - there is a strong potential the Dll files in the registry have become corrupt or removed. Since Vss issues are Microsoft issues - have your SA re-apply SP2 from CD, then re-register the necessary Dll files associated with System state writers - also = run a Ntbackup against system state. If all works then its a TSM issue. If you execute "vssadmin list writers" from the C prompt and it returns nothing - then you have to re-registry ALL the DLLs associated with system state.
Good Luck

[Dyermull]

I was having the exact same errors and problems when backing up SYSTEMSTATE:

ANS1375E The snapshot operation failed.
ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

Sgabriel's advice to register the VSS writers fixed the issue for me (Thanks!). Here are the steps on how to properly register the VSS writers in case you were wondering.

You may or may not need the reboot at the end (I didn't).

According to Microsoft Premier Support this is the correct procedure to follow when re-registering the Volume Shadow Copy service DLL's.

Open a command prompt session

CD to Windows\System32

Net stop vss

Net stop swprv

regsvr32 ole32.dll

regsvr32 vss_ps.dll

Vssvc /Register

regsvr32 /i swprv.dll

regsvr32 /i eventcls.dll

regsvr32 es.dll

regsvr32 stdprov.dll

regsvr32 vssui.dll

regsvr32 msxml.dll

regsvr32 msxml3.dll

regsvr32 msxml4.dll

Reboot your server.

After reboot, open a command prompt and run:

vssadmin list writers

Ensure that all your writers are displayed w/o errors.

[sgabriel62]

Let me know if any of you do these registrations - I too have the same issues on several 64 bit servers and Im about to get the nerve to ask our SAs if I can run these.
Id like to know if its simple or not and if a reboot is required
Thanks

[tsmhelum]

Thanks maheshl_1982 and sgabriel62

I did the “vssadmin list writers” command as you suggested (see below) it returned no errors.

Sgabriel62,

I have never run a Ntbackup against the system state? Would that cause a problem with anything? How would that be done?

Also, The backups were completing fine before the upgrade to 5.5.0.2 it didn’t start failing until after the upgrade. I noticed the Passive node of the Cluster is completing successfully I’m confused on that also, Unless it’s due to the “Cluster resources drives H and the Q (Quorum) being on the Active node which shouldn’t make a difference.

Thanks Dyermull! we will see if it comes to that fix

[sgabriel62]

Ntbackup against systemstate is no issue - simply launch the Ntbackup utility, from the GUI or wizard - simply choose you want to select what to back up - another window should appear - from there select systemstate - and the destination of the flatfile being generated - and thats it.

Now Ntbackup - also uses VSS services - so if it "hangs" for more than 5 minutes - then kill it. Its not going to pass but simply time out.

Interesting note about the upgrade vs the cluster cause my issues are cluster related as well and I'll look but I'm thinking a security update or a patch is the issue - but I could be wrong and overlooking something as well

[Efim]

I had had very similar problem with backup SystemState. After trace the client i found that some files from system state (it is a set of files) didn't copy during vss backup, but i could copy it in explorer . After run scandisk the problem was solved (i had to restart server for correct it). I didn't receive any errors in Windows event log.

My way was the similar: i registered dll's, found a lot of patches for VSS, reinstall SP, install Resourse Kit for VSS......
NTbackup use a bit different API than TSM as VSS requester. In my case NTbackup backup SystemState without errors.

Efim
My English is still bad. Sorry...

[sgabriel62]

Efim
It would be nice to know whichfiles did not get copied over during vss backup but you could copy it from explorer - Im assuming from the TSM client extract. How does scandisk have anything to do with it - interesting this.....

[sgabriel62]

Update:
Our local fix is to reapply the current VSS patch and reboot - once the writers come back and become stable - all is well.
Still checking on the cluster quorum idea though.

[Efim]

Efim
It would be nice to know whichfiles did not get copied over during vss backup but you could copy it from explorer - Im assuming from the TSM client extract. How does scandisk have anything to do with it - interesting this.....

No, explorer = microsoft file explorer.
It was only my advice - run scandisk.
ok. I found all logs and traces.
It was two files diskpart.exe and ntmssvc.dll
I ressived those errors: ANS5250E, ANS1950E, ANS1228E, ANS1375E.
I hope i answered you.
Efim

[tsmhelum]

UPDATE:

I did the the “vssadmin list writers” command again and there is a error. Anyone how I would correct the issue?

Thanks!

Writer name: 'Cluster Service Writer'
Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
Writer Instance Id: {6017d216-3f66-472e-a5ca-c47d20b8e67b}
State: [1] Stable
Last error: Non-retryable error



C:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {12a259f2-0085-43d6-bc40-a52d4642841f}
State: [1] Stable
Last error: No error

Writer name: 'MSDEWriter'
Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
Writer Instance Id: {b237d0c3-62da-49ef-9ea2-8cff01b5f935}
State: [1] Stable
Last error: No error

Writer name: 'Event Log Writer'
Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
Writer Instance Id: {264710f2-31a6-4797-9161-55104945a645}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {c8a26a46-3d08-4dd4-82af-17cebebce127}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {fddbd722-348a-47b4-836e-1ac56418396e}
State: [1] Stable
Last error: No error

Writer name: 'Cluster Service Writer'
Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
Writer Instance Id: {6017d216-3f66-472e-a5ca-c47d20b8e67b}
State: [1] Stable
Last error: Non-retryable error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {f9eaec31-24e6-454f-bcee-92fc2dee87ba}
State: [1] Stable
Last error: No error

[sgabriel62]

What you could do is two things - first have the SA look at it.
Second - remove c:\adm.sys folder and its contents - remove the associated filespaces on the TSM server - and rerun your incremental again. If the error comes back after option two - is an OS issue

[tsmhelum]

sgabriel62 Thanks,

The SA will look at it and let me know. Also, If I were to remove the c:\adm.sys folder and its contents along with the associated filespaces would that cause an issue? Also, I have the filespaces listed below I assume the ones that would be removed are the ones I have numbered excluding number 1?

Filespace Name FSID
1 Asr
2 GOGICIANK1\Systemstate\NULL\SYSTEM\STATE\SYSTEMSTA TE 3 SYSTEM SERVICES
4 SYSTEM STATE
\\gogiciank1\c$
\\gogiciank1\\d$
\\gogiciank1\\i$
\\gogiciank1\\m$
\\gogiciank1\\n$
\\gogiciank1\\o$

[sgabriel62]

No issues. A clean slate sort to speak. And yes FSID 2 3 4 will do.
Also I may have mentioned this already - if I have sorry - but to test at the OS level - run NTBackup against system state.
I would expect it to time out or hang in your current state

[scott-davidson]

I was having the exact same errors and problems when backing up SYSTEMSTATE:

ANS1375E The snapshot operation failed.
ANS5258E An error occurred initializing a VSS request. For more information, see the TSM client error log.

Sgabriel's advice to register the VSS writers fixed the issue for me (Thanks!). Here are the steps on how to properly register the VSS writers in case you were wondering.

You may or may not need the reboot at the end (I didn't).

According to Microsoft Premier Support this is the correct procedure to follow when re-registering the Volume Shadow Copy service DLL's.

Open a command prompt session

CD to Windows\System32

Net stop vss

Net stop swprv

regsvr32 ole32.dll

regsvr32 vss_ps.dll

Vssvc /Register

regsvr32 /i swprv.dll

regsvr32 /i eventcls.dll

regsvr32 es.dll

regsvr32 stdprov.dll

regsvr32 vssui.dll

regsvr32 msxml.dll

regsvr32 msxml3.dll

regsvr32 msxml4.dll

Reboot your server.

After reboot, open a command prompt and run:

vssadmin list writers

Ensure that all your writers are displayed w/o errors.

This post was very helpful in more than one way. When I ran the vssadmin list command, I noticed that I only had two writers working. I knew from past experience that there should have been more working.

My next step was work through the many steps of re-registering the dll's. After trying to stop the vss service and it's companion service, I discovered that they were not running. So, rather than re-register all the dll's I elected to simply reboot the server. That fixed the issue.

I am not going to try and figure out what caused these services to stop working. I would like to thank everyone for their contribution to this thread. VSS is good when it works. I just wish MS fix this service.