Failed using "IBM Spectrum Protect VSS" snapshot

[Sharon-Iscar]

Greetings Saviors ,

We have 200+ VM's which are failing with VSS backup.

*In the OPT : INCLUDE.VMSNAPSHOTATTEMPTS * 3 1

*i'm including several screenshot for better understanding.

ERROR LOG :

03/12/2021 18:26:02 ANS4066I Snapshot operation attempt 1 of 4 for the guest virtual machine 'Iscar-3DEXPMono1' failed using "IBM Spectrum Protect VSS" snapshot.
Reattempting snapshot with "IBM Spectrum Protect VSS using Microsoft System Provider" snapshot.
03/12/2021 18:27:38 ANS9417E IBM Spectrum Protect application protection could not freeze the VSS writers on the virtual machine named '3dxdb-qa'. See the error log for more details.

As suggested by ibm , on the vm i ran : vssadmin list writers , And got this :

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {5300b8af-9c1c-4e77-a873-62de2eea7c88}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {3b1713ce-f05c-4d71-994d-13499d977ebf}
State: [5] Waiting for completion
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {cca6c45d-4649-46c2-9dbb-03ec292d3e58}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {952d5501-320e-4ddf-93ca-df4a889f3afa}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {afe8550a-939f-4885-bf9a-a22489a015c3}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {c67b8773-fd58-49f3-89c0-85f989f2fd63}
State: [1] Stable
Last error: No error

This is just one Example , all server return stable , and no error.
Application protection is Enabled.

Appreciate any kind of information \ tips.

 

[Trident]

Hi,

I cannot find any attachments here

What does error/event/task logs show when you run backup?

 

[Sharon-Iscar]

FROM dsmerror:

03/12/2021 18:26:02 ANS4066I Snapshot operation attempt 1 of 4 for the guest virtual machine 'Iscar-3DEXPMono1' failed using "IBM Spectrum Protect VSS" snapshot.
Reattempting snapshot with "IBM Spectrum Protect VSS using Microsoft System Provider" snapshot.
03/12/2021 18:27:38 ANS9417E IBM Spectrum Protect application protection could not freeze the VSS writers on the virtual machine named '3dxdb-qa'. See the error log for more details.


vssadmin list writers returns the following:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {5300b8af-9c1c-4e77-a873-62de2eea7c88}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {3b1713ce-f05c-4d71-994d-13499d977ebf}
State: [5] Waiting for completion
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {cca6c45d-4649-46c2-9dbb-03ec292d3e58}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {952d5501-320e-4ddf-93ca-df4a889f3afa}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {afe8550a-939f-4885-bf9a-a22489a015c3}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {c67b8773-fd58-49f3-89c0-85f989f2fd63}
State: [1] Stable
Last error: No error

 

[LED888]

Two messages from the dsmerror.log, not much to go by.
Are there any error messages in the vmware.log around the time frame of the messages in the dsmerror.log?
Are there any message in the event/task log?


The way that I understand it that TDP for VE doesn't actually perform the backup, it relies on the Microsoft VSS writers to perform the backup.

I suggest performing the steps described in "Manual VMware snapshot compared to Tivoli StorageManagers".
If this fails, this confirms that the issue is not with TDP for VE but with Windows.

Good Luck,
Sias

 

[Sharon-Iscar]

Thanks for your reply
I did what you suggested , the snapshot via vmtools works fine.
I guess its a TDP issue .
Any other suggestions on what to check \ perform ?

 

[RecoveryOne]

Since it's calling out application protection, are you attempting to do a backup vm during a peak work load? Have you performed a non-application protection backup?

 

[Sharon-Iscar]

Im doing the backup in the middle of the night while the VM has no traffic at all , non application protection backup is complete succsesfully .

 

[Trident]

Hi,

We still need to see some more logs to help you out. Below is one thing that you may follow up on

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {3b1713ce-f05c-4d71-994d-13499d977ebf}
State: [5] Waiting for completion
Last error: No error

And please look into sql logs and windows eventlog from the same time frame as backup fails.

 

[Sharon-Iscar]

I will refer this issue to our DBA and VE Admin , and come back with some more logs and information.
I appreciate the help

 

[RecoveryOne]

Im doing the backup in the middle of the night while the VM has no traffic at all , non application protection backup is complete succsesfully .

And as Trident said, need a bit more to help you out. On the other hand, good to know a non-application aware backup is working. I've yet to deploy application aware backups as our DBA's like to control everything via their tools, so afraid I might not be much more help.