Random VM backup errors

[DanGiles]

I have VM backups running fairly smoothly, but there are a couple of clusters that consistently gives me backup errors. There are usually just a couple failing, but which ones fail are pretty random, so getting a VM trace is nearly impossible.
The errors are one of the following:
06/29/2017 20:41:44 ANS9365E VMware vStorage API error for virtual machine 'XXXX'.
IBM Spectrum Protect function name : VixDiskLib_Cleanup
IBM Spectrum Protect file : ..\..\common\vm\vmvddksdk.cpp (1817)
API return code : 26
API error message : Unable to connect to the host

06/28/2017 20:54:46 ANS0361I DIAG: miniThreadManagerThread:VmThreadReadTop(): thread queue error: rc=107.
06/28/2017 20:54:53 ANS4174E Full VM backup of VMware Virtual Machine 'YYYY' failed with RC=-1 mode=Incremental Forever - Incremental, target node name='VEBU_BURLINGTON', data mover node name='VEBU_BURLINGTON_DMM4'

06/28/2017 20:46:45 ANS9365E VMware vStorage API error for virtual machine 'BUPNVRINav1'.
IBM Spectrum Protect function name : VixDiskLib_Open
IBM Spectrum Protect file : ..\..\common\vm\vmvddksdk.cpp (1868)
API return code : 3014
API error message : Insufficient permissions in the host operating system

No errors in vSphere or on the guest machine itself. And no, this has NOTHING to do with permissions!

Anyone else see behaviour like this?

 

[dcabro]

I see this is an old post, but strangely, I'm seeing similar with DP for VE 8.1.9 with vCenter/ESXi 6.7.

44 VMs completed, 3 failed. There were no errors yesterday, and I don't think there will be any tomorrow.

05/05/2020 20:49:22 ANS4174E Full VM backup of VMware Virtual Machine 'XXX' failed with RC=137 mode=Incremental Forever - Incremental, target node name='DM', data mover node name='DM'
05/05/2020 20:49:22 ANS1228E Sending of object 'XXX' failed.
05/05/2020 20:49:22 ANS1025E Session rejected: Authentication failure

05/05/2020 20:49:28 ANS9365E VMware vStorage API error for virtual machine 'XXX'.
IBM Spectrum Protect function name : VixDiskLib_Cleanup
IBM Spectrum Protect file : ..\..\common\vm\vmvddksdk.cpp (1896)
API return code : 26
API error message : Unable to connect to the host

 

[dcabro]

As I suspected, the next run completed without errors...

 

[dcabro]

There is also "ANS1314E File data currently unavailable on server" that I randomly get on Hyper-V VM backups.

 

[dcabro]

And there is one thing in common when this happens, all servers have the same error in the act log:

06/15/2020 23:05:12 ANR0548W Retrieve or restore failed for session 4339 for
node NODENAME () processing file space
\VMFULL-NODENAME (fsId 15) for file
\NODENAME\SNAPSHOT_000269000_20200614230418\Hard Disk
2\JOB000269002\MBLK000002e1.CTL stored as Backup - error
detected. (SESSION: 4339)

I have a PMR open but this looks like internal server error to me. Will give you an update when I have one.

 

[marclant]

During a VM backup, the client needs to restore the control (.ctl) files from the previous backup to determine what has changed. You will get those errors if the volumes that they are on is unavailable. There's probably other errors in the activity log referencing unavailable volumes.

 

[dcabro]

Nope, no other errors. What's worse, backup will complete fine if I kick it off manually, or during next scheduled run.