Scheduled UNC backup fails with ANS1228E, ANS1074W, ANS1028S, ANS9999E

[combato]

Hi Pro's

I have recently started to run scheduled client backups of my NAS via CIFS shares added as UNC paths in my dsm.opt file. All small volumes I previously tried with worked perfectly. But now I started a backup of a 10TB big volume. After ~400 GB the backup fails with this errors:

2016-03-11 11:31:44 ANS9999E ..\..\common\winnt\ntrc.cpp(765): Received Win32 RC 87 (0x00000057) from FileRead(): ReadFile '\\filer1\file4876_vol2$\proj12\SW_27168_H.TXT'. Error desription: The parameter is incorrect.
.....
......
2016-03-11 11:31:45 ANS1228E Sending of object '\\filer1\file4876_vol2$\file1.igs' failed.
2016-03-11 11:31:45 ANS1074W The operation was stopped by the user.

2016-03-11 11:31:45 ANS1999E Incremental processing of '\\filer1\file4876_vol2$' stopped.
2016-03-11 11:31:45 --- SCHEDULEREC STATUS BEGIN
....
.......
2016-03-11 11:31:45 ANS1028S An internal program error occurred.
2016-03-11 11:31:45 --- SCHEDULEREC OBJECT END M_4250_INC 2016-03-11 10:03:00
2016-03-11 11:31:45 ANS1512E Scheduled event 'M_4250_INC' failed. Return code = 12.
....
......
2016-03-11 11:31:45 Server Version 6, Release 3, Level 4.200

 

[LED888]

Check out the following techdoc -

ANS9999E Received Win32 RC 87 from fioScanDirEntry()" during the backup

What version of the TSM B/A Client are we using and what flavor of Windows is hosting the client?

Good Luck,
Sias

 

[LED888]

After some further thought, the following techdoc looks closer to what we are enountering.

ANS9999E error during backup of corrupted file

Good Luck,
Sias

 

[combato]

Thank you for the answer!

My problem is that the file is located on a read-only filesystem (it's a netapp secondary) and the file can not be removed. I tried to copy the file to another location and then it works to backup the file.
If TSM finds a file it can't backup I thought that it would skip it rather than get an internal program error...

 

[marclant]

If TSM finds a file it can't backup I thought that it would skip it rather than get an internal program error...

It depends on the reason it can't back it up. If it was standard access denied error (Received Win32 RC 5), then yes. It would show an access denied error and continue.

With RC 87, it's different because it means "The parameter is incorrect". So TSM has no way to know that it's because of read access, because that's not the error returned by Windows.

Here's how to look up Win32 return codes:


I would tend to go with the first technote LED888 shared instead of the second due to the RC 87 making it a closer match.
What's your version of OnTap?
What's your version of Windows on the machine doing the backup?
What's your TSM Client version?

 

[combato]

Thanks!
I have read the both technotes and the first is pretty close to my problem but I run:

Netapp (cDOT) Clustred Data OnTap 8.3
Windows Server 2012 Standard
TSM 6.3.4.200

To resolve the problem was to upgrade to OnTap 7.2.7 or later. That version is for Netapp 7-mode systems. Maybe an old error has come back on the new cDOT systems?
I still have some old Netapp 7-mode systems running OnTap 8.1.4. I will try to backup one volume from that system and see if the problem exists there.

 

[combato]

Some update regarding this:

I still haven't solved it... even after some upgrades and error requests.
Had a PMR with IBM for a month and they told me after some troubleshooting that this error is from the OS (Windows 2012 R2) or the source disk system (Netapp cDOT Ontap 8.3.1P2).
After IBM closed the case I opened a case with Netapp and they still haven't found anything. I now have the following versions of my systems:

Backup node/backup server: Windows server 2012 R2 with TSM client 7.1.4.4 and TSM server 7.1.5.200 (same server)
Filer where I have my CIFS share: Netapp cDOT 8.3.1P2, with a vserver (SVM) with SMB3 enabled

I have done some wireshark and netapp packet tracing and I get the following error the same time I get the TSM error:

Wireshark trace info:
No. Time Source Destination Protocol Length Info
514856 11:11:29.409731 192.168.1.92 192.168.1.152 SMB2 131 Ioctl Response, Error: STATUS_INVALID_DEVICE_REQUEST

TSM error:
2016-05-30 11:11:29 ANS9999E ..\..\common\winnt\ntrc.cpp(771): Received Win32 RC 87 (0x00000057) from FileRead(): ReadFile '\\filer1\clone$\APPL\FRAME\KURS'. Error description: The parameter is incorrect.
2016-05-30 11:11:29 ANS1028S An internal program error occurred

I do get the "STATUS_INVALID_DEVICE_REQUEST" on other files as well and I can back them up. Can't find anything else in the trace that is unique at the time the error occur in TSM.

Anyone that has any input?

 

[combato]

Opened a case with IBM a month ago after I found that the files TSM was complaining about was broken symlinks created in Linux with the 'ln' command. This was a NTFS volume but it have some unix data on it.
I got a temporary fix for this:

testflag CONTINUEWITHUNKNOWNRC:87

/C

 

[rpandey]

Hi
I am also having same issue. Have you solved this issue?
NS9999E ..\..\common\winnt\ntrc.cpp(780): Received Win32 RC 225 (0x000000e1) from FileOpen(): CreateFile. Error description: Operation did not complete successfully because the file contains a virus or potentially unwanted software.
ANS2698E Snapshot operation failed.

 

[combato]

No, in my case I used the temporary solution, but now I don't use CIFS backup anylonger. I am using CAB backup instead.

testflag CONTINUEWITHUNKNOWNRC:87

 

[rpandey]

Thanks
In my case, it is testflag CONTINUEWITHUNKNOWNRC:225

What is CAB backup?

 

[combato]

CAB backup is Cluster Aware Backup. I have my source data on Netapp storage and SVMs (storage virtual machines) and then CAB backup is useful.

 

[rpandey]

Thanks
What is require to implement CAB backup? Is there any document? If you are ok to share, here is my email address - [email protected]. Thanks

 

[combato]

Spectrum Protect supports CAB backup: Here is a page that may be useful:
https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.0/srv.admin/t_ndmp_reconfiguring_for_cab.html