Incremental Backup Error - ANS9999E - Insufficient system resources

[jasondrummond]

Hi,

I am running Incremental Backups of a Windows 2003 Cluster. There is a node which backs up the O: and R: drives which has now stopped backing up. I get the following errors in the Sched.log and Error.logs. There appears to be nothing in the event log on the server to give me any clues.

SCHED.LOG

04/19/2007 00:25:01 --- SCHEDULEREC OBJECT BEGIN APP_INCREMENTAL_WEEKDAYS 04/18/2007 23:00:00
04/19/2007 00:25:01 Incremental backup of volume '\\server1\o$'
04/19/2007 00:25:01 Incremental backup of volume '\\server1\r$'

04/19/2007 00:54:05 ANS9999E ntrc.cpp(928): Received Win32 RC 1450 (0x000005aa) from FileRead(): ReadFile '\\server1\o$\oracle\oradata\ORAID\archive\ARC_001_23910.LOG'. Error desription: Insufficient system resources exist to complete the requested service.

04/19/2007 00:54:05 Normal File--> 104,857,088 \\server1\o$\oracle\oradata\VQ67\archive\ARC_001_23910.LOG ** Unsuccessful **

04/19/2007 01:00:28 ANS1512E Scheduled event 'APP_INCREMENTAL_WEEKDAYS' failed. Return code = 12.

ERROR.LOG

04/19/2007 00:54:05 ANS9999E ntrc.cpp(928): Received Win32 RC 1450 (0x000005aa) from FileRead(): ReadFile '\\server1\o$\oracle\oradata\ORAID\archive\ARC_001_23910.LOG'. Error desription: Insufficient system resources exist to complete the requested service.
04/19/2007 00:54:05 ANS1028S An internal program error occurred.
04/19/2007 01:00:28 ANS1512E Scheduled event 'APP_INCREMENTAL_WEEKDAYS' failed. Return code = 12.

Does anyone have any ideas?

Cheers

Jason

 

[max]

hi,

this might be due to an error in Windows, at least this is what ibm claimed, and you may want to read the following info from microsoft:
************************************
Q304101 - Backup Fails with Event ID 1450

Q247904 - How To Configure the Paged Address Pool and System Page Table Entry Memory Areas

Q142719 - Windows Reports Out Of Resources Error When Memory Is Available
Q236964 - Delayed Return of Paged Pool Causes Error 1450 "Insufficient Resources"
Q192409 - Open Files Can Cause Kernel to Report
.
There are some suggestions in document 304101 that involve changing registry settings.
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q142719
***************************************************
this is what IBM stated (copy and paste from documentation i saved):

ANS9999E ntrc.cpp(921): Received Win32 RC 1450
(0x000005aa) from FileRead(): ReadFile 'XXXXX '. Error description: Insufficient system resources exist to complete the requested service.

The return code( RC 1450) is a common error that have been seen during backups of big filesystems, or particularly large files. Windows has a certain amount of memory pool space that it can allocate to programs, and TSM is using the memory available from that pool such that there is no more memory left to allocate. This is typically caused when TSM cannot continue the backup of this file since no memory is left, so the operation has to stop. The limitation here is in Windows, and is not a TSM problem (although TSM is the program using the memory). This is a Windows limitation. Setting memoryefficient backups will not help.

hope this helps
max

 

[pstacy]

I wanted to follow up on this thread because we're having a similar issue. We're running Windows 2003 Server/TSM 5.5.0.3 on a SQL2005 server.

We utilize both the TSM for SQL agent as well as MS SQL's own backup utility and backing up the files it generates as some of the data on this server is very critical to maintain.

Lately we've been having troubles with the capturing of the backup files generated by SQL. The TSM for SQL backups still run fine. The databases on this machine range anywhere from a few MB in size up to several GB.

Generally the failure appears in any of about five different folders. The databases within these folders range between 3-7GB in size.

We've tried running manual backups during the day so we can monitor memory usage on the server and there is minimal change in the amount of memory usage. Sometimes the backups would work when done manually (or at least capture items the scheduled job didn't) but for the past week we've gone from occassional failures to daily ones.

A question I'd like to pose to everyone...might there be a limit to how much memory a client can have that Tivoli will recognize? I think back to the days of DOS and PCs that were just starting to have 1Meg of RAM or more. Many software programs had to have modifications made to their configuration to recognize memory above 640K and even then some wouldn't see the additional memory capacity. Because the failures has been intermittent (once a week or so) I've been theorizing that perhaps this client is operating right at the limit of the resources Tivoli can use, and perhaps now it has finally exceeded them.

We have reviewed the KB article mentioned in the previous post and will look into implementing its suggestions if we have to, but we're concerned about how effective it might be just because what we see in the Task Manager during a run suggests there is still a good amount of memory available to the machine.

Thank you.

 

[tenpin]

I had this problem on a Windows 2003 node trying to backup CD images, after some research I found an alternate windows page that lists NT, 2000 and 2003 with some registry fixes. I also ran across some posts indicating that journal based backups somehow could resolve this problem so I gave it a shot - it worked! Try implementing the journal engine and see if it clears up the problem!

 

[pstacy]

I had forgotten to post a followup on my earlier post. We implemented the first part of the fix in the KB article (it said the first part had to be done and the second part might be needed) and it started working again reliably after that.