Windows 2003 backup, missing system files [ntldr, boot.ini, etc.]

[ohwell]

Hi All,

I've got problem backing up a Windows 2003 server. After running some tests it appeared that there are system files missing in node backup such as ntldr, boot.ini, ntdetect.com, ntoskrnl.exe and so on.

Backup tasks are finishing successfully, files are not excluded, VSS support is enabled, dsmsched.log contains information about successfull incremental backup of SystemState (and other files), but in auditlog (full) there is no information about those files - except for ones in \\hostname\c$\WINDOWS\$NtServicePackUninstall$\ or \\hostname\c$\WINDOWS\ServicePackFiles\i386\


TBMR restores end with unbootable system. (restore to another machine)

-----------------------

additionally i ran these command on dsmc clients with following results:

query backup “{hostname\SystemState\NULL\System State\SystemState}\boot.ini”
ANS1092W No files matching search criteria were found
query backup “{hostname\SystemState\NULL\System State\SystemState\ntdetect.com”
ANS1092W No files matching search criteria were found
query backup “{hostname\SystemState\NULL\System State\SystemState}\ntldr”
ANS1092W No files matching search criteria were found

-----------------------

Client OS is Windows2003 SE SP2, TSM B/A Client 6.1.3.1, TSM server 6.1.3.4
-----------------------

Does anyone have ever had similar problem?
Any help would be appreciated.

[hogmaster]

I would open a PMR with IBM regarding this. Also contact Cristie to see what they know about this problem.
We did some testing recently with TSM version 6.2.1.1 on both Windows2003 and Windows2008 without any problem.
I know your problem have been observed before, I do not know anything about what release or anything.

Is this a test or is it a critical situation?

[ssterry]

Hi,

It sounds like the process which enumerates VSS writers and files is not passing the correct file list to TSM. There are a couple of routes you can take to investigate this.

From a windows command prompt run diskshadow /l c:\diskshadowout.txt

this will enter a diskshadow prompt, from here enter.

list writers detailed

Once complete type exit on the prompt.

In the output file used check for the missing files. If they are not there the problem would point to the VSS writer information passed to TSM being incorrect.

You can raise a call with Microsoft about this, but also contact TSM support as they may be able help workaround this problem with the VSS writers list.

Thanks

Scott

[ohwell]

Firstable, thank you for your quick response!

@hogmaster:
I've already contacted Cristie support, but as soon as it came out that some files are not in backup, they washed their hands and redirect me to IBM.
Fortunatelly this are only tests, but they concerns critical servers so it is very important that we could restore it in minimum time.

@ssterry:
I tried do what you suggested, but there is no diskshadow utility on the system. Quick google search told me that this utility is available only in windows 2008, is there any other utility i could use to perform test that you recommended available in windows 2003 system?

[ssterry]

Try the Vshadow utility which is part of the Microsoft VSS SDK. The links to download the SDK, as well as full instructions on the Vshadow commands that need to be run can be found in the following DCF:
http://www-01.ibm.com/support/docvie...id=swg21412250.

If you have a base image you can play with it is worth removing applications to see which one is affecting the way VSS works. There are many services and applications which change the writers list. Once you know which one is causing your problems you at least know which vendor to work with and have the choice to remove it and allow your backups to complete until the problem can be fixed.

Let me know how you get on or if you need more help.

[ohwell]

@ssterry:
"The SDK needs to be installed to access the vshadow.exe file. The SDK does not need to be installed on the same system that will be analyzed since it is a stand-alone application but the copy of vshadow.exe needs to be from the correct operating system version [...]"

If I understand correctly, I don't have to install VSS SDK on the machine i will analyze, but I can install it, on a different machine. Is that right?
I'm asking this because affected server is very important machine running 24/7/365 - neither additional software installation nor restarts allowed.

[SpecialK]

Have you tried running the DSM GUI, clicking on the Backup button and then browsing under local filesystems to these files to see if they are visible.

They shouldn't be visible under the local filesystem (as they are systemstate), if they are visible then TSM doesn't recognize them as systemstate. If they are visible, do they have the "no entry" symbol next to them, suggesting that they are excluded by an include/exclude list?

[SpecialK]

The other thing you might want to try is run a command shell and change directory to %SystemRoot%\System32

Then run the command

vssadmin list writers

You should get a list of 6 writers with no errors. 'System Writer', 'WMI Writer', 'Registry Writer', COM+ REGDB Writer', 'MSDEWriter' and 'Event Log Writer'.

[ohwell]

@SpecialK:
Thank you for your reply! As for the vssadmin list writers I've checked it before and they're all stable showing no errors. However very interesting thing about those files - I ran dsm and checked local file system, and those files could indeed be seen there - ntldr, boot.ini, ntdetect.com - they are all there without exclude mark. To make sure i've checked in some random two others server and those file cannot be seen there.

As you said it would indicate that those files are not recognized by TSM as system files, but if this is true, then why during normal backup (i performed a full audit) there is no entry about those files?
And far more important, how to fix this?
Have you ever had similar problem before?

[SpecialK]

Do you have any HP software running on that server, for example "HP Performance Agent Software" ? Any 3rd party performance analysis software?

I'm not sure if you can attach text files to these posts, but you could run the following command ( assuming your Windows is installed on C: )

dsmc s c:\ntldr -traceflags=service -tracefile=dsmc.log

Then try and post the dsm.log file.

[ohwell]

@SpecialK:

As for third party software i could find only those as suspects (however no HP software installed):
-Cimplicity HMI
-iHistorian
-PingPlotter
-Symantec Enpoint Protection
-Symantec PcAnywhere

As attachment there is log you've asked me for, it's to big for a text file in this forum so i've packed it as .zip archive.
(Please note that i didn't want to reveal any information about server, so i've changed hostname&nodename in that file - better safe than sorry)

[SpecialK]

If this isn't a production system, I'd suggest you try

sfc /SCANNOW

But you'd need the original Windows installation CD for it to run OK.

[ohwell]

@SpecialK
Unfortunatelly this is one of the most critical production servers, so there is no way i could bring him down.
Could you tell me a little more of what do you suspect? Why do you think it is neccessary or what could've happened?

[SpecialK]

I suspect that something has broken your VSS API. The usual suspects would be any 3rd party applications you have installed. The VSS writers are reporting themselves without errors, so whatever it is, it's subtle.

It looks like TSM doesn't "see" your boot files as being part of systemstate, so they aren't backed up as part of systemstate. The TSM client determines what is system state by using one of the VSS APIs. The other part of the TSM client that backs up "normal" files, does "see" your boot files as being systemstate, so it quite rightly ignores them. Unfortunately, this means that the files are not backed up at all.

I would suspect you have more files missing than the 3 boot files you mentioned.

You could use NTbackup to backup those boot files, this may also be broken if it uses the same VSS APIs.

[SpecialK]

I did notice this in your log file though

11/09/2010 13:02:53.812 [002896] [5464] : ntfileio.cpp (5494): fioScanDirEntry(): EXCL BootDir=bTrue && BootFile=bTrue: ntldr

Do you have any server side excludes? Any "unusual" excludes in your dsm.opt file?

One other thing you could try is to add the following option to your dsm.opt file.

VSSUSESYSTEMPROVIDER YES

Then re-run the dsmc command in post#10, without the trace flags.

Then query the backup to see if ntldr is in it.

[ohwell]

@SpecialK:
Yes, I have excluded all .mdf and .ldf files as there is also SQL server on that machine.
please take a look at inclexcl_.txt in attachment.

[ohwell]

[...]
I would suspect you have more files missing than the 3 boot files you mentioned.
[...]

It is possible, the files i know for sure are missing are ntldr, boot.ini, ntdetect.com, ntoskrnl.exe but there maybe more...

[SpecialK]

One other thing you could try is to add the following option to your dsm.opt file.

VSSUSESYSTEMPROVIDER YES

Then re-run the dsmc command in post#10, without the trace flags.

Then query the backup to see if ntldr is in it.

[ohwell]

@SpecialK

I did as you requested however the result is the same.

C:\Program Files\Tivoli\TSM\baclient>dsmc.exe s c:\ntldr

[...]

Selective Backup function invoked.

Preparing to backup 'c:' using 'VSS' snapshot.
Directory--> 0 \\hostname\c$\ [Sent]
ANS1016I No eligible files were found.
Selective Backup processing of '\\hostname\c$\ntldr' finished without failure.



Total number of objects inspected: 1
Total number of objects backed up: 1
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 0
Total number of objects failed: 0
Total number of subfile objects: 0
Total number of bytes transferred: 325 B
Data transfer time: 0.00 sec
Network data transfer rate: 0.00 KB/sec
Aggregate data transfer rate: 0.04 KB/sec
Objects compressed by: 0%
Subfile objects reduced by: 0%
Elapsed processing time: 00:00:07



tsm> query backup "{hostname\SystemState\NULL\System State\SystemState}\ntldr"

ANS1092W No files matching search criteria were found

[ohwell]

I'll try to ask server team to run

sfc /scannow

or at least

sfc /verifyonly

so we could tell for sure if this is or isn't problem with corrupted files.
However if you can think of any other possible reasons of this situation, then i would kindly ask you to share your ideas, so time needed for server team to run this command won't be wasted.

[SpecialK]

Have you tried an incremental backup of your systemstate; then re-run the query to see if the boot files have been picked up this time?

After that, I'd have to give it some serious thought.

[ohwell]

Incremental backup of server (including systemstate) is performed every night, so I can already tell that it doesn't help.
Of course if needed I may perform it again, but will it be any different than everynight incremental backup?

[SpecialK]

One other thing you can try is to add the following lines to your dsm.opt (Don't forget to remove them after your test!).

TRACEFLAGS EXCLUDECACHE
TESTFLAG SHOWEXCLUDECACHE
TRACEFILE C:\dsmc.log

Then run the dsmc command

dsmc s c:\ntldr

And post the trace log C:\dsmc.log if possible.

[ohwell]

Here is the dsmclog.txt from that try