1. Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING) Click the link to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This message will disappear after you have made at least 12 posts. Thank you for your cooperation.

Windows 2003 backup, missing system files [ntldr, boot.ini, etc.]

Discussion in 'Backup / Archive Discussion' started by ohwell, Nov 8, 2010.

Thread Status:
Not open for further replies.
  1. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    Hi All,

    I've got problem backing up a Windows 2003 server. After running some tests it appeared that there are system files missing in node backup such as ntldr, boot.ini, ntdetect.com, ntoskrnl.exe and so on.

    Backup tasks are finishing successfully, files are not excluded, VSS support is enabled, dsmsched.log contains information about successfull incremental backup of SystemState (and other files), but in auditlog (full) there is no information about those files - except for ones in \\hostname\c$\WINDOWS\$NtServicePackUninstall$\ or \\hostname\c$\WINDOWS\ServicePackFiles\i386\


    TBMR restores end with unbootable system. (restore to another machine)

    -----------------------

    additionally i ran these command on dsmc clients with following results:

    query backup “{hostname\SystemState\NULL\System State\SystemState}\boot.ini”
    ANS1092W No files matching search criteria were found
    query backup “{hostname\SystemState\NULL\System State\SystemState\ntdetect.com”
    ANS1092W No files matching search criteria were found
    query backup “{hostname\SystemState\NULL\System State\SystemState}\ntldr”
    ANS1092W No files matching search criteria were found

    -----------------------

    Client OS is Windows2003 SE SP2, TSM B/A Client 6.1.3.1, TSM server 6.1.3.4
    -----------------------

    Does anyone have ever had similar problem?
    Any help would be appreciated.
     
  2.  
  3. hogmaster

    hogmaster Senior Member

    Joined:
    Nov 9, 2002
    Messages:
    518
    Likes Received:
    20
    Occupation:
    Storage engineer
    I would open a PMR with IBM regarding this. Also contact Cristie to see what they know about this problem.
    We did some testing recently with TSM version 6.2.1.1 on both Windows2003 and Windows2008 without any problem.
    I know your problem have been observed before, I do not know anything about what release or anything.

    Is this a test or is it a critical situation?
     
  4. ssterry

    ssterry New Member

    Joined:
    Feb 16, 2009
    Messages:
    6
    Likes Received:
    0
    Hi,

    It sounds like the process which enumerates VSS writers and files is not passing the correct file list to TSM. There are a couple of routes you can take to investigate this.

    From a windows command prompt run diskshadow /l c:\diskshadowout.txt

    this will enter a diskshadow prompt, from here enter.

    list writers detailed

    Once complete type exit on the prompt.

    In the output file used check for the missing files. If they are not there the problem would point to the VSS writer information passed to TSM being incorrect.

    You can raise a call with Microsoft about this, but also contact TSM support as they may be able help workaround this problem with the VSS writers list.

    Thanks

    Scott
     
  5. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    Firstable, thank you for your quick response!

    @hogmaster:
    I've already contacted Cristie support, but as soon as it came out that some files are not in backup, they washed their hands and redirect me to IBM.
    Fortunatelly this are only tests, but they concerns critical servers so it is very important that we could restore it in minimum time.

    @ssterry:
    I tried do what you suggested, but there is no diskshadow utility on the system. Quick google search told me that this utility is available only in windows 2008, is there any other utility i could use to perform test that you recommended available in windows 2003 system?
     
  6. ssterry

    ssterry New Member

    Joined:
    Feb 16, 2009
    Messages:
    6
    Likes Received:
    0
    Try the Vshadow utility which is part of the Microsoft VSS SDK. The links to download the SDK, as well as full instructions on the Vshadow commands that need to be run can be found in the following DCF:
    http://www-01.ibm.com/support/docview.wss?uid=swg21412250.

    If you have a base image you can play with it is worth removing applications to see which one is affecting the way VSS works. There are many services and applications which change the writers list. Once you know which one is causing your problems you at least know which vendor to work with and have the choice to remove it and allow your backups to complete until the problem can be fixed.

    Let me know how you get on or if you need more help.
     
  7. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @ssterry:
    "The SDK needs to be installed to access the vshadow.exe file. The SDK does not need to be installed on the same system that will be analyzed since it is a stand-alone application but the copy of vshadow.exe needs to be from the correct operating system version [...]"

    If I understand correctly, I don't have to install VSS SDK on the machine i will analyze, but I can install it, on a different machine. Is that right?
    I'm asking this because affected server is very important machine running 24/7/365 - neither additional software installation nor restarts allowed.
     
    Last edited: Nov 9, 2010
  8. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    Have you tried running the DSM GUI, clicking on the Backup button and then browsing under local filesystems to these files to see if they are visible.

    They shouldn't be visible under the local filesystem (as they are systemstate), if they are visible then TSM doesn't recognize them as systemstate. If they are visible, do they have the "no entry" symbol next to them, suggesting that they are excluded by an include/exclude list?
     
  9. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    The other thing you might want to try is run a command shell and change directory to %SystemRoot%\System32

    Then run the command

    vssadmin list writers

    You should get a list of 6 writers with no errors. 'System Writer', 'WMI Writer', 'Registry Writer', COM+ REGDB Writer', 'MSDEWriter' and 'Event Log Writer'.
     
  10. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @SpecialK:
    Thank you for your reply! As for the vssadmin list writers I've checked it before and they're all stable showing no errors. However very interesting thing about those files - I ran dsm and checked local file system, and those files could indeed be seen there - ntldr, boot.ini, ntdetect.com - they are all there without exclude mark. To make sure i've checked in some random two others server and those file cannot be seen there.

    As you said it would indicate that those files are not recognized by TSM as system files, but if this is true, then why during normal backup (i performed a full audit) there is no entry about those files?
    And far more important, how to fix this?
    Have you ever had similar problem before?
     
  11. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    Do you have any HP software running on that server, for example "HP Performance Agent Software" ? Any 3rd party performance analysis software?

    I'm not sure if you can attach text files to these posts, but you could run the following command ( assuming your Windows is installed on C: )

    dsmc s c:\ntldr -traceflags=service -tracefile=dsmc.log

    Then try and post the dsm.log file.
    [FONT=&quot]

    [/FONT]
     
    Last edited: Nov 9, 2010
  12. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @SpecialK:

    As for third party software i could find only those as suspects (however no HP software installed):
    -Cimplicity HMI
    -iHistorian
    -PingPlotter
    -Symantec Enpoint Protection
    -Symantec PcAnywhere

    As attachment there is log you've asked me for, it's to big for a text file in this forum so i've packed it as .zip archive.
    (Please note that i didn't want to reveal any information about server, so i've changed hostname&nodename in that file - better safe than sorry)
     

    Attached Files:

  13. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    If this isn't a production system, I'd suggest you try

    sfc /SCANNOW

    But you'd need the original Windows installation CD for it to run OK.
     
  14. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @SpecialK
    Unfortunatelly this is one of the most critical production servers, so there is no way i could bring him down.
    Could you tell me a little more of what do you suspect? Why do you think it is neccessary or what could've happened?
     
  15. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    I suspect that something has broken your VSS API. The usual suspects would be any 3rd party applications you have installed. The VSS writers are reporting themselves without errors, so whatever it is, it's subtle.

    It looks like TSM doesn't "see" your boot files as being part of systemstate, so they aren't backed up as part of systemstate. The TSM client determines what is system state by using one of the VSS APIs. The other part of the TSM client that backs up "normal" files, does "see" your boot files as being systemstate, so it quite rightly ignores them. Unfortunately, this means that the files are not backed up at all.

    I would suspect you have more files missing than the 3 boot files you mentioned.

    You could use NTbackup to backup those boot files, this may also be broken if it uses the same VSS APIs.
     
  16. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    I did notice this in your log file though

    11/09/2010 13:02:53.812 [002896] [5464] : ntfileio.cpp (5494): fioScanDirEntry(): EXCL BootDir=bTrue && BootFile=bTrue: ntldr

    Do you have any server side excludes? Any "unusual" excludes in your dsm.opt file?

    One other thing you could try is to add the following option to your dsm.opt file.

    VSSUSESYSTEMPROVIDER YES

    Then re-run the dsmc command in post#10, without the trace flags.

    Then query the backup to see if ntldr is in it.
     
    Last edited: Nov 9, 2010
  17. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @SpecialK:
    Yes, I have excluded all .mdf and .ldf files as there is also SQL server on that machine.
    please take a look at inclexcl_.txt in attachment.
     

    Attached Files:

  18. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    It is possible, the files i know for sure are missing are ntldr, boot.ini, ntdetect.com, ntoskrnl.exe but there maybe more...
     
  19. SpecialK

    SpecialK New Member

    Joined:
    Jul 10, 2009
    Messages:
    47
    Likes Received:
    2
    One other thing you could try is to add the following option to your dsm.opt file.

    VSSUSESYSTEMPROVIDER YES

    Then re-run the dsmc command in post#10, without the trace flags.

    Then query the backup to see if ntldr is in it.
     
  20. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    @SpecialK

    I did as you requested however the result is the same.

    C:\Program Files\Tivoli\TSM\baclient>dsmc.exe s c:\ntldr

    [...]

    Selective Backup function invoked.

    Preparing to backup 'c:' using 'VSS' snapshot.
    Directory--> 0 \\hostname\c$\ [Sent]
    ANS1016I No eligible files were found.
    Selective Backup processing of '\\hostname\c$\ntldr' finished without failure.



    Total number of objects inspected: 1
    Total number of objects backed up: 1
    Total number of objects updated: 0
    Total number of objects rebound: 0
    Total number of objects deleted: 0
    Total number of objects expired: 0
    Total number of objects failed: 0
    Total number of subfile objects: 0
    Total number of bytes transferred: 325 B
    Data transfer time: 0.00 sec
    Network data transfer rate: 0.00 KB/sec
    Aggregate data transfer rate: 0.04 KB/sec
    Objects compressed by: 0%
    Subfile objects reduced by: 0%
    Elapsed processing time: 00:00:07



    tsm> query backup "{hostname\SystemState\NULL\System State\SystemState}\ntldr"

    ANS1092W No files matching search criteria were found
     
  21. ohwell

    ohwell New Member

    Joined:
    Aug 12, 2010
    Messages:
    23
    Likes Received:
    0
    I'll try to ask server team to run

    sfc /scannow

    or at least

    sfc /verifyonly

    so we could tell for sure if this is or isn't problem with corrupted files.
    However if you can think of any other possible reasons of this situation, then i would kindly ask you to share your ideas, so time needed for server team to run this command won't be wasted.
     
Thread Status:
Not open for further replies.

Share This Page