ADSM-L

Re: [ADSM-L] VSS and system state failures on windows 2003 clients

2010-02-12 20:37:17
Subject: Re: [ADSM-L] VSS and system state failures on windows 2003 clients
From: Wanda Prather <wprather AT ICFI DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Fri, 12 Feb 2010 20:36:04 -0500
The TSM symptom was that when you start the GUI (this was on 5.5), it takes
a Looong time to open up.  Turns out that is because the GUI does some VSS
initialization even before you back anything up.

The Windows symptom, explored once we figured that out, was that vssadmin
list writers would also hang for a looong time before returning - over 5
minutes, when the response should come back in seconds.  Also sometimes
errors in the Windows event log.

My customer reported that to Microsoft, they asked for a trace and said to
upgrade the service pack on MSSQL.  (MSSQL 2000 SP2 to SP4 on WIN2K3 SP2).
Wouldn't have believed it, unless I had been through it, but it solved the
problem.

Anyway, it was another case of something unexpected affecting VSS, where the
TSM client was just an innocent bystander.  If you get bad behavior with
VSSADMIN LIST WRITERS, whether it be an unsatisfactory status or very bad
performance or notes in the Event log, report it to Microsoft.

W


On Fri, Feb 12, 2010 at 3:34 PM, Fred Johanson <Fred AT uchicago DOT edu> wrote:

> Wanda,
>
> Tell us more about #4, please!
>
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT vm.marist DOT edu] On Behalf 
> Of
> Wanda Prather
> Sent: Thursday, February 11, 2010 10:47 AM
> To: ADSM-L AT vm.marist DOT edu
> Subject: Re: [ADSM-L] VSS and system state failures on windows 2003 clients
>
> There are soooo many things that can go wrong with VSS.  (MS tells Tivoli
> to
> call a bunch of different VSS writer API's to back up system state, but
> obviously, ntbackup does it a different way.  Coincidence? Conspiracy?)
>
> There are so many MS patches for VSS, it's hard to figure out which one is
> related to whatever problem you are having.  It's almost always a Windows
> thing.
>
> 1) If you have TSM OFS support enabled on this machine, turn it off.  See
> if
> you get different results.
>
> 2) From a windows cmd line, do:
>
> vssadmin list writers
>
> They should all say STABLE.  If not, call Microsoft.  (vssadmin list
> writers
> is a Windows command, has nothing to do with TSM.)
>
> 3) It looks from the errors like you have the Removable Storage Manager
> service running on these clients?  Do you need it?  Have you tried
> disabling
> that service?
>
> 4) If you start the TSM BA client, does it hang a looooong time before
> opening?  I've seen that in an interaction with (believe it or not) the
> version of MSSQL on the machine.
>
> 5) Have you opened a PMR with Tivoli?
>
> 6) Since you have a good backup of systemstate going to a flat file, there
> is nothing wrong with adding
> domain all-local -systemstate
> to the dsm.opt file, and just disabling the TSM systemstate backup on any
> machines you don't have the time or inclination to work through with
> Microsoft and/or Tivoli.
>
> "VSS is like a nose.  Sometimes it runs, sometimes it blows..."
>
> Wanda
>
>
>
> On Thu, Feb 11, 2010 at 11:07 AM, Dury, John C. <JDury AT duqlight DOT com>
> wrote:
>
> > I've been fighting this for several weeks now and I'm getting very
> > frustrated. I have several windows 2003 clients that consistently get the
> > errors listed below. I've done some research and installed the following
> > patches as per recommendations from several others and nothing seems to
> fix
> > the errors. I've tried multiple version of the TSM client (v5 and v6).
> I've
> > delete the file space that has the current system state. All of the
> latest
> > maintenance is installed on the Windows 2003 clients.  Sometimes I can
> get
> > the "dsmc backup systemstate" to work for one night but then the very
> next
> > night, it fails again. This is on several windows 2003 clients.  I did
> add
> > the line "PRESCHEDULECMD "ntbackup backup systemstate /F
> > C:\temp\SystemStateBackup.bkf /L:S"" in the dsm.opt for the offending
> > clients so I at least have a clean backup of the system state. I'm not
> sure
> > where to turn now. Is windows 2003 support in TSM really this bad? Any
> ideas
> > suggestions or hints are more than welcome.
> >
> >
> > TSM Client version tried (all 32bit)
> > 5.5.2.7
> > 5.5.2.10
> > 6.1.2.0
> > 6.1.3.1
> >
> > Errors received on several windows 2003 clients
> > CLIENT 1
> > 02/11/2010 09:26:25 ANS1577I The Windows console event handler received a
> > 'Ctrl-C' console event.
> > 02/11/2010 09:33:29 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 09:33:29 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state (VSS_WS_STABLE) is not
> > valid for the current operation.
> > 02/11/2010 09:33:29 ANS5274E A Microsoft Volume Shadow Copy Services
> writer
> > is in an invalid state after backup completion.
> > 02/11/2010 09:33:29 ANS5250E An unexpected error was encountered.
> >   TSM function name : CompleteVssSnapshot
> >   TSM function      : psVssBackupComplete() failed
> >   TSM return code   : 4345
> >   TSM file          : txncon.cpp (4324)
> > 02/11/2010 09:33:34 ANS1999E Incremental processing of '\\client1\d$'
> > stopped.
> > 02/11/2010 09:33:35 ANS4006E Error processing
> > '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3': directory path not
> found
> >
> > CLIENT2
> > 02/11/2010 04:10:24 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 04:10:24 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state
> (VSS_WS_FAILED_AT_FREEZE)
> > is not valid for the current operation.
> > 02/11/2010 04:10:54 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 04:10:54 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state
> (VSS_WS_FAILED_AT_FREEZE)
> > is not valid for the current operation.
> > 02/11/2010 04:11:24 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 04:11:24 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state
> (VSS_WS_FAILED_AT_FREEZE)
> > is not valid for the current operation.
> > 02/11/2010 04:11:54 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 04:11:54 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state
> (VSS_WS_FAILED_AT_FREEZE)
> > is not valid for the current operation.
> > 02/11/2010 04:11:54 ANS5271E A Microsoft Volume Shadow Copy Services
> writer
> > is in an invalid state before snapshot initialization.
> > 02/11/2010 04:11:54 ANS5250E An unexpected error was encountered.
> >   TSM function name : baHandleSnapshot
> >   TSM function      : BaStartSnapshot() failed.
> >   TSM return code   : 4353
> >   TSM file          : backsnap.cpp (3767)
> > 02/11/2010 04:11:54 ANS1327W The snapshot operation for
> > 'client2\SystemState\NULL\System State\SystemState' failed with error
> code:
> > 4353.
> > 02/11/2010 04:11:54 ANS5283E The operation was unsuccessful.
> >
> > CLIENT3
> > 02/11/2010 08:59:14 ANS1577I The Windows console event handler received a
> > 'Ctrl-C' console event.
> > 02/11/2010 09:06:14 VssRequestor::checkWriterStatus:
> > VssRequestor::checkWriterStatus failed with
> hr=VSS_E_WRITER_NOT_RESPONDING
> > 02/11/2010 09:06:14 ANS5268W The Microsoft Volume Shadow Copy Services
> > writer 'Removable Storage Manager' current state (VSS_WS_STABLE) is not
> > valid for the current operation.
> > 02/11/2010 09:06:14 ANS5274E A Microsoft Volume Shadow Copy Services
> writer
> > is in an invalid state after backup completion.
> > 02/11/2010 09:06:14 ANS5250E An unexpected error was encountered.
> >   TSM function name : CompleteVssSnapshot
> >   TSM function      : psVssBackupComplete() failed
> >   TSM return code   : 4345
> >   TSM file          : txncon.cpp (6428)
> > 02/11/2010 09:06:15 ANS1228E Sending of object '\\CLIENT3\e$\ABURNS\Work
> > Plan Analyst\2008 Files\2008 Damage Claims\WO #315451' failed
> > 02/11/2010 09:06:15 ANS4021E Error processing '\\ CLIENT3\e$\ABURNS\Work
> > Plan Analyst\2008 Files\2008 Damage Claims\WO #315451': file system not
> > ready
> > 02/11/2010 09:06:17 ANS1228E Sending of object '\\ CLIENT3\e$\gklinsk'
> > failed
> > 02/11/2010 09:06:17 ANS4005E Error processing '\\ CLIENT3\e$\gklinsk':
> file
> > not found
> > 02/11/2010 09:06:17 ANS1228E Sending of object '\\ CLIENT3\e$\glee'
> failed
> > 02/11/2010 09:06:17 ANS4005E Error processing '\\ CLIENT3\e$\glee': file
> > not found
> > Lots more on CLIENT3 like the above. The filesystem is definitely there
> and
> > working.
> >
> > Patches installed
> >
> > WindowsServer2003-KB934016-x86-ENU.exe
> > WindowsServer2003-KB940349-v3-x86-ENU.exe
> > WindowsServer2003-KB951568-x86-ENU.exe
> >
> > Did this:
> > cd /d %windir%\system32
> > Net stop vss
> > Net stop swprv
> > regsvr32 ole32.dll
> > regsvr32 oleaut32.dll
> > regsvr32 vss_ps.dll
> > Vssvc /Register
> > regsvr32 /i swprv.dll
> > regsvr32 /i eventcls.dll
> > regsvr32 es.dll
> > regsvr32 stdprov.dll
> > regsvr32 vssui.dll
> > regsvr32 msxml.dll
> > regsvr32 msxml3.dll
> > regsvr32 msxml4.dll
> >
>
>
>