ADSM-L

Re: [ADSM-L] VSS crashing W2K3?

2008-05-07 11:26:13
Subject: Re: [ADSM-L] VSS crashing W2K3?
From: Matthew Large <Matthew.Large AT BARCLAYSWEALTH DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 7 May 2008 16:23:19 +0100
Hi Andy,

You're right of course - I know list etiquette, I should have stated my levels:

Client 5.4.1.6

I wasn't sure about posting the entire crash analysis, but here it is:

----------------------------------------------------------------------------
BugCheck B8, {9071b3f0, f7732090, f78f7000, 0}

*** ERROR: Module load completed but symbols could not be loaded for tsmlvsa.sys
*** ERROR: Module load completed but symbols could not be loaded for vsp.sys
Probably caused by : tsmlvsa.sys ( tsmlvsa+92db ) - Driver that caused the crash



Followup: MachineOwner
---------
ATTEMPTED_SWITCH_FROM_DPC (b8)
A wait operation, attach process, or yield was attempted from a DPC routine.
This is an illegal operation and the stack track will lead to the offending
code and original DPC routine.
Arguments:
Arg1: 9071b3f0, Original thread which is the cause of the failure
Arg2: f7732090, New thread
Arg3: f78f7000, Stack address of the original thread
Arg4: 00000000

Debugging Details:
------------------

FAULTING_THREAD:  9071b3f0
DEFAULT_BUCKET_ID:  DRIVER_FAULT
BUGCHECK_STR:  0xB8
PROCESS_NAME:  Idle
CURRENT_IRQL:  1b
LAST_CONTROL_TRANSFER:  from 8083334b to 8088d6ce
STACK_TEXT:
f78bac50 8083334b 9071b3f0 9071b498 00000002 nt!KiSwapContext+0x26
f78bac7c 80829572 8e36dbb8 f78bacf4 90614f8c nt!KiSwapThread+0x2e5
f78bacc4 f72ab2db 90615090 00000000 00000000 nt!KeWaitForSingleObject+0x346

WARNING: Stack unwind information not available. Following frames may be wrong.

f78bacf8 f72b1a92 f78bad10 8e36dd24 9068ac90 tsmlvsa+0x92db - Driver loaded in 
stack when server rebooted
f78bae2c f72b1e5b 90613018 8e36dbb8 8081deb9 tsmlvsa+0xfa92
f78bae4c f750ff06 8a704000 8f315858 90687358 tsmlvsa+0xfe5b
f78bae68 f750b1da 90687358 e036dbb8 8f3159a3 vsp+0x8f06
f78baea4 f7316590 8a704000 d3a00000 00000003 vsp+0x41da
f78baebc 8081e057 00000000 8f315858 8f1a07f0 
ftdisk!FtpRefCountCompletionRoutine+0x38
f78baeec f725a7ab f78baf1c f725aba9 90618030 nt!IopfCompleteRequest+0xcd
f78baef4 f725aba9 90618030 8f315858 00000001 CLASSPNP!ClassCompleteRequest+0x11
f78baf1c 8081e057 00000000 9039c798 9039c930 CLASSPNP!TransferPktComplete+0x1fd
f78baf4c f726dce8 f78baf78 f78baf8c f7275df2 nt!IopfCompleteRequest+0xcd
f78baf58 f7275df2 9039c798 00000001 00000000 storport!RaidCompleteRequestEx+0x1c
f78baf8c f726d02e 00000000 9072545c f78baff4 
storport!RaidUnitCompleteRequest+0x56
f78baf9c 80831fde 9072545c 907253e8 00000000 
storport!RaidpAdapterDpcRoutine+0x28
f78baff4 8088d72f f78f61d0 00000000 00000000 nt!KiRetireDpcList+0xca
f78baff8 f78f61d0 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x3f
8088d72f 00000000 0000000a bb837775 00000128 0xf78f61d0

STACK_COMMAND:  .thread 0xffffffff9071b3f0 ; kb

FOLLOWUP_IP:
tsmlvsa+92db
f72ab2db 8bc6            mov     eax,esi

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  tsmlvsa+92db

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: tsmlvsa

IMAGE_NAME:  tsmlvsa.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  476038b1

FAILURE_BUCKET_ID:  0xB8_tsmlvsa+92db

BUCKET_ID:  0xB8_tsmlvsa+92db

Followup: MachineOwner
---------

Driver details of  tsmlvsa.sys:
===================
start    end        module name
f72a2000 f72bed00   tsmlvsa    (no symbols)
    Loaded symbol image file: tsmlvsa.sys
    Image path: tsmlvsa.sys
    Image name: tsmlvsa.sys
    Timestamp:        Thu Dec 13 01:08:25 2007 (476038B1)
    CheckSum:         000241E4
    ImageSize:        0001CD00
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0
---------------------------------------------------------------------------

The reboot was performed directly after the installation. I've not used the 
Driver Verifier before, so I'll need to find some commands to disable 
verification of tsmlvsa.sys

I'll see if the owners will let me do it..

Does the crash analysis reveal any more?

Many Thanks,
Matthew

------
Matthew Large
TSM Consultant
Storage Services
Barclays Wealth Technology

Desk: +44 (0) 207 977 3262
Mobile: +44 (0) 7736 44 8808
Alpha Room, Ground Floor Murray House
1 Royal Mint Court
London EC3N 4HH

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Andrew Raibeck
Sent: 07 May 2008 15:56
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] VSS crashing W2K3?

Which TSM client version do you use? (Full version, like "5.3.4.2", not
"5.3")

The dump analysis appears edited. For example, the exception type is
missing. Can you provide the complete analysis?

Is the Microsoft driver verifier running on this system? If so, disable
verification for tsmlvsa.sys.

Did you reboot after installing the client?

Regards,

Andy

Andy Raibeck
IBM Software Group
Tivoli Storage Manager Client Product Development
Level 3 Team Lead
Internal Notes e-mail: Andrew Raibeck/Tucson/IBM@IBMUS
Internet e-mail: storman AT us.ibm DOT com

IBM Tivoli Storage Manager support web page:
http://www.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageManager.html

The only dumb question is the one that goes unasked.
The command line is your friend.
"Good enough" is the enemy of excellence.

"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 05/07/2008
05:22:09 AM:

> Thanks Del,
>
> Microsoft send us back their analysis of the MSdump and highlighted
these:
>
> ---
> *** ERROR: Module load completed but symbols could not be loaded for
> tsmlvsa.sys
> *** ERROR: Module load completed but symbols could not be loaded for
vsp.sys
> Probably caused by : tsmlvsa.sys ( tsmlvsa+92db ) - Driver that
> caused the crash
>
> ---
> STACK_TEXT:(excerpt)
> WARNING: Stack unwind information not available. Following frames
> may be wrong.
>
> f78bacf8 f72b1a92 f78bad10 8e36dd24 9068ac90 tsmlvsa+0x92db - Driver
> loaded in stack when server rebooted
> f78bae2c f72b1e5b 90613018 8e36dbb8 8081deb9 tsmlvsa+0xfa92
> f78bae4c f750ff06 8a704000 8f315858 90687358 tsmlvsa+0xfe5b
>
> ----
> FOLLOWUP_IP:
> tsmlvsa+92db
> f72ab2db 8bc6            mov     eax,esi
>
> SYMBOL_STACK_INDEX:  3
> SYMBOL_NAME:  tsmlvsa+92db
> FOLLOWUP_NAME:  MachineOwner
> MODULE_NAME: tsmlvsa
> IMAGE_NAME:  tsmlvsa.sys
> DEBUG_FLR_IMAGE_TIMESTAMP:  476038b1
> FAILURE_BUCKET_ID:  0xB8_tsmlvsa+92db
> BUCKET_ID:  0xB8_tsmlvsa+92db
> ----
>
> Would the ServiceCenter app return a 'Writer name' in 'vssadmin list
> writers'? If it _is_ using VSS then it isn't obvious.
>
> BW are averse to shutting down these systems, even out-of-hours, but
> I will push to have SC shut down prior to backup.
>
> I might have to read up on the VSHADOW command before running those
> tests, thanks for the advice though.
>
> Much appreciated,
> Matthew
>
>
> ------
> Matthew Large
> TSM Consultant
> Storage Services
> Barclays Wealth Technology
>
> Desk: +44 (0) 207 977 3262
> Mobile: +44 (0) 7736 44 8808
> Alpha Room, Ground Floor Murray House
> 1 Royal Mint Court
> London EC3N 4HH
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> Behalf Of Del Hoobler
> Sent: 07 May 2008 12:52
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] VSS crashing W2K3?
>
> Matthew,
>
> Do you know if HP Openview ServiceCenter has a VSS writer?
> Could its writer be causing a crash after getting signalled by VSS?
>
> As a first diagnostic step, I would shutdown HP Openview ServiceCenter
> and retry the backup. The result of that step may tell you if you
> need to contact HP (for Openview support) or IBM (for TSM support).
> If the backup succeeds, you should contact HP, if it still fails,
> you should contact IBM TSM Support.
>
> If you are familiar with the "VSHADOW" tool, you can also try
> creating a VSS snapshot of the volumes using VSHADOW to see
> if it also crashes the machine.  VSHADOW will allow you to perform
> very basic snapshots and remove TSM from the equation.
>
> Thanks,
>
> Del
>
> ----------------------------------------------------
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 05/07/2008
> 04:41:00 AM:
>
> > [image removed]
> >
> > VSS crashing W2K3?
> >
> > Matthew Large
> >
> > to:
> >
> > ADSM-L
> >
> > 05/07/2008 04:42 AM
> >
> > Sent by:
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > Please respond to "ADSM: Dist Stor Manager"
> >
> > Guys,
> >
> > Perhaps you have seen something like this before: The W2K3sp1
> > machine which holds the HP Openview ServiceCenter database is
> > crashing while backing up. The backup solution was changed very
> > recently to TSM as BackupExec is being phased out, but both pieces
> > of software are crashing during backup with no immediately visible
> > particular cause.
> >
> > Since I'm only interested in TSM, this is what it is wrote to the
> > log before falling over:
> >
> > The description for Event ID ( 4097 ) in Source ( TsmVssPlugin )
> > cannot be found. The local computer may not have the necessary
> > registry information or message DLL files to display messages from a
> > remote computer. You may be able to use the /AUXSOURCE= flag to
> > retrieve this description; see Help and Support for details. The
> > following information is part of the event: Signaling VSS writers
> > that the backup operation has completed..
> >
> > Any help hugely appreciated.
> >
> > Many thanks,
> > Matthew
> > ------
> Barclays Wealth is the wealth management division of Barclays Bank
> PLC. This email may relate to or be sent from other members of the
> Barclays Group.
>
> The availability of products and services may be limited by the
> applicable laws and regulations in certain jurisdictions. The
> Barclays Group does not normally accept or offer business
> instructions via internet email. Any action that you might take upon
> this message might be at your own risk.
>
> This email and any attachments are confidential and intended solely
> for the addressee and may also be privileged or exempt from
> disclosure under applicable law. If you are not the addressee, or
> have received this email in error, please notify the sender
> immediately, delete it from your system and do not copy, disclose or
> otherwise act upon any part of this email or its attachments.
>
> Internet communications are not guaranteed to be secure or virus-
> free. The Barclays Group does not accept responsibility for any loss
> arising from unauthorised access to, or interference with, any
> Internet communications by any third party, or from the transmission
> of any viruses. Replies to this email may be monitored by the
> Barclays Group for operational or business reasons.
>
> Any opinion or other information in this email or its attachments
> that does not relate to the business of the Barclays Group is
> personal to the sender and is not given or endorsed by the Barclays
Group.
>
> Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
> Registered Office: 1 Churchill Place, London, E14 5HP, United Kingdom.
>
> Barclays Bank PLC is authorised and regulated by the Financial
> Services Authority.
Barclays Wealth is the wealth management division of Barclays Bank PLC. This 
email may relate to or be sent from other members of the Barclays Group.

The availability of products and services may be limited by the applicable laws 
and regulations in certain jurisdictions. The Barclays Group does not normally 
accept or offer business instructions via internet email. Any action that you 
might take upon this message might be at your own risk.

This email and any attachments are confidential and intended solely for the 
addressee and may also be privileged or exempt from disclosure under applicable 
law. If you are not the addressee, or have received this email in error, please 
notify the sender immediately, delete it from your system and do not copy, 
disclose or otherwise act upon any part of this email or its attachments.

Internet communications are not guaranteed to be secure or virus-free. The 
Barclays Group does not accept responsibility for any loss arising from 
unauthorised access to, or interference with, any Internet communications by 
any third party, or from the transmission of any viruses. Replies to this email 
may be monitored by the Barclays Group for operational or business reasons.

Any opinion or other information in this email or its attachments that does not 
relate to the business of the Barclays Group is personal to the sender and is 
not given or endorsed by the Barclays Group.

Barclays Bank PLC. Registered in England and Wales (registered no. 1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial Services 
Authority.