Hi Zoltan,
I do not think the problem is really with MSVCR110.dll, and the stack
traces in the dsmcrash.log file are not going to be particularly insightful
(other than knowing when a crash occurs, how often, and exception type). I
suppose if you upgraded the client but did not reboot if necessary, that
*might* be a contributing factor. But we would have to see the
corresponding dsmcrash.dmp file to get a better sense of why it occurs.
Regards,
Andy
____________________________________________________________________________
Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
IBM Tivoli Storage Manager links:
Product support:
https://www.ibm.com/support/entry/portal/product/tivoli/tivoli_storage_manager
Online documentation:
http://www.ibm.com/support/knowledgecenter/SSGSG7/landing/welcome_ssgsg7.html
Product Wiki:
https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%20Storage%20Manager
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
12:28:07:
> From: Zoltan Forray <zforray AT VCU DOT EDU>
> To: ADSM-L AT VM.MARIST DOT EDU
> Date: 2017-10-06 12:29
> Subject: Re: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> Andy,
>
> From the dsmcrash.log file:
>
> 2017/10/04 10:25:54
> IBM Spectrum Protect
> Version: 8.1.0.2
> Build date: Mon May 01 16:15:45 2017
>
> dsmcsvc.exe caused exception C0000005 (EXCEPTION_ACCESS_VIOLATION) at
> 00000000722C7C65
>
> with pointers to MSVCR110.dll.
>
> Can you tell me what level/version this dll is supposed to be at? I
> suspect something didn't get updated when the client was upgraded.
>
>
> On Fri, Oct 6, 2017 at 11:26 AM, Andrew Raibeck <storman AT us.ibm DOT com>
wrote:
>
> > Hi Zoltan,
> >
> > It sounds like you are seeing multiple issues, best to deal with them
one
> > at a time.
> >
> > 1) For the crash dump messages, it is possible that there is only one
dump,
> > and the message just keeps repeating because the client detects the
same
> > file over and over again. Check the corresponding dsmcrash.log file
(start
> > at the bottom of that file) to see if crashes are constantly occurring,
and
> > when they occurred. If the crashes are persistent, you should open a
PMR
> > and upload the dump, so it can be reviewed. If it is a one-time crash,
you
> > can do the same thing, or you can delete the dump file (dsmcrash.dmp),
> > which should eliminate the messages. If the problem reoccurs, open a
PMR,
> > etc.
> >
> > 2) I am not sure what the hung sessions are. Sessions in IdleW state
mean
> > that the session is not in transaction with the server. Most likely
these
> > are consumer sessions waiting for work, possibly related to item 3,
below.
> >
> > 3) You mentioned the backups run longer, and you observe "Process Dirs"
as
> > the category with the most time in it. "Process Dirs" is the time the
> > client spends traversing the file system, searching for changed files.
A
> > lengthy "Process Dirs" time is not necessarily a "problem" for file
systems
> > with a large number, e.g., hundreds of thousands, or millions, of
files;
> > and that number can be the largest if the number of changed files is
very
> > small in comparison. This could also explain consumer sessions in IdleW
> > state (item 2, above).
> >
> > Does the dsminstr.log file include data from when the client level was
> > 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with
the
> > earliest 8.1.0.2 metrics to see if there was truly some big difference
> > introduced at the time you upgraded the client. Off the top of my head,
I
> > cannot think of anything in the client code itself that would suddenly
> > cause such a performance issue.
> >
> > Without actual data to see what is going on (complete dsmsched.log
file,
> > dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to
provide
> > more specific suggestions.
> >
> > Regards,
> >
> > Andy
> >
> > ____________________________________________________________
> > ________________
> >
> > Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
> >
> > IBM Tivoli Storage Manager links:
> > Product support:
> > https://www.ibm.com/support/entry/portal/product/tivoli/
> > tivoli_storage_manager
> >
> > Online documentation:
> > http://www.ibm.com/support/knowledgecenter/SSGSG7/
> > landing/welcome_ssgsg7.html
> >
> > Product Wiki:
> > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> > 20Storage%20Manager
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
> > 10:36:15:
> >
> > > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > > To: ADSM-L AT VM.MARIST DOT EDU
> > > Date: 2017-10-06 10:37
> > > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > >
> > > We have this server that is used to backup DFS mounts/files. It has
30+
> > > TSM nodes configured to it.
> > >
> > > In the past, everything has been running like it should, albeit a
little
> > > slow sometimes due to the size of some of the DFS mounts but the
backups
> > > would usually finish.
> > >
> > > After upgrading the client to 8.1.0.2, we are experiencing constant
> > > problems. Many of the backups are not completing, some node sessions
are
> > > running 10+ hours to examine 1000 files and backup 7MB in changes,
etc.
> > >
> > > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect
core
> > > file or crash report was found:" crash-dumps for most of the nodes,
often
> > > numerous times within the same backup session.
> > >
> > > Instrumentation statistics on a few nodes I examined say the largest
time
> > > consumer is "Process Dirs".
> > >
> > > From the SP server perspective, these nodes are in a constant "Idle
> > Wait".
> > > Session info shows lots of metadata send to the client, but very
little
> > > coming from the client.
> > >
> > > The only change has been the SP client upgrade. Yes we have rebooted
the
> > > server (Windows 2012) multiple times to clear the hung sessions
> > (sometimes
> > > running multiple days and never ending).
> > >
> > > Any way to determine is going on?
> > >
> > >
> > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zforray AT vcu DOT edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations
will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more
details
> > > visit https://urldefense.proofpoint.com/v2/url?
> > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
> > >
> > nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG
> > srMocoLUGDXZU&e=
> >
> > >
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit https://urldefense.proofpoint.com/v2/url?
> u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=SSNQ9Hyzp-
>
uXTAnZNWKcnvvYR9klhqPdbivWnLFH6Gk&s=pOYK0915YykOlToqNC0KdOnwwh2r93uz68JThlrbepU&e=
>
|
