Re: [ADSM-L] Problems after upgrading client from 7.1.6.3 to 8.1.0.2

We rebooted it yesterday.   I just did a virgin install of the 8.1.0.2
package on a new VM and we are going to compare all of the files in
*baclient* and *Common Files* to see if there are any discrepancies.  Any
other directories that contain files we should be looking at?

On Fri, Oct 6, 2017 at 12:56 PM, Andrew Raibeck <storman AT us.ibm DOT com> 
wrote:

> Hi Zoltan,
>
> I do not think the problem is really with MSVCR110.dll, and the stack
> traces in the dsmcrash.log file are not going to be particularly insightful
> (other than knowing when a crash occurs, how often, and exception type). I
> suppose if you upgraded the client but did not reboot if necessary, that
> *might* be a contributing factor. But we would have to see the
> corresponding dsmcrash.dmp file to get a better sense of why it occurs.
>
> Regards,
>
> Andy
>
> ____________________________________________________________
> ________________
>
> Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
>
> IBM Tivoli Storage Manager links:
> Product support:
> https://www.ibm.com/support/entry/portal/product/tivoli/
> tivoli_storage_manager
>
> Online documentation:
> http://www.ibm.com/support/knowledgecenter/SSGSG7/
> landing/welcome_ssgsg7.html
>
> Product Wiki:
> https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> 20Storage%20Manager
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
> 12:28:07:
>
> > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 2017-10-06 12:29
> > Subject: Re: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > Andy,
> >
> > From the dsmcrash.log file:
> >
> > 2017/10/04 10:25:54
> > IBM Spectrum Protect
> > Version: 8.1.0.2
> > Build date:  Mon May 01 16:15:45 2017
> >
> > dsmcsvc.exe caused exception C0000005 (EXCEPTION_ACCESS_VIOLATION) at
> > 00000000722C7C65
> >
> > with pointers to MSVCR110.dll.
> >
> > Can you tell me what level/version this dll is supposed to be at?  I
> > suspect something didn't get updated when the client was upgraded.
> >
> >
> > On Fri, Oct 6, 2017 at 11:26 AM, Andrew Raibeck <storman AT us.ibm DOT com>
> wrote:
> >
> > > Hi Zoltan,
> > >
> > > It sounds like you are seeing multiple issues, best to deal with them
> one
> > > at a time.
> > >
> > > 1) For the crash dump messages, it is possible that there is only one
> dump,
> > > and the message just keeps repeating because the client detects the
> same
> > > file over and over again. Check the corresponding dsmcrash.log file
> (start
> > > at the bottom of that file) to see if crashes are constantly occurring,
> and
> > > when they occurred. If the crashes are persistent, you should open a
> PMR
> > > and upload the dump, so it can be reviewed. If it is a one-time crash,
> you
> > > can do the same thing, or you can delete the dump file (dsmcrash.dmp),
> > > which should eliminate the messages. If the problem reoccurs, open a
> PMR,
> > > etc.
> > >
> > > 2) I am not sure what the hung sessions are. Sessions in IdleW state
> mean
> > > that the session is not in transaction with the server. Most likely
> these
> > > are consumer sessions waiting for work, possibly related to item 3,
> below.
> > >
> > > 3) You mentioned the backups run longer, and you observe "Process Dirs"
> as
> > > the category with the most time in it. "Process Dirs" is the time the
> > > client spends traversing the file system, searching for changed files.
> A
> > > lengthy "Process Dirs" time is not necessarily a "problem" for file
> systems
> > > with a large number, e.g., hundreds of thousands, or millions, of
> files;
> > > and that number can be the largest if the number of changed files is
> very
> > > small in comparison. This could also explain consumer sessions in IdleW
> > > state (item 2, above).
> > >
> > > Does the dsminstr.log file include data from when the client level was
> > > 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with
> the
> > > earliest 8.1.0.2 metrics to see if there was truly some big difference
> > > introduced at the time you upgraded the client. Off the top of my head,
> I
> > > cannot think of anything in the client code itself that would suddenly
> > > cause such a performance issue.
> > >
> > > Without actual data to see what is going on (complete dsmsched.log
> file,
> > > dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to
> provide
> > > more specific suggestions.
> > >
> > > Regards,
> > >
> > > Andy
> > >
> > > ____________________________________________________________
> > > ________________
> > >
> > > Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
> > >
> > > IBM Tivoli Storage Manager links:
> > > Product support:
> > > https://www.ibm.com/support/entry/portal/product/tivoli/
> > > tivoli_storage_manager
> > >
> > > Online documentation:
> > > http://www.ibm.com/support/knowledgecenter/SSGSG7/
> > > landing/welcome_ssgsg7.html
> > >
> > > Product Wiki:
> > > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> > > 20Storage%20Manager
> > >
> > > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 
> > > 2017-10-06
> > > 10:36:15:
> > >
> > > > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > > > To: ADSM-L AT VM.MARIST DOT EDU
> > > > Date: 2017-10-06 10:37
> > > > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > > >
> > > > We have this server that is used to backup DFS mounts/files.  It has
> 30+
> > > > TSM nodes configured to it.
> > > >
> > > > In the past, everything has been running like it should, albeit a
> little
> > > > slow sometimes due to the size of some of the DFS mounts but the
> backups
> > > > would usually finish.
> > > >
> > > > After upgrading the client to 8.1.0.2, we are experiencing constant
> > > > problems. Many of the backups are not completing, some node sessions
> are
> > > > running 10+ hours to examine 1000 files and backup 7MB in changes,
> etc.
> > > >
> > > > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect
> core
> > > > file or crash report was found:" crash-dumps for most of the nodes,
> often
> > > > numerous times within the same backup session.
> > > >
> > > > Instrumentation statistics on a few nodes I examined say the largest
> time
> > > > consumer is "Process Dirs".
> > > >
> > > > From the SP server perspective, these nodes are in a constant "Idle
> > > Wait".
> > > > Session info shows lots of metadata send to the client, but very
> little
> > > > coming from the client.
> > > >
> > > > The only change has been the SP client upgrade.  Yes we have rebooted
> the
> > > > server (Windows 2012) multiple times to clear the hung sessions
> > > (sometimes
> > > > running multiple days and never ending).
> > > >
> > > > Any way to determine is going on?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Zoltan Forray*
> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > Xymon Monitor Administrator
> > > > VMware Administrator
> > > > Virginia Commonwealth University
> > > > UCC/Office of Technology Services
> > > > www.ucc.vcu.edu
> > > > zforray AT vcu DOT edu - 804-828-4807
> > > > Don't be a phishing victim - VCU and other reputable organizations
> will
> > > > never use email to request that you reply with your password, social
> > > > security number or confidential personal information. For more
> details
> > > > visit https://urldefense.proofpoint.com/v2/url?
> > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
> > > >
> > > nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG
> > > srMocoLUGDXZU&e=
> > >
> > > >
> > >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray AT vcu DOT edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit https://urldefense.proofpoint.com/v2/url?
> > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=SSNQ9Hyzp-
> >
> uXTAnZNWKcnvvYR9klhqPdbivWnLFH6Gk&s=pOYK0915YykOlToqNC0KdOnwwh2r93
> uz68JThlrbepU&e=
>
> >
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/