Andy,
>From the dsmcrash.log file:
2017/10/04 10:25:54
IBM Spectrum Protect
Version: 8.1.0.2
Build date: Mon May 01 16:15:45 2017
dsmcsvc.exe caused exception C0000005 (EXCEPTION_ACCESS_VIOLATION) at
00000000722C7C65
with pointers to MSVCR110.dll.
Can you tell me what level/version this dll is supposed to be at? I
suspect something didn't get updated when the client was upgraded.
On Fri, Oct 6, 2017 at 11:26 AM, Andrew Raibeck <storman AT us.ibm DOT com>
wrote:
> Hi Zoltan,
>
> It sounds like you are seeing multiple issues, best to deal with them one
> at a time.
>
> 1) For the crash dump messages, it is possible that there is only one dump,
> and the message just keeps repeating because the client detects the same
> file over and over again. Check the corresponding dsmcrash.log file (start
> at the bottom of that file) to see if crashes are constantly occurring, and
> when they occurred. If the crashes are persistent, you should open a PMR
> and upload the dump, so it can be reviewed. If it is a one-time crash, you
> can do the same thing, or you can delete the dump file (dsmcrash.dmp),
> which should eliminate the messages. If the problem reoccurs, open a PMR,
> etc.
>
> 2) I am not sure what the hung sessions are. Sessions in IdleW state mean
> that the session is not in transaction with the server. Most likely these
> are consumer sessions waiting for work, possibly related to item 3, below.
>
> 3) You mentioned the backups run longer, and you observe "Process Dirs" as
> the category with the most time in it. "Process Dirs" is the time the
> client spends traversing the file system, searching for changed files. A
> lengthy "Process Dirs" time is not necessarily a "problem" for file systems
> with a large number, e.g., hundreds of thousands, or millions, of files;
> and that number can be the largest if the number of changed files is very
> small in comparison. This could also explain consumer sessions in IdleW
> state (item 2, above).
>
> Does the dsminstr.log file include data from when the client level was
> 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the
> earliest 8.1.0.2 metrics to see if there was truly some big difference
> introduced at the time you upgraded the client. Off the top of my head, I
> cannot think of anything in the client code itself that would suddenly
> cause such a performance issue.
>
> Without actual data to see what is going on (complete dsmsched.log file,
> dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide
> more specific suggestions.
>
> Regards,
>
> Andy
>
> ____________________________________________________________
> ________________
>
> Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
>
> IBM Tivoli Storage Manager links:
> Product support:
> https://www.ibm.com/support/entry/portal/product/tivoli/
> tivoli_storage_manager
>
> Online documentation:
> http://www.ibm.com/support/knowledgecenter/SSGSG7/
> landing/welcome_ssgsg7.html
>
> Product Wiki:
> https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> 20Storage%20Manager
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
> 10:36:15:
>
> > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 2017-10-06 10:37
> > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > We have this server that is used to backup DFS mounts/files. It has 30+
> > TSM nodes configured to it.
> >
> > In the past, everything has been running like it should, albeit a little
> > slow sometimes due to the size of some of the DFS mounts but the backups
> > would usually finish.
> >
> > After upgrading the client to 8.1.0.2, we are experiencing constant
> > problems. Many of the backups are not completing, some node sessions are
> > running 10+ hours to examine 1000 files and backup 7MB in changes, etc.
> >
> > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core
> > file or crash report was found:" crash-dumps for most of the nodes, often
> > numerous times within the same backup session.
> >
> > Instrumentation statistics on a few nodes I examined say the largest time
> > consumer is "Process Dirs".
> >
> > From the SP server perspective, these nodes are in a constant "Idle
> Wait".
> > Session info shows lots of metadata send to the client, but very little
> > coming from the client.
> >
> > The only change has been the SP client upgrade. Yes we have rebooted the
> > server (Windows 2012) multiple times to clear the hung sessions
> (sometimes
> > running multiple days and never ending).
> >
> > Any way to determine is going on?
> >
> >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray AT vcu DOT edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit https://urldefense.proofpoint.com/v2/url?
> > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
> >
> nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG
> srMocoLUGDXZU&e=
>
> >
>
--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
|
