Hi Zoltan,
It sounds like you are seeing multiple issues, best to deal with them one
at a time.
1) For the crash dump messages, it is possible that there is only one dump,
and the message just keeps repeating because the client detects the same
file over and over again. Check the corresponding dsmcrash.log file (start
at the bottom of that file) to see if crashes are constantly occurring, and
when they occurred. If the crashes are persistent, you should open a PMR
and upload the dump, so it can be reviewed. If it is a one-time crash, you
can do the same thing, or you can delete the dump file (dsmcrash.dmp),
which should eliminate the messages. If the problem reoccurs, open a PMR,
etc.
2) I am not sure what the hung sessions are. Sessions in IdleW state mean
that the session is not in transaction with the server. Most likely these
are consumer sessions waiting for work, possibly related to item 3, below.
3) You mentioned the backups run longer, and you observe "Process Dirs" as
the category with the most time in it. "Process Dirs" is the time the
client spends traversing the file system, searching for changed files. A
lengthy "Process Dirs" time is not necessarily a "problem" for file systems
with a large number, e.g., hundreds of thousands, or millions, of files;
and that number can be the largest if the number of changed files is very
small in comparison. This could also explain consumer sessions in IdleW
state (item 2, above).
Does the dsminstr.log file include data from when the client level was
7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the
earliest 8.1.0.2 metrics to see if there was truly some big difference
introduced at the time you upgraded the client. Off the top of my head, I
cannot think of anything in the client code itself that would suddenly
cause such a performance issue.
Without actual data to see what is going on (complete dsmsched.log file,
dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide
more specific suggestions.
Regards,
Andy
____________________________________________________________________________
Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
IBM Tivoli Storage Manager links:
Product support:
https://www.ibm.com/support/entry/portal/product/tivoli/tivoli_storage_manager
Online documentation:
http://www.ibm.com/support/knowledgecenter/SSGSG7/landing/welcome_ssgsg7.html
Product Wiki:
https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%20Storage%20Manager
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
10:36:15:
> From: Zoltan Forray <zforray AT VCU DOT EDU>
> To: ADSM-L AT VM.MARIST DOT EDU
> Date: 2017-10-06 10:37
> Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> We have this server that is used to backup DFS mounts/files. It has 30+
> TSM nodes configured to it.
>
> In the past, everything has been running like it should, albeit a little
> slow sometimes due to the size of some of the DFS mounts but the backups
> would usually finish.
>
> After upgrading the client to 8.1.0.2, we are experiencing constant
> problems. Many of the backups are not completing, some node sessions are
> running 10+ hours to examine 1000 files and backup 7MB in changes, etc.
>
> Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core
> file or crash report was found:" crash-dumps for most of the nodes, often
> numerous times within the same backup session.
>
> Instrumentation statistics on a few nodes I examined say the largest time
> consumer is "Process Dirs".
>
> From the SP server perspective, these nodes are in a constant "Idle
Wait".
> Session info shows lots of metadata send to the client, but very little
> coming from the client.
>
> The only change has been the SP client upgrade. Yes we have rebooted the
> server (Windows 2012) multiple times to clear the hung sessions
(sometimes
> running multiple days and never ending).
>
> Any way to determine is going on?
>
>
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit https://urldefense.proofpoint.com/v2/url?
> u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
>
nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHGsrMocoLUGDXZU&e=
>
|
