Can you get me this file:
ANS2250W A IBM Spectrum Protect core file or crash report was found:
C:\Program Files\Tivoli\TSM\baclient\dsmcrash.dmp
Hi Andy,
Thanks for the reply. I have plenty of logs but obviously I can't upload
them here. This includes the instrumentation data. The upgrade was done
last Friday.
I will pull down and remove the dsmcrash.dmp file.
I pulled this from one of the dsminstr.log files. As you can see, the
processing time has jumped a lot.
Elapsed processing time: 00:00:35
Elapsed processing time: 00:00:18
Elapsed processing time: 00:00:52
Elapsed processing time: 00:00:28
Elapsed processing time: 00:00:34
Elapsed processing time: 00:00:28
Elapsed processing time: 00:01:05
Elapsed processing time: 00:00:27
Elapsed processing time: 00:02:52
Elapsed processing time: 00:02:54
Elapsed processing time: 00:02:36
Elapsed processing time: 00:02:25
This is for a small node with <1,400 objects.
Total number of objects inspected: 1,321
Total number of objects inspected: 1,324
Total number of objects inspected: 1,327
Total number of objects inspected: 1,331
Total number of objects inspected: 1,332
Total number of objects inspected: 1,339
Total number of objects inspected: 1,339
Total number of objects inspected: 1,339
Total number of objects inspected: 1,340
Total number of objects inspected: 1,342
Total number of objects inspected: 1,344
Total number of objects inspected: 1,348
---------- Forwarded message ----------
From: Andrew Raibeck <storman AT us.ibm DOT com>
Date: Fri, Oct 6, 2017 at 11:26 AM
Subject: Re: [ADSM-L] Problems after upgrading client from 7.1.6.3 to
8.1.0.2
To: ADSM-L AT vm.marist DOT edu
Hi Zoltan,
It sounds like you are seeing multiple issues, best to deal with them one
at a time.
1) For the crash dump messages, it is possible that there is only one dump,
and the message just keeps repeating because the client detects the same
file over and over again. Check the corresponding dsmcrash.log file (start
at the bottom of that file) to see if crashes are constantly occurring, and
when they occurred. If the crashes are persistent, you should open a PMR
and upload the dump, so it can be reviewed. If it is a one-time crash, you
can do the same thing, or you can delete the dump file (dsmcrash.dmp),
which should eliminate the messages. If the problem reoccurs, open a PMR,
etc.
2) I am not sure what the hung sessions are. Sessions in IdleW state mean
that the session is not in transaction with the server. Most likely these
are consumer sessions waiting for work, possibly related to item 3, below.
3) You mentioned the backups run longer, and you observe "Process Dirs" as
the category with the most time in it. "Process Dirs" is the time the
client spends traversing the file system, searching for changed files. A
lengthy "Process Dirs" time is not necessarily a "problem" for file systems
with a large number, e.g., hundreds of thousands, or millions, of files;
and that number can be the largest if the number of changed files is very
small in comparison. This could also explain consumer sessions in IdleW
state (item 2, above).
Does the dsminstr.log file include data from when the client level was
7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the
earliest 8.1.0.2 metrics to see if there was truly some big difference
introduced at the time you upgraded the client. Off the top of my head, I
cannot think of anything in the client code itself that would suddenly
cause such a performance issue.
Without actual data to see what is going on (complete dsmsched.log file,
dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide
more specific suggestions.
Regards,
Andy
____________________________________________________________________________
Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
IBM Tivoli Storage Manager links:
Product support:
https://www.ibm.com/support/entry/portal/product/tivoli/
tivoli_storage_manager
Online documentation:
http://www.ibm.com/support/knowledgecenter/SSGSG7/
landing/welcome_ssgsg7.html
Product Wiki:
https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
20Storage%20Manager
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
10:36:15:
> From: Zoltan Forray <zforray AT VCU DOT EDU>
> To: ADSM-L AT VM.MARIST DOT EDU
> Date: 2017-10-06 10:37
> Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> We have this server that is used to backup DFS mounts/files. It has 30+
> TSM nodes configured to it.
>
> In the past, everything has been running like it should, albeit a little
> slow sometimes due to the size of some of the DFS mounts but the backups
> would usually finish.
>
> After upgrading the client to 8.1.0.2, we are experiencing constant
> problems. Many of the backups are not completing, some node sessions are
> running 10+ hours to examine 1000 files and backup 7MB in changes, etc.
>
> Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core
> file or crash report was found:" crash-dumps for most of the nodes, often
> numerous times within the same backup session.
>
> Instrumentation statistics on a few nodes I examined say the largest time
> consumer is "Process Dirs".
>
> From the SP server perspective, these nodes are in a constant "Idle
Wait".
> Session info shows lots of metadata send to the client, but very little
> coming from the client.
>
> The only change has been the SP client upgrade. Yes we have rebooted the
> server (Windows 2012) multiple times to clear the hung sessions
(sometimes
> running multiple days and never ending).
>
> Any way to determine is going on?
>
>
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit https://urldefense.proofpoint.com/v2/url?
> u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
>
nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG
srMocoLUGDXZU&e=
>
--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
|
