Oops - ignore first sentence in my reply - was sending that to the OS tech
responsible for this box.....
On Fri, Oct 6, 2017 at 11:48 AM, Zoltan Forray <zforray AT vcu DOT edu> wrote:
> Can you get me this file:
> ANS2250W A IBM Spectrum Protect core file or crash report was found:
> C:\Program Files\Tivoli\TSM\baclient\dsmcrash.dmp
>
> Hi Andy,
>
> Thanks for the reply. I have plenty of logs but obviously I can't upload
> them here. This includes the instrumentation data. The upgrade was done
> last Friday.
>
> I will pull down and remove the dsmcrash.dmp file.
>
> I pulled this from one of the dsminstr.log files. As you can see, the
> processing time has jumped a lot.
>
> Elapsed processing time: 00:00:35
> Elapsed processing time: 00:00:18
> Elapsed processing time: 00:00:52
> Elapsed processing time: 00:00:28
> Elapsed processing time: 00:00:34
> Elapsed processing time: 00:00:28
> Elapsed processing time: 00:01:05
> Elapsed processing time: 00:00:27
> Elapsed processing time: 00:02:52
> Elapsed processing time: 00:02:54
> Elapsed processing time: 00:02:36
> Elapsed processing time: 00:02:25
>
> This is for a small node with <1,400 objects.
>
> Total number of objects inspected: 1,321
> Total number of objects inspected: 1,324
> Total number of objects inspected: 1,327
> Total number of objects inspected: 1,331
> Total number of objects inspected: 1,332
> Total number of objects inspected: 1,339
> Total number of objects inspected: 1,339
> Total number of objects inspected: 1,339
> Total number of objects inspected: 1,340
> Total number of objects inspected: 1,342
> Total number of objects inspected: 1,344
> Total number of objects inspected: 1,348
>
>
> ---------- Forwarded message ----------
> From: Andrew Raibeck <storman AT us.ibm DOT com>
> Date: Fri, Oct 6, 2017 at 11:26 AM
> Subject: Re: [ADSM-L] Problems after upgrading client from 7.1.6.3 to
> 8.1.0.2
> To: ADSM-L AT vm.marist DOT edu
>
>
> Hi Zoltan,
>
> It sounds like you are seeing multiple issues, best to deal with them one
> at a time.
>
> 1) For the crash dump messages, it is possible that there is only one dump,
> and the message just keeps repeating because the client detects the same
> file over and over again. Check the corresponding dsmcrash.log file (start
> at the bottom of that file) to see if crashes are constantly occurring, and
> when they occurred. If the crashes are persistent, you should open a PMR
> and upload the dump, so it can be reviewed. If it is a one-time crash, you
> can do the same thing, or you can delete the dump file (dsmcrash.dmp),
> which should eliminate the messages. If the problem reoccurs, open a PMR,
> etc.
>
> 2) I am not sure what the hung sessions are. Sessions in IdleW state mean
> that the session is not in transaction with the server. Most likely these
> are consumer sessions waiting for work, possibly related to item 3, below.
>
> 3) You mentioned the backups run longer, and you observe "Process Dirs" as
> the category with the most time in it. "Process Dirs" is the time the
> client spends traversing the file system, searching for changed files. A
> lengthy "Process Dirs" time is not necessarily a "problem" for file systems
> with a large number, e.g., hundreds of thousands, or millions, of files;
> and that number can be the largest if the number of changed files is very
> small in comparison. This could also explain consumer sessions in IdleW
> state (item 2, above).
>
> Does the dsminstr.log file include data from when the client level was
> 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with the
> earliest 8.1.0.2 metrics to see if there was truly some big difference
> introduced at the time you upgraded the client. Off the top of my head, I
> cannot think of anything in the client code itself that would suddenly
> cause such a performance issue.
>
> Without actual data to see what is going on (complete dsmsched.log file,
> dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to provide
> more specific suggestions.
>
> Regards,
>
> Andy
>
> ____________________________________________________________
> ________________
>
> Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
>
> IBM Tivoli Storage Manager links:
> Product support:
> https://www.ibm.com/support/entry/portal/product/tivoli/tivo
> li_storage_manager
>
> Online documentation:
> http://www.ibm.com/support/knowledgecenter/SSGSG7/landing/
> welcome_ssgsg7.html
>
> Product Wiki:
> https://www.ibm.com/developerworks/community/wikis/home/
> wiki/Tivoli%20Storage%20Manager
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2017-10-06
> 10:36:15:
>
> > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 2017-10-06 10:37
> > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > We have this server that is used to backup DFS mounts/files. It has 30+
> > TSM nodes configured to it.
> >
> > In the past, everything has been running like it should, albeit a little
> > slow sometimes due to the size of some of the DFS mounts but the backups
> > would usually finish.
> >
> > After upgrading the client to 8.1.0.2, we are experiencing constant
> > problems. Many of the backups are not completing, some node sessions are
> > running 10+ hours to examine 1000 files and backup 7MB in changes, etc.
> >
> > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect core
> > file or crash report was found:" crash-dumps for most of the nodes, often
> > numerous times within the same backup session.
> >
> > Instrumentation statistics on a few nodes I examined say the largest time
> > consumer is "Process Dirs".
> >
> > From the SP server perspective, these nodes are in a constant "Idle
> Wait".
> > Session info shows lots of metadata send to the client, but very little
> > coming from the client.
> >
> > The only change has been the SP client upgrade. Yes we have rebooted the
> > server (Windows 2012) multiple times to clear the hung sessions
> (sometimes
> > running multiple days and never ending).
> >
> > Any way to determine is going on?
> >
> >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray AT vcu DOT edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit https://urldefense.proofpoint.com/v2/url?
> > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38-
> >
> nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35Cf
> ELFHGsrMocoLUGDXZU&e=
>
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807 <(804)%20828-4807>
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/
>
--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray AT vcu DOT edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
|
