ADSM-L

Re: [ADSM-L] Extra client sessions

2016-09-07 09:14:14
Subject: Re: [ADSM-L] Extra client sessions
From: Karel Bos <tsm.wad AT GMAIL DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 7 Sep 2016 15:12:09 +0200
Keep in mind maxnummp is ignored when going to random access storage like
disk pool.

Op 7 sep. 2016 15:07 schreef "Zoltan Forray" <zforray AT vcu DOT edu>:

> Andy,
>
> No proxies in use for any backups.
>
> Here is their whole dsm.sys (Linux/Centos).  No, I do not know what their
> pre/post commands do...
>
> # more /opt/tivoli/tsm/client/ba/bin/dsm.sys
> SErvername  PROCESSOR
>   COMMmethod         TCPip
>   TCPPort            1500
>   TCPServeraddress   tsmlinux9.ucc.vcu.edu
>   TCPWindowsize      127
>   TCPBuffsize        32
>   TXNBytelimit       25600
>   TCPNODELAY         Yes
>   Inclexcl           /opt/tivoli/tsm/client/ba/bin/inclexcl.sys
>   SCHEDMODe          Prompted
>   SCHEDLOGname       /opt/tivoli/tsm/client/ba/bin/dsmsched.vcu-gs1.log
>   ErrorLogName       /opt/tivoli/tsm/client/ba/bin/dsmerror.vcu-gs1.log
>   SCHEDLOGRETENTION  14 D
>   ERRORLOGRETENTION  14 D
>   PasswordAccess     Generate
>   Compression        Yes
>
>   NodeName           vcu-gs1.chpc.vcu.edu
>
>   PreScheduleCmd "/opt/tivoli/tsm/client/ba/bin/tsm_checker.sh --pre"
>   PostScheduleCmd "/opt/tivoli/tsm/client/ba/bin/tsm_checker.sh --post"
>
>   resourceutilization 5
>
>
> And here is a q node f=d
>
> 9:01:39 AM   PROCESSOR : q node vcu-gs1.chpc.vcu.edu f=d
>                         Node Name: VCU-GS1.CHPC.VCU.EDU
>                          Platform: Linux x86-64
>                   Client OS Level: 2.6.32-431.17.1.el6
>                    Client Version: Version 7, release 1, level 6.2
>                Policy Domain Name: CHPC
>             Last Access Date/Time: 09/04/2016 13:06:21
>            Days Since Last Access: 3
>            Password Set Date/Time: 09/04/2016 13:06:17
>           Days Since Password Set: 3
>             Invalid Sign-on Count: 0
>                           Locked?: No
>                           Contact: Carlisle Childress
>                       Compression: Yes
>           Archive Delete Allowed?: Yes
>            Backup Delete Allowed?: No
>            Registration Date/Time: 06/09/2014 15:17:09
>         Registering Administrator: ZFORRAY
>    Last Communication Method Used: Tcp/Ip
>       Bytes Received Last Session: 9,796,989.78 M
>           Bytes Sent Last Session: 712.69 M
>          Duration of Last Session: 444,134.42
>       Pct. Idle Wait Last Session: 0.74
>      Pct. Comm. Wait Last Session: 41.48
>      Pct. Media Wait Last Session: 0.00
>                         Optionset:
>                               URL:
>                         Node Type: Client
>        Password Expiration Period:
>                 Keep Mount Point?: No
>      Maximum Mount Points Allowed: 1
>            Auto Filespace Rename : No
>                 Validate Protocol: No
>                       TCP/IP Name: godel21
>                    TCP/IP Address: 192.168.160.21
>                Globally Unique ID:
> 6f.1c.6b.7e.f3.16.11.e3.a7.3b.00.23.8b.17.2-
>                                     0.17
>             Transaction Group Max: 0
>                   Data Write Path: ANY
>                    Data Read Path: ANY
>                Session Initiation: ClientOrServer
>                High-level Address:
>                 Low-level Address:
>            Collocation Group Name:
>                  Proxynode Target:
>                   Proxynode Agent:
>                       Node Groups:
>                     Email Address:
>                     Deduplication: ServerOnly
>          Users allowed to back up: All
>                              Role: Server
>                     Role Override: UseReported
>                  Processor Vendor:
>                   Processor Brand:
>                    Processor Type:
>                   Processor Model:
>                   Processor Count: 0
>                        Hypervisor:
>                   API Application: No
>                        Scan Error: No
>                       MAC Address:
>                 Replication State: None
>                  Replication Mode: None
>           Backup Replication Rule: DEFAULT
>          Archive Replication Rule: DEFAULT
> Space Management Replication Rule: DEFAULT
>                    Client OS Name: LNX:CentOS release 6.5 (Final)
>     Client Processor Architecture: x64
>         Client Products Installed: BA
>             Client Target Version: (?)
>                    Authentication: Local
>                      SSL Required: No
>
>
> On Wed, Sep 7, 2016 at 8:56 AM, Andrew Raibeck <storman AT us.ibm DOT com> 
> wrote:
>
> > Hi Zoltan,
> >
> > Is your user doing proxied sessions? If yes, where is the MAXNUMMP
> > configured: on the agent node (NODENAME aaaaa)? Or the target node
> > (ASNODENAME bbbbb)? You need to ensure that the target node name has the
> > limited MAXNUMMP when proxied backups are performed.
> >
> > Best regards,
> >
> > Andy
> >
> > ____________________________________________________________
> > ________________
> >
> > Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT com
> >
> > IBM Tivoli Storage Manager links:
> > Product support:
> > https://www.ibm.com/support/entry/portal/product/tivoli/
> > tivoli_storage_manager
> >
> > Online documentation:
> > http://www.ibm.com/support/knowledgecenter/SSGSG7/
> > landing/welcome_ssgsg7.html
> >
> > Product Wiki:
> > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> > 20Storage%20Manager
> >
> > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2016-09-07
> > 08:19:42:
> >
> > > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > > To: ADSM-L AT VM.MARIST DOT EDU
> > > Date: 2016-09-07 08:29
> > > Subject: Re: Extra client sessions
> > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > >
> > > Hi Andy,
> > >
> > > But my point is that the user is getting whatever they set for
> > > RESOURCEUTILIZATION vs what I set for MAXNUMPOINT, without the noise.
> > This
> > > user is running 4-multi terabyte backup streams since they set
> > > RESOURCEUTILIZATION 5 while I still have MAXNUMPOINT set to 1/default.
> > >
> > > The users aren't supposed to be able to override the server
> > > settings/maximums. It would be anarchy!
> > >
> > > On Tue, Sep 6, 2016 at 7:10 PM, Andrew Raibeck <storman AT us.ibm DOT com>
> > wrote:
> > >
> > > > Hi Zoltan,
> > > >
> > > > Yes, if MAXNUMMP is too low to accommodate a non-default
> > > > RESOURCEUTILIZATION value, then while the backup should work okay,
> you
> > may
> > > > see some warning messages about there being insufficient mount
> points.
> > The
> > > > client will continue, albeit with fewer sessions. This is why I
> suggest
> > > > making sure that if you have a higher RESOURCEUTILIZATION setting,
> you
> > have
> > > > the MAXNUMMP to avoid the "noise", and make sure that mutual
> > expectations
> > > > are met (the server is configured to deliver what the client
> requests).
> > > > Regardless, though, you are right, in the end it's not fatal to the
> > > > operation if the settings are mismatched.
> > > >
> > > > Best regards,
> > > >
> > > > Andy
> > > >
> > > > ____________________________________________________________
> > > > ________________
> > > >
> > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | storman AT us.ibm DOT 
> > > > com
> > > >
> > > > IBM Tivoli Storage Manager links:
> > > > Product support:
> > > > https://www.ibm.com/support/entry/portal/product/tivoli/
> > > > tivoli_storage_manager
> > > >
> > > > Online documentation:
> > > > http://www.ibm.com/support/knowledgecenter/SSGSG7/
> > > > landing/welcome_ssgsg7.html
> > > >
> > > > Product Wiki:
> > > > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> > > > 20Storage%20Manager
> > > >
> > > > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 
> > > > 2016-09-01
> > > > 08:40:33:
> > > >
> > > > > From: Zoltan Forray <zforray AT VCU DOT EDU>
> > > > > To: ADSM-L AT VM.MARIST DOT EDU
> > > > > Date: 2016-09-01 08:43
> > > > > Subject: Re: Extra client sessions
> > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > > > >
> > > > > Thanks for the info.  Yes the user does(did) have
> RESOURCEUTILIZATION
> > 4
> > > > > configured.
> > > > >
> > > > > I note the APAR you refer to is still open. It refers to v7.1 but
> how
> > far
> > > > > back does it go?  The client recently upgrade all of his nodes to
> > > > 7.1.6.2,
> > > > > the latest available for Linux - not sure what level he was at
> when I
> > > > first
> > > > > saw this issue.
> > > > >
> > > > > As I said, I always though if MAXNUMPOINTS was set to 1 (the
> > default),
> > > > then
> > > > > what you specified for RESOURCEUTILZATION was ignored and you were
> > only
> > > > > supposed to get 2-sessions?  Am I wrong in this assumption?
> > > > >
> > > > > On Wed, Aug 31, 2016 at 5:36 PM, Andrew Raibeck <
> storman AT us.ibm DOT com>
> > > > wrote:
> > > > >
> > > > > > Yes, do not use a RESOURCEUTILIZATION higher than the MAXNUMMP
> > setting.
> > > > > >
> > > > > > Having said that, there is an APAR that might ("might" is the
> > operative
> > > > > > word!) be a match for this issue, IT16004:
> > > > > >
> > > > > > https://www.ibm.com/support/docview.wss?uid=swg1IT16004
> > > > > >
> > > > > > In this case, the symptom is seeing more consumer sessions than
> you
> > > > would
> > > > > > expect given the RESOURCEUTILIZATION setting. Even if the
> specific
> > > > symptoms
> > > > > > described in the APAR do not match your scenario, if no other
> > logical
> > > > > > explanation fits, it might stil be a match. You can contact
> support
> > for
> > > > > > further problem determination assistance.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Andy
> > > > > >
> > > > > > ____________________________________________________________
> > > > > > ________________
> > > > > >
> > > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 |
> storman AT us.ibm DOT com
> > > > > >
> > > > > > IBM Tivoli Storage Manager links:
> > > > > > Product support:
> > > > > > https://www.ibm.com/support/entry/portal/product/tivoli/
> > > > > > tivoli_storage_manager
> > > > > >
> > > > > > Online documentation:
> > > > > > http://www.ibm.com/support/knowledgecenter/SSGSG7/
> > > > > > landing/welcome_ssgsg7.html
> > > > > >
> > > > > > Product Wiki:
> > > > > >
> > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> > > > > > 20Storage%20Manager
> > > > > >
> > > > > > "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on
> > 2016-08-31
> > > > > > 17:22:19:
> > > > > >
> > > > > > > From: Karel Bos <tsm.wad AT GMAIL DOT COM>
> > > > > > > To: ADSM-L AT VM.MARIST DOT EDU
> > > > > > > Date: 2016-08-31 17:23
> > > > > > > Subject: Re: Extra client sessions
> > > > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > > > > > >
> > > > > > > Might want to check resourceutil settings as that limits the
> > number
> > > > of
> > > > > > > sessions clients try to setup. It should match maxnummp or be
> > lower.
> > > > > > >
> > > > > > > Op 31 aug. 2016 22:21 schreef "Zoltan Forray" <zforray AT vcu DOT 
> > > > > > > edu
> >:
> > > > > > >
> > > > > > > > AHA - so I am not loosing my mind (at least in this
> situation).
> > I
> > > > too
> > > > > > have
> > > > > > > > been seeing clients getting >3-sessions eventhough the NODE
> > > > > > maxnumpoints is
> > > > > > > > 1!  I was always under the impression that maxnumpoints
> trumps
> > > > > > > > resourceutilization.
> > > > > > > >
> > > > > > > > On Wed, Aug 31, 2016 at 3:40 PM, Thomas Denier <
> > > > > > > > Thomas.Denier AT jefferson DOT edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We are occasionally seeing some odd behavior in our TSM
> > > > environment.
> > > > > > > > >
> > > > > > > > > We write incoming client files to sequential disk storage
> > pools.
> > > > > > Almost
> > > > > > > > > all of our client nodes use the default maxnummp value of
> 1.
> > > > > > > > >
> > > > > > > > > When the odd behavior occurs, a number of clients will go
> > through
> > > > the
> > > > > > > > > following sequence of events:
> > > > > > > > > 1.The TSM server will send a request to start a backup.
> > > > > > > > > 2.The client will almost immediately open a TCP connection
> to
> > be
> > > > used
> > > > > > as
> > > > > > > > a
> > > > > > > > > producer session (a session used to obtain information from
> > the
> > > > TSM
> > > > > > > > > database).
> > > > > > > > > 3.Somewhere between tens of seconds and a few minutes later
> > the
> > > > > > client
> > > > > > > > > will open a TCP connection to be used as a consumer session
> > (a
> > > > > > session
> > > > > > > > used
> > > > > > > > > to send copies of new and changed files).
> > > > > > > > > 4.Sometime later the client will open a third TCP
> connection
> > and
> > > > > > start
> > > > > > > > > using it as a consumer session.
> > > > > > > > > 5.The TSM server will report large numbers of transaction
> > > > failures
> > > > > > > > because
> > > > > > > > > it considers the original consumer session to be tying up
> the
> > one
> > > > > > mount
> > > > > > > > > point allowed for the node and hence has no way of storing
> > files
> > > > > > arriving
> > > > > > > > > on the new consumer session.
> > > > > > > > >
> > > > > > > > > In most cases, all of the affected clients will hit step
> four
> > > > within
> > > > > > an
> > > > > > > > > interval of a couple of minutes.
> > > > > > > > >
> > > > > > > > > My current theory is that step four occurs when the client
> > system
> > > > > > detects
> > > > > > > > > a condition that is viewed as a fatal error in the original
> > > > consumer
> > > > > > > > > session, triggering the opening of a replacement consumer
> > > > session. In
> > > > > > > > most
> > > > > > > > > cases the TSM server never detects a problem with the
> > original
> > > > > > consumer
> > > > > > > > > session, and eventually terminates the session after five
> > hours
> > > > of
> > > > > > > > > inactivity (we have database backups that can legitimately
> go
> > > > through
> > > > > > > > long
> > > > > > > > > periods with no data transfer). More rarely the TSM server
> > > > eventually
> > > > > > > > > reports that the original consumer session was severed.
> > > > > > > > >
> > > > > > > > > We occasionally see cases where the replacement consumer
> > session
> > > > is
> > > > > > in
> > > > > > > > > turn replaced by another new session, and even cases where
> > the
> > > > latter
> > > > > > > > > session is replaced by yet another session.
> > > > > > > > >
> > > > > > > > > Our client population is a bit over half Windows, but
> almost
> > all
> > > > > > > > instances
> > > > > > > > > of the odd behavior involve only Windows client systems.
> > > > > > > > >
> > > > > > > > > The affected systems are frequently split between two data
> > > > centers,
> > > > > > each
> > > > > > > > > with its own TSM server.
> > > > > > > > >
> > > > > > > > > We have usually not found any correlation between the odd
> TSM
> > > > > > behavior
> > > > > > > > and
> > > > > > > > > issues with other applications. The most recent case was an
> > > > > > exception.
> > > > > > > > > There were some e-mail delivery failures at about the same
> > time
> > > > as
> > > > > > step
> > > > > > > > > four of the odd TSM behavior. The failures occurred when
> > e-mail
> > > > > > servers
> > > > > > > > > were unable to perform LDAP queries.
> > > > > > > > >
> > > > > > > > > When we have asked our Network Operations group to check on
> > > > previous
> > > > > > > > > occurrences of the odd behavior they have consistently
> > reported
> > > > that
> > > > > > they
> > > > > > > > > found no evidence of a network problem.
> > > > > > > > >
> > > > > > > > > Each of our TSM servers runs under zSeries Linux on a z10
> BC.
> > > > Each
> > > > > > server
> > > > > > > > > has a VIPA address with two associated network interfaces
> on
> > > > > > different
> > > > > > > > > subnets.
> > > > > > > > >
> > > > > > > > > I would welcome any suggestions for finding the underlying
> > cause
> > > > of
> > > > > > the
> > > > > > > > > odd behavior.
> > > > > > > > >
> > > > > > > > > Thomas Denier,
> > > > > > > > > Thomas Jefferson University
> > > > > > > > > The information contained in this transmission contains
> > > > privileged
> > > > > > and
> > > > > > > > > confidential information. It is intended only for the use
> of
> > the
> > > > > > person
> > > > > > > > > named above. If you are not the intended recipient, you are
> > > > hereby
> > > > > > > > notified
> > > > > > > > > that any review, dissemination, distribution or duplication
> > of
> > > > this
> > > > > > > > > communication is strictly prohibited. If you are not the
> > intended
> > > > > > > > > recipient, please contact the sender by reply email and
> > destroy
> > > > all
> > > > > > > > copies
> > > > > > > > > of the original message.
> > > > > > > > >
> > > > > > > > > CAUTION: Intended recipients should NOT use email
> > communication
> > > > for
> > > > > > > > > emergent or urgent health care matters.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Zoltan Forray*
> > > > > > > > TSM Software & Hardware Administrator
> > > > > > > > Xymon Monitor Administrator
> > > > > > > > VMware Administrator (in training)
> > > > > > > > Virginia Commonwealth University
> > > > > > > > UCC/Office of Technology Services
> > > > > > > > www.ucc.vcu.edu
> > > > > > > > zforray AT vcu DOT edu - 804-828-4807
> > > > > > > > Don't be a phishing victim - VCU and other reputable
> > organizations
> > > > will
> > > > > > > > never use email to request that you reply with your password,
> > > > social
> > > > > > > > security number or confidential personal information. For
> more
> > > > details
> > > > > > > > visit http://infosecurity.vcu.edu/phishing.html
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Zoltan Forray*
> > > > > TSM Software & Hardware Administrator
> > > > > Xymon Monitor Administrator
> > > > > VMware Administrator (in training)
> > > > > Virginia Commonwealth University
> > > > > UCC/Office of Technology Services
> > > > > www.ucc.vcu.edu
> > > > > zforray AT vcu DOT edu - 804-828-4807
> > > > > Don't be a phishing victim - VCU and other reputable organizations
> > will
> > > > > never use email to request that you reply with your password,
> social
> > > > > security number or confidential personal information. For more
> > details
> > > > > visit http://infosecurity.vcu.edu/phishing.html
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > TSM Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator (in training)
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zforray AT vcu DOT edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more details
> > > visit http://infosecurity.vcu.edu/phishing.html
> > >
> >
>
>
>
> --
> *Zoltan Forray*
> TSM Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator (in training)
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://infosecurity.vcu.edu/phishing.html
>