ADSM-L

Re: client session stopps with 'no space available in storage... and all successor pools'

2006-08-30 08:31:48
Subject: Re: client session stopps with 'no space available in storage... and all successor pools'
From: Rainer Wolf <rainer.wolf AT UNI-ULM DOT DE>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 30 Aug 2006 14:20:38 +0200
David,
the second is of type file and the mountlimit is set to 20
in the file-devclass.
The third and last Pool is a tapepool with a maximum of 4 drives available .

From the help of...

UPDATE DEVCLASS -- FILE
...
MOUNTLimit
     Specifies the maximum number of files that can be simultaneously open
     for input/output. This parameter is optional. You can specify a number
     from 1 to 4096.

     If you plan to use the simultaneous write function, ensure that
     sufficient drives are available for the write operation. If the number
     of drives needed for a simultaneous write operation is greater than the
     value of the MOUNTLIMIT parameter for a device class, the transaction
     will fail. For details about the simultaneous write function, refer to
     the Administrator's Guide.
...
... I don't understand whats about the 'drives' mentioned.
So I'm confused now if I should increase the mountlimit to eg 40 ?
or better decrease  ? ... to the number of the maximum available drives of the
tape-destination that comes after the file-pool ?

Cheers
Rainer

David le Blanc schrieb:
I believe this can happen at 5.3 clients....

Are any of your pools (in the chain of pools the client writes to) of
type FILE ?

Try increasing the number of mount points for the device class for that
pool.




-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU]
On Behalf Of Rainer Wolf
Sent: Wednesday, 30 August 2006 7:34 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] client session stopps with 'no space
available in storage... and all successor pools'

Arnaud,
no thats not hurting cause
we have no cache enabled on the disk storage pools ...
checked the other things
of the tech-note and still nothing applies to this problem.
Especially there are available scratch tapes and the number
of the available tapes
in the tapepool is high enough.
think I look through the 'fixed things' of 5.3 Client
... it seems to only happen at the 5.2 Clients .

Cheers
Rainer


PAC Brion Arnaud schrieb:


Rainer,

Just found this technote :
http://www-1.ibm.com/support/docview.wss?uid=swg21079391

which refers to

ANS1311E and ANR0522W  problems, and states this possible reason :

- Cache is enabled on the disk storage pool that the TSM

Client backup

data is being sent to, and cached data cannot be deleted

quickly enough

to allow the backup data to be written to the disk storage

pool. In this

case, update the storage pool so that cache is not enabled.

Couldn't this be hurting you ?

Cheers


Arnaud



**************************************************************
**********

******
Panalpina Management Ltd., Basle, Switzerland,
CIT Department Viadukstrasse 42, P.O. Box 4002 Basel/CH
Phone:  +41 (61) 226 11 11, FAX: +41 (61) 226 17 01
Direct: +41 (61) 226 19 78
e-mail: arnaud.brion AT panalpina DOT com


**************************************************************
**********

******

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU]

On Behalf Of

Rainer Wolf
Sent: Wednesday, 30 August, 2006 09:31
To: ADSM-L AT VM.MARIST DOT EDU
Subject: client session stopps with 'no space available in

storage...

and all successor pools'

Dear TSmers,

this happens on tsm server 5.3.3.2 / solaris ,3494

and Clients: linux86 5.2.3.1 , linux86 5.2.3.0 , solaris 5.2.5.0 ,
solaris 5.2.2.6 , winnt 5.2.3.11

we have a strange problem with occasionly stopped client

sessions with

the message 'no space available in storage pool BACKUPPOOL and all
successor pools' .
If this happens it happens withe clients running bigger transfers in
time and data - mostly on initial backups.
The data flow is set up as
random access disk pool -->  sequential file pool -->

sequential tape

pool

It may happen that the first 2 stages are going to be full but the
tapepool always has free and usable scratch volumes available.

The question is : is this a bug at the server or do I have to change
something in the setups of the pools ?
The space in the random-access pools are normally migrating down to
about 50 % -- is it better to bring this down to 0% Usage as a daily
task ?

I thought that sessions that don't have enough space in the
backup/filepools would directly write to tape if it is needed.
But if this stopping happens it seems to be just happening

on that long

and large running sessions starting to write on backuppool and then
switching to filepool ... it seems to be that there is no second
switching on the tapepool possible ?

I just checked the client-versions of all nodes where this

happens and

all of them have 5.2.X.X ... so is it just a client-problem

with the old

5.2.X.X clients ?

Thanks a lot in advance for any hints !
Rainer


tsm: TSM1>q actlog begint=-20 search=94090


Date/Time                Message
--------------------
----------------------------------------------------------
08/29/06   08:13:01      ANR0406I Session 94090 started for node
ULLI187.CHEMIE
                          (Linux86) (Tcp/Ip
134.60.42.187(1039)).(SESSION: 94090)
08/29/06   20:17:08      ANR8340I FILE volume
/tsmdata3/tsm1/file8/00006B4D.BFS
                          mounted.(SESSION: 94090)
08/29/06   20:17:08      ANR0511I Session 94090 opened output volume


/tsmdata3/tsm1/file8/00006B4D.BFS.(SESSION:

94090)
08/29/06   20:17:24      ANR8341I End-of-volume reached for

FILE volume



/tsmdata3/tsm1/file8/00006B4D.BFS.(SESSION:

94090)
08/29/06   20:17:24      ANR0514I Session 94090 closed volume


/tsmdata3/tsm1/file8/00006B4D.BFS.(SESSION:

94090)
08/29/06   20:17:24      ANR0522W Transaction failed for

session 94090

for node
                          ULLI187.CHEMIE (Linux86) - no

space available

in storage
                          pool BACKUPPOOL8 and all successor
pools.(SESSION: 94090)
08/29/06   20:17:53      ANR0403I Session 94090 ended for node
ULLI187.CHEMIE
                          (Linux86).(SESSION: 94090)



tsm: TSM1>q actlog search=94086 begind=-2

Date/Time                Message
--------------------
----------------------------------------------------------
08/29/06   08:10:22      ANR0406I Session 94086 started for node
ULLI187.CHEMIE
                          (Linux86) (Tcp/Ip
134.60.42.187(1038)).(SESSION: 94086)
08/29/06   20:17:54      ANE4952I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects inspected:
1,458,833(SESSION: 94086)
08/29/06   20:17:54      ANE4954I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects backed up:
1,457,166(SESSION: 94086)
08/29/06   20:17:54      ANE4958I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects updated:
0(SESSION: 94086)
08/29/06   20:17:54      ANE4960I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects rebound:
0(SESSION: 94086)
08/29/06   20:17:54      ANE4957I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects deleted:
0(SESSION: 94086)
08/29/06   20:17:54      ANE4970I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects expired:
0(SESSION: 94086)
08/29/06   20:17:54      ANE4959I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of objects failed:
1(SESSION: 94086)
08/29/06   20:17:54      ANE4961I (Session: 94086, Node:

ULLI187.CHEMIE)

Total
                          number of bytes transferred: 245.28
GB(SESSION: 94086)
08/29/06   20:17:54      ANE4963I (Session: 94086, Node:

ULLI187.CHEMIE)

Data
                          transfer time:                112,550.97
sec(SESSION:
                          94086)
08/29/06   20:17:54      ANE4966I (Session: 94086, Node:

ULLI187.CHEMIE)

Network
                          data transfer rate:        2,285.17
KB/sec(SESSION:
                          94086)
08/29/06   20:17:54      ANE4967I (Session: 94086, Node:

ULLI187.CHEMIE)

Aggregate
                          data transfer rate:      5,913.70
KB/sec(SESSION: 94086)
08/29/06   20:17:54      ANE4968I (Session: 94086, Node:

ULLI187.CHEMIE)

Objects
                          compressed by:

0%(SESSION:

94086)
08/29/06   20:17:54      ANE4964I (Session: 94086, Node:

ULLI187.CHEMIE)

Elapsed
                          processing time:

12:04:52(SESSION:

94086)
08/29/06   20:48:39      ANR0482W Session 94086 for node

ULLI187.CHEMIE

(Linux86)
                          terminated - idle for more than 30
minutes.(SESSION:
                          94086)

The storage pools look like:

Storage         Device          Estimated       Pct

Pct     High

Low     Next Stora-
Pool Name       Class Name       Capacity      Util

Migr      Mig

Mig     ge Pool


       Pct

Pct
-----------     ----------     ----------     -----

-----     ----

---     -----------
BACKUPPOOL8     DISK                160 G      63.9

56.2       90

50     FILEPOOL8
FILEPOOL8       FILE8               405 G      76.2

77.5       90

70     TAPE_O262
TAPE_O262       3592             22,288 G      24.6

60.0      100

70



tsm: TSM1>q stg BACKUPPOOL8 f=d

               Storage Pool Name: BACKUPPOOL8
               Storage Pool Type: Primary
               Device Class Name: DISK
              Estimated Capacity: 160 G
              Space Trigger Util: 63.9
                        Pct Util: 63.9
                        Pct Migr: 56.2
                     Pct Logical: 99.4
                    High Mig Pct: 90
                     Low Mig Pct: 50
                 Migration Delay: 0
              Migration Continue: Yes
             Migration Processes: 1
           Reclamation Processes:
               Next Storage Pool: FILEPOOL8
            Reclaim Storage Pool:
          Maximum Size Threshold: No Limit
                          Access: Read/Write
                     Description:
               Overflow Location:
           Cache Migrated Files?: No
                      Collocate?:
           Reclamation Threshold:
       Offsite Reclamation Limit:
 Maximum Scratch Volumes Allowed:
  Number of Scratch Volumes Used:
   Delay Period for Volume Reuse:
          Migration in Progress?: No
            Amount Migrated (MB): 176,075.38 Elapsed Migration Time
(seconds): 16,556
        Reclamation in Progress?:
  Last Update by (administrator): xx
           Last Update Date/Time: 08/29/06   17:00:14
        Storage Pool Data Format: Native
            Copy Storage Pool(s):
         Continue Copy on Error?:
                        CRC Data: No
                Reclamation Type:


tsm: TSM1>

tsm: TSM1>q stg filePOOL8 f=d

               Storage Pool Name: FILEPOOL8
               Storage Pool Type: Primary
               Device Class Name: FILE8
              Estimated Capacity: 405 G
              Space Trigger Util: 98.4
                        Pct Util: 76.2
                        Pct Migr: 77.5
                     Pct Logical: 99.9
                    High Mig Pct: 90
                     Low Mig Pct: 70
                 Migration Delay: 0
              Migration Continue: Yes
             Migration Processes: 1
           Reclamation Processes: 1
               Next Storage Pool: TAPE_O262
            Reclaim Storage Pool:
          Maximum Size Threshold: No Limit
                          Access: Read/Write
                     Description:
               Overflow Location:
           Cache Migrated Files?:
                      Collocate?: Group
           Reclamation Threshold: 100
       Offsite Reclamation Limit:
 Maximum Scratch Volumes Allowed: 200
  Number of Scratch Volumes Used: 155
   Delay Period for Volume Reuse: 1 Day(s)
          Migration in Progress?: No
            Amount Migrated (MB): 91,059.84 Elapsed Migration Time
(seconds): 2,587
        Reclamation in Progress?: No
  Last Update by (administrator): xx
           Last Update Date/Time: 08/30/06   08:08:01
        Storage Pool Data Format: Native
            Copy Storage Pool(s):
         Continue Copy on Error?:
                        CRC Data: No
                Reclamation Type: Threshold


tsm: TSM1>q stg TAPE_O262 f=d

               Storage Pool Name: TAPE_O262
               Storage Pool Type: Primary
               Device Class Name: 3592
              Estimated Capacity: 22,288 G
              Space Trigger Util:
                        Pct Util: 24.6
                        Pct Migr: 60.0
                     Pct Logical: 97.1
                    High Mig Pct: 100
                     Low Mig Pct: 70
                 Migration Delay: 0
              Migration Continue: Yes
             Migration Processes: 1
           Reclamation Processes: 1
               Next Storage Pool:
            Reclaim Storage Pool:
          Maximum Size Threshold: No Limit
                          Access: Read/Write
                     Description:
               Overflow Location:
           Cache Migrated Files?:
                      Collocate?: Group
           Reclamation Threshold: 100
       Offsite Reclamation Limit:
 Maximum Scratch Volumes Allowed: 50
  Number of Scratch Volumes Used: 32
   Delay Period for Volume Reuse: 8 Day(s)
          Migration in Progress?: No
            Amount Migrated (MB): 0.00
Elapsed Migration Time (seconds): 0
        Reclamation in Progress?: No
  Last Update by (administrator): xx
           Last Update Date/Time: 08/29/06   16:33:34
        Storage Pool Data Format: Native
            Copy Storage Pool(s):
         Continue Copy on Error?:
                        CRC Data: No
                Reclamation Type: Threshold




--


--------------------------------------------------------------
----------

Rainer Wolf                          eMail:

rainer.wolf AT uni-ulm DOT de

kiz - Abt. Infrastruktur           Tel/Fax:      ++49 731

50-22482/22471

Universitaet Ulm                     wwweb:

http://kiz.uni-ulm.de


--
--------------------------------------------------------------
----------
Rainer Wolf                          eMail:
rainer.wolf AT uni-ulm DOT de
kiz - Abt. Infrastruktur           Tel/Fax:      ++49 731
50-22482/22471
Universitaet Ulm                     wwweb:
http://kiz.uni-ulm.de





--
------------------------------------------------------------------------
Rainer Wolf                          eMail:       rainer.wolf AT uni-ulm DOT de
kiz - Abt. Infrastruktur           Tel/Fax:      ++49 731 50-22482/22471
Universitaet Ulm                     wwweb:        http://kiz.uni-ulm.de