TSM Server crashed after running out of Database Space

[javajockey]

Hello all,

My TSM server is 6.1.4.2 on Aix 6.1
Storage pools reside on Sepaton VTL
NO TSM Dedup is used.
My current DB size is 921GB after adding 110G Sunday night
The DB,LOG and Archlog resides on VMAX.

My TSM server has started to become unstable. The DB grew 80GB in less than one hour on Sunday night. I presume this could be the DB2 Reorg process using temp space. The server Crashed at Sunday around 19:00 and I added 110GB after I brought the TSM server back up
I had to move some storage pools around and allocate /stgsysbk1 (stgsysbk1 is also on VMAX) to prevent the server from crashing again. I now have 170GB free for a total of 921G


This is what my current dpspace looks like

/stgsysbk1                         102,400.00                         16.00
/tsm/db/JGB/db9                    102,400.00                         102,399.84
/tsm/db/JGB/db7                    204,800.00                         204,799.96
/tsm/db/JGB/db5                    20,480.00                          20,479.77
/tsm/db/JGB/db4                    20,480.00                          20,479.80
/tsm/db/JGB/db3                    20,480.00                          20,479.79
/tsm/db/JGB/db2                    20,480.00                          20,479.77
/tsm/db/JGB/db1                    20,480.00                          20,479.95
/tsm/db/JGB/db6                    102,400.00                         102,399.92
/tsm/db/JGB/db8                    102,400.00                         102,399.86
/tsm/db/JGB/db10                   102,400.00                         65,329.14
/tsm/db/JGB/db11                   102,400.00                         65,329.18

For some reason, even after I allocated the additional space, My server crashed again on Monday night (24 hours later). I have not received any alerts like I previously did about the DB running low on space. The system just crashed. I opened a PMR with IBM and was informed that DB space will not be available until TSM Server restarts. Can anybody else confirm this? I have requested a second option from IBM because I can't believe that DB2 must be restarted in order to allocate additional space.

Has anybody else experienced unusual Fast DB growth like this (100GB in one hour)

 

[moon-buddy]

The position or comment if IBM is correct.

See 'help extend dbspace' - the help has a very good explanation what TSM does with the new space.

 

[javajockey]

I'm not seeing where a reboot is required. Unless IBM updated the help files in later versions of TSM.

help extend db3.22 EXTEND DBSPACE (Increase space for the database)


Use this command to increase space for the database by adding
directories for the database to use.






Privilege class


To issue this command, you must have system privilege or unrestricted
storage privilege.






Syntax


                   .-,------------.
                   V              |
>>-EXTend DBSpace----db_directory-+----------------------------><








Parameters


db_directory (Required)
   Specifies the directories for database storage. The directories must
   be empty and accessible by the user ID of the database manager. A
   directory name must be a fully qualified name. The maximum length of
   the name is 175 characters. Enclose the name in quotation marks if it
   contains embedded blanks, an equal sign, or other special characters.






Example: Add directories to the storage space for the database


Add two directories (/tsm_db/stg1 and tsm_db/stg2) under the /tsm_db
directory to the storage space for the database. Issue the command:


extend dbspace /tsm_db/stg1,/tsm_db/stg2






Related commands


Table 148. Commands related to EXTEND DBSPACE
+----------------------------------+----------------------------------+
| Command                          | Description                      |
+----------------------------------+----------------------------------+
| QUERY DB                         | Displays allocation information  |
|                                  | about the database.              |
+----------------------------------+----------------------------------+
| QUERY DBSPACE                    | Displays information about the   |
|                                  | storage space defined for the    |
|                                  | database.                        |
+----------------------------------+----------------------------------+

 

[moon-buddy]

I'm not seeing where a reboot is required. Unless IBM updated the help files in later versions of TSM.

help extend db3.22 EXTEND DBSPACE (Increase space for the database)


Use this command to increase space for the database by adding
directories for the database to use.






Privilege class


To issue this command, you must have system privilege or unrestricted
storage privilege.






Syntax


                   .-,------------.
                   V              |
>>-EXTend DBSpace----db_directory-+----------------------------><








Parameters


db_directory (Required)
   Specifies the directories for database storage. The directories must
   be empty and accessible by the user ID of the database manager. A
   directory name must be a fully qualified name. The maximum length of
   the name is 175 characters. Enclose the name in quotation marks if it
   contains embedded blanks, an equal sign, or other special characters.






Example: Add directories to the storage space for the database


Add two directories (/tsm_db/stg1 and tsm_db/stg2) under the /tsm_db
directory to the storage space for the database. Issue the command:


extend dbspace /tsm_db/stg1,/tsm_db/stg2






Related commands


Table 148. Commands related to EXTEND DBSPACE
+----------------------------------+----------------------------------+
| Command                          | Description                      |
+----------------------------------+----------------------------------+
| QUERY DB                         | Displays allocation information  |
|                                  | about the database.              |
+----------------------------------+----------------------------------+
| QUERY DBSPACE                    | Displays information about the   |
|                                  | storage space defined for the    |
|                                  | database.                        |
+----------------------------------+----------------------------------+

Ok, your version is lower than mine. I believe this issue was identified sometime after your version was released. I run version 6.1.5.104. TSM 6.2 (If I remember right) use DB2 9.7 which manages the database a little different. TSM 6.1 use DB2 9.5

3.22 EXTEND DBSPACE (Increase space for the database)

Use this command to increase space for the database by adding
directories for the database to use.



Privilege class

To issue this command, you must have system privilege or unrestricted
storage privilege.



After a new directory is added to a Tivoli Storage Manager server, the
directory might not be fully used immediately. Some Tivoli Storage
Manager events can cause the added database space to become increasingly
used over time. For example, table reorganizations or some temporary
database transactions, such as long running select statements, can cause
the added database space to begin filling up. The database space
redistribution among all directories can require a few days or even
weeks. If the existing database directories are nearly full when the new
directory is added, the server might encounter an out of space condition
(reported in the db2diag.log).

If this condition occurs, halt and restart the server. If the restart
does not correct the condition, remove the database and then restore it
to the same or new directories.



Syntax

                   .-,------------.
                   V              |
>>-EXTend DBSpace----db_directory-+----------------------------><

 

[javajockey]

I did find an APAR that did indeed mention the space may take some time to become available. I don't see how restarting TSM will then add the space. Restoring the DB would in fact add the space. It's pretty scary if space you add to the DB isn't available 24 hours after you add using the "extend db" command.

 

[tsm_dude]

Since your TSM server is running on AIX, why not extend the existing filesystems used for the DB ?
DB2 will write to the filesystem with the most free space, so you extend all existing filesystems the same amount if possible and your done. The TSM6 with DB2 uses system Managed Storage for the DB so it will just create addtional containers automatically.

 

[javajockey]

Since your TSM server is running on AIX, why not extend the existing filesystems used for the DB ?
DB2 will write to the filesystem with the most free space, so you extend all existing filesystems the same amount if possible and your done. The TSM6 with DB2 uses system Managed Storage for the DB so it will just create addtional containers automatically.

Sadly, I don't have Root Access on my servers, only the instance owner of TSM. I also have to coordinate additional storage w/ my SAN team.