TSM DB grows rapidly, maybe activity log problem ?

[polvdv]

The last weeks my TSM DB grows with almost 0,5 % a day

First I have checked all my clients to see if there is an abnormal growth with daily number of files backed up (using TSMManager)
but I didn't find anything unusual

Then I checked with ' Q Stat' and I think there is a problem with my activity log, this is the result

Activity Log Retention: 15 Day(s)
Activity Log Number of Records: 18446744073709519231
Activity Log Size: -1 M

The size shows ' -1 M ' but I do not think the number of records is quite realistic

TSM server version 5.3.2.0 (on Z/OS 1.7)
TSM DB = 18 GB (88 % utilized)

Thanks for any help;

Paul

 

[steinweo]

Hi,

Is there any ANR9999D in your activity log? It might indicate a problem with the database. Otherwise, did you check if the expiration is still running without any problem every day? If not it should explain why database is growing.

Regards, Olivier.

 

[polvdv]

Hi Olivier,

Expiration is running succesfull every day

There was an ANR9999D on 26/03, but DB was already growing since beginning of March, and this ANR9999D is caused by canceling a Space reclamation process

26.03.2007 05:06:07 ANR9999D BFUTIL(5311): ThreadId<15002> Error 18 in moving
cached transfer list with not NULL transaction. (PROCESS:
58)
26.03.2007 05:06:07 ANR9999D ThreadId<15002> issued message 9999 from: <-
StdPutText+4A2<- outDiagf+28C<- TransferBitFileList+732<-
BfTransferBitFile+138<- AfHandleOffsiteFile+2E6<-
RclmOffsiteVols+AC2<- AfRclmVolumeThread+2FC<-
pkThreadHead+502 (PROCESS: 58)

Paul

 

[slapelu]

Are these UNIX systems? If so you need to check to make sure your systems do not have an error in tht einclude-exclude file with a rougue symbolic link.

 

[steinweo]

Hi,

Concerning Q STAT, you might have problem described in APAR PK20343 (Q STATUS SHOWS INCORRECT ACTIVITY LOG SIZE). It is solved in TSM 5.3.3.

As mentioned in APAR: You can verify that you are facing this defect if you compare the output from the Q STATUS with the output of select sum(length(message)) from actlog.

- OR -

Read this one:

Problem
'QUERY STATUS' output shows two fields that have unusual values, "Activity Log Number of Records" and "Activity Log Size"

Cause
New in TSM Server v5.3.x, is the ability to control the ACTLOGRETENTION either by number of DAYS or SIZE in MB; but the output of QUERY STATUS may be affected.

Solution
If there are unusual values for the TSM Server "QUERY STATUS" output in the "Activity Log Number of Records" or "Activity Log Size" fields, here are two solutions:

1) To permanently fix for the TSM Server ActLog tracking, update the TSM Server "dsmserv.opt" option file with the statement:
RECALCACTLOGSTATS YES
Remember, after changing TSM option files, we must halt, then restart TSM to pick up the changes. After this option is added and the TSM Server is restarted, the "QUERY STATUS" output should no longer be reporting ridiculous values for the "Activity Log Number of Records".

2) For a quick workaround to determine the actual current value for the "Activity Log Number of Records", issue these commands from the TSM Server admin command line:
create sqlt Activity.Log ACLG
select count(*) from ACLG
drop sqlt ACLG
The number reported by these commands is the correct value. After the fix from #1 above, this ActLog record count should be identical to the QUERY STATUS output.

Regards, Olivier.

 

[polvdv]

Hi Olivier,

I did execute select sum(length(message)) from actlog
giving a result of 66741726

I did also execute
create sqlt Activity.Log ACLG
select count(*) from ACLG
drop sqlt ACLG
giving as result 656866

So I think we suffer indeed from the above described apar and I will add
RECALCACTLOGSTATS YES in my server opt file next time we stop/start the server


Paul

 

[polvdv]

Hi Slapelu,

TSM server is running on mainframe, some clietns are AIX

Paul

 

[steinweo]

Good news ... I mean that at least one problem is solved. But it did not explain why TSM DB is growing.

Did you modifiy policies? I mean did you change versioning or retention within one or more backup copy groups recently?

Regards, Olivier.

PS: TSM DB only grows if you store more or expire less.

 

[polvdv]

I am also happy it is not an activity log problem but only some wrong figures.

The last months nothing was changed concerning policies or versioning, there were neither created a lot of new clients nor 'big spender' clients were created

I have also corrected my first posting stating 'almost 1 % a day' to 'almost 0,5 % a day' and in fact it is even better to say '10% the last month'.

Because I can't find one specific client who could be responsible for this growth, I think I must conclude that all the clients together are responsible for this unusual DB growth and the only thing to do is : expand my DB

 

[steinweo]

But, it is quite strange because 10% means about 1.64 GB. As you might know each file version is responsible for about 600 B + 200 B (if we assume only one copy in copy pools and no caching in disk storage pools).

Therefore, it means that TSM stored about 2,200,000 files more in one months. It is hard to believe. Furthermore, your TSM DB size will double before end of year if it continue to grow like this.

Regards, Olivier.

PS: 1.64 * (1024 * 1024 * 1024) / 800 = 2,201,170.74

 

[polvdv]

Yes I find it also strange, certainly because during the month of February the DB size didn't grow at all (zero %) and suddenly in March I see a non-stop growth every day without being able to find a cause.

Like I said before, I will expand my DB and check it every day afterwards.

Olivier, many thanks for helping me out with the Activity log figures and all your other helpfull comments, very much appreciated !!!

Paul

 

[dms19]

Silly question, but have you recently started doing archives?

 

[polvdv]

Hi dms19,
We are doing archives since a few years now, the average is approx 10 Gig a day. No changes concerning archives during the last three months.
Paul

 

[polvdv]

Maybe this can be of any help for someone else with the same problem

I think I found the reason for this problem.

One of our clients has millions of directories and files in the recycler bin.
Although we exclude the files residing in the recycler bin, the directory structure itself gets backed up and by consequence the DB keeps growing and growing rapidly.

In the meantime I did (try to) delete all backup data from this recycler bin
and I also changed the 'Exclude' option to 'Exclude.dir'

Although I do not succeed to get rid of ALL the data from this directory I see already a significant decrease of my DB utilisation with about 6 %

Probably I found the cause of the problem and will DB growth slowdown now to a normal proportion

Greetings,
Paul

 

[steinweo]

Great job ...
I would never have thought about files/directories in the recycle bin.

Regards, Olivier.