Try looking at q filesystem from day to day to identify the culprit, then
look at dsmsched.log or the output from the dsmc command that backed them
up.  When this has happened to me in the past it has been due to files with
unique machine-generated names e.g. an app that created 5000 files a day
with the name nul.dat.nnnn where nnnn was a number that incremented by one
for each new file.
Files with timestamps or pids in the name are other possibilities.
Once identified, match them into a mgmtclass with short retention period to
get rid of most of the existing ones.
You can then exclude them permanently if that is desirable.

Regards

Steve Harris
AIX/ADSM Admin
Queensland Health, Brisbane Australia