The Active Log was never more than 2% full, and yes, I have an Archive
Failover Log, which was never written to and remained empty. Q DB F=D
shows blanks for Last Database Reorganization, which means it has never
been done? Hopefully never just since the last reboot. At any rate, DB
reorg was not what caused this to happen. (Odd, because our other
production V6.2 server does reorgs fairly often.)

What happened was that some client backed up 3,000,000 small files.
(That's a separate problem; it happens.) Since they were all small, the
Active Log never got pinned for more than a second or two, and the
storage pools never got full, but the Archive Log sure got hammered.

On our system, it appears that db2dump/db2diag.0.log is in the instance
owner ID's homedir, not the instance directory which I have set up to be
a subdir of its homedir. But anyway, I found it, at
/home/adsm_4/sqllib/db2dump/db2diag.0.log and it was very interesting.
(Among other things, it looks like it did its last DB reorg over a year
ago.) It appears from this log file that the first DB backup was
cleaning up and futzing with the Volume History File over and over, when
the second DB backup was triggered. I guess I'm lucky it only happened
twice.

Thanks Wanda and Skylar, it looks like we need to go to 6.2.5 soon. And
also increase the size of the main Archive Log because 3,000,000 small
files can happen again.

Roger Deschner      University of Illinois at Chicago     rogerd AT uic DOT edu
======I have not lost my mind -- it is backed up on tape somewhere.=====



>Skylar Thompson wrote:
>We had the same issue until going to v6.3. DB reorgs were frequently the
>cause of the problems. We had the ALLOWREORGTABLE server option off to
>help prevent this, although there's a long-term performance hit from
>that as well.
>
>If you look in your TSM instance directory, you'll find a
>db2dump/db2diag.0.log file that you can tail to confirm that. It'll log
>every time reorganization starts and stops, along with other interesting
>details.
>
>-- Skylar Thompson (skylar2 AT u.washington DOT edu)
>-- Genome Sciences Department, System Administrator
>-- Foege Building S046, (206)-685-7354
>-- University of Washington School of Medicine
>
>On 03/22/13 01:38 PM, Prather, Wanda wrote:
>
>    Could the active log also be filling because the archive log was full?
>    There are known issues with DB backup triggers in 6.2.2.  (although we're
>    Windows, I think the problem is the same)
>
>    We've seen cases where the server would just fire db backup after db backup
>    after dbbackup, because the active log was getting full.
>    As the db backup doesn't clear the active log anyway, it was pointless.
>    That's fixed in 6.2.5, we haven't seen it since the upgrade.
>
>    -----Original Message-----
>    From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On 
> Behalf Of
>    Roger Deschner
>    Sent: Friday, March 22, 2013 2:01 PM
>    To: ADSM-L AT VM.MARIST DOT EDU
>    Subject: [ADSM-L] Second full DB backup triggered while first was still 
> underway
>
>    The Archive Log got to 80% full, so it triggered a full TSM DB backup.
>    So far, so good.
>
>    However, as the first one was finishing, and before it could empty the 
> Archive
>    Log, a second full TSM DB backup was triggered. For a short while, there 
> were
>    two full TSM DB backups running. This second backup was a pointless 
> duplicate,
>    and may possibly be corrupted. It appears there is a window where the 
> trigger
>    does not know that a full DB backup has just been run.
>
>    TSM Server 6.2.2.30 on AIX 5.3.
>
>    Roger Deschner      University of Illinois at Chicago     rogerd AT uic 
> DOT edu
>    ======I have not lost my mind -- it is backed up on tape somewhere.=====
>
>
>
>