ADSM-L

Log pinned, filled, almost crashed

2005-11-08 12:37:56
Subject: Log pinned, filled, almost crashed
From: Roger Deschner <rogerd AT UIC DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 8 Nov 2005 11:37:35 -0600
Last night, a hung client backup session pinned the log, it filled, and
I got called in the middle of the night just in time to avert a messy
log-full crash. This has happened before.

This is the greatest looming fear I have - a Log fillup. The Log is in
Normal Mode, and is 12GB - can't get much larger. Seems that it is most
often caused by an errant client node, who for unknown reasons suffers
from poor throughput. (I do, sometimes, go out and investigate poor
client performance, sometimes finding a misprogrammed or defective
network switch. But this is hit and miss.)

ITSM should not be this vulnerable to the vagaries of its clients. Among
other things, this is an open door to a DOS attack.

I had built a daemon that, once every 15 minutes, checks the log
fullness. If it's over a threshold level, it queries to see who has it
pinned, using SHOW LOGPIN, and then it cancels that session. This works
most of the time, but apparently not last night because this client node
was waiting for a tape for some reason, perhaps to back up a file that
was larger than MAXSIZE for the disk storage pool. (Or not. There was a
free tape drive. Could have been waiting for a specific tape for some
reason.) That daemon had already done its job and issued 3 CANCEL
SESSION commands to no avail. I finally got that client session to
cancel, after cancelling all processes (reclamation...) which had tapes.

Short of a poison pill daemon that does a HALT at 95% log full, what do
others do to positively protect the Log from filling up?

Roger Deschner      University of Illinois at Chicago     rogerd AT uic DOT edu
============ "In theory, theory and practice are the same, =============
========= but in practice, theory and practice are different." =========

<Prev in Thread] Current Thread [Next in Thread>