Amanda-Users

Re: Amanda's dumper going amok

2004-11-04 13:12:11
Subject: Re: Amanda's dumper going amok
From: Eric Siegerman <erics AT telepres DOT com>
To: amanda-users AT amanda DOT org
Date: Thu, 4 Nov 2004 13:04:10 -0500
On Thu, Nov 04, 2004 at 09:57:52AM +0100, Paul Bijnens wrote:
> Flynn wrote:
> 
> >Amdump sometimes goes crazy apparently eating up all the machine resources
> >and I can't get any access to anything when this happens, because I think
> >it's lost managing memory swap pages or something.

Yes, your symptoms do sound like severe page thrashing.  I've
never seen that with Amanda either, but there's always a first
time :-/

> the loadavg sometimes goes to 12-15 (the 15-minute one!), but I
> still can connect to it, run amstatus etc.

That's CPU contention.  Stuff slows down, but a lot more
gracefully than when it's RAM that's the problem, as it seems to
be in this case.

> Try to gather some data while you use it (e.g. with crontab) about
> load and memory use, long list of processes, etc. and hopefully
> you see something just before the machines locks up.

One easy way is to just run "vmstat 30" all night -- use the
"script" command to capture the output.  That'll show you whether
it is indeed a paging problem.  Note that vmstat doesn't print
the time, so it can sometimes be useful to run a script like
this:
        while [ 1 ]; do
                date
                sleep 300
        done
in the background in the *same* window as vmstat.  The output
will be a bit jumbled, but at least it'll be timestamped.

Then (or at the same time, in another window) you can run a loop
like the above but with a "ps -le" in it, again capturing the
output with "script".

(I prefer "script" to simple output redirection for stuff like
this, because I can both watch the commands as they run and
capture their output at the same time.)

Of course, if your system has the "sar" stuff installed, you can
use that, but there's a bit more learning and setup involved.
What I described above is the quick and dirty approach.

--

|  | /\
|-_|/  >   Eric Siegerman, Toronto, Ont.        erics AT telepres DOT com
|  |  /
The animal that coils in a circle is the serpent; that's why so
many cults and myths of the serpent exist, because it's hard to
represent the return of the sun by the coiling of a hippopotamus.
        - Umberto Eco, "Foucault's Pendulum"