Amanda-Users

Re: Unable to Flush Held Backup Jobs

2005-06-21 18:31:42
Subject: Re: Unable to Flush Held Backup Jobs
From: Frank Smith <fsmith AT hoovers DOT com>
To: Andrew Crawford <amanda AT evermore DOT com>, amanda-users AT amanda DOT org
Date: Tue, 21 Jun 2005 17:24:57 -0500
--On Tuesday, June 21, 2005 14:28:42 -0700 Andrew Crawford <amanda AT evermore 
DOT com> wrote:

> Greetings,
> 
> I am working with an Amanda installation that was a little neglected.
> It backs up nightly and I have run a number of successful restores with
> it.  So, it appears operational.
> 
> Several tape changes were missed over the last year and the unflushed
> jobs on the holding disk have accumulated.  There appear to be 26 of
> them, now.  I searched the web (mostly on the Amanda site) and looked
> through docs and FAQ to see if there was an easy way to just delete the
> held jobs, since they are pretty much irrelevant, now.  However, I
> wasn't able to find anything.  It seems like it should be possible to
> manually delete held jobs but, I hate to risk making a mess.
> 
> So, I decided that I would flush the jobs to tape, just to be safe.  The
> problem is that they won't flush.  First, I tried to flush all of the
> held jobs.  That job ran for about 10 hours before I killed it.  I
> figured that there is no way it was taking that long to fill up a 12G
> DAT tape.
> 
> The second attempt, I told it to just flush the first 4 held jobs in the
> list.  I killed that job after 20 hours for the same reason.  After
> killing the jobs and running amcleanup, I get email that suggests that
> amanda wasn't doing anything during the flushes:
> 
>> *** THE DUMPS DID NOT FINISH PROPERLY!
>> 
>> The dumps were flushed to tape XXXDaily18.
>> The next tape Amanda expects to use is: XXXDaily19.
>> 
>> 
>> STATISTICS:
>>                           Total       Full      Daily
>>                         --------   --------   --------
>> Estimate Time (hrs:min)    0:00
>> Run Time (hrs:min)         0:00
>> Dump Time (hrs:min)        0:00       0:00       0:00
>> Output Size (meg)           0.0        0.0        0.0
>> Original Size (meg)         0.0        0.0        0.0
>> Avg Compressed Size (%)     --         --         -- 
>> Filesystems Dumped            0          0          0
>> Avg Dump Rate (k/s)         --         --         -- 
>> 
>> Tape Time (hrs:min)        0:00       0:00       0:00
>> Tape Size (meg)             0.0        0.0        0.0
>> Tape Used (%)               0.0        0.0        0.0
>> Filesystems Taped             0          0          0
>> Avg Tp Write Rate (k/s)     --         --         -- 
>> 
>> 
>> DUMP SUMMARY:
>>                                      DUMPER STATS            TAPER STATS 
>> HOSTNAME     DISK        L ORIG-KB OUT-KB COMP% MMM:SS  KB/s MMM:SS  KB/s
>> -------------------------- --------------------------------- ------------
>> xxxfs.ex /home         NO FILE TO FLUSH -----------------------------
>> 
>> (brought to you by Amanda version 2.4.3)
> 
> The (amflush) log seems to indicate that everything went fine:
> 
>> amflush: datestamp 20050620
>> driver: pid 28678 executable driver version 2.4.3
>> driver: send-cmd time 0.002 to taper: START-TAPER 20050620
>> taper: pid 28679 executable taper version 2.4.3
>> taper: page size is 4096
>> taper: buffer size is 32768
>> taper: buffer[00] at 0x40208000
>> taper: buffer[01] at 0x40210000
>> taper: buffer[02] at 0x40218000
>> taper: buffer[03] at 0x40220000
>> taper: buffer[04] at 0x40228000
>> taper: buffer[05] at 0x40230000
>> taper: buffer[06] at 0x40238000
>> taper: buffer[07] at 0x40240000
>> taper: buffer[08] at 0x40248000
>> taper: buffer[09] at 0x40250000
>> taper: buffer[10] at 0x40258000
>> taper: buffer[11] at 0x40260000
>> taper: buffer[12] at 0x40268000
>> taper: buffer[13] at 0x40270000
>> taper: buffer[14] at 0x40278000
>> taper: buffer[15] at 0x40280000
>> taper: buffer[16] at 0x40288000
>> taper: buffer[17] at 0x40290000
>> taper: buffer[18] at 0x40298000
>> taper: buffer[19] at 0x402a0000
>> taper: buffer structures at 0x402a8000 for 240 bytes
>> taper: read label `XXXDaily18' date `20050503'
>> taper: wrote label `XXXDaily18' date `20050620'
>> driver: adding holding disk 0 dir /var/tmp size 296960
>> reserving 296960 out of 296960 for degraded-mode dumps
>> driver: start time 76140.288 inparallel 4 bandwidth 2000 diskspace 296960 
>> dir OBSOLETE datestamp 20050620 driver: drain-ends tapeq LFFO big-dumpers ttt
>> driver: result time 76140.289 from taper: TAPER-OK 
>> driver: state time 76140.301 free kps: 2000 space: 296960 taper: idle 
>> idle-dumpers: 4 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 86400 driver-idle: 
>> not-idle
>> driver: interface-state time 76140.301 if : free 600 if ETH0: free 400 if 
>> LOCAL: free 1000
>> driver: hdisk-state time 76140.301 hdisk 0: free 296960 dumpers 0
>> driver: QUITTING time 76140.301 telling children to quit
>> driver: send-cmd time 76140.301 to taper: QUIT
>> taper: DONE [idle wait: 76136.019 secs]
>> taper: writing end marker. [XXXDaily18 OK kb 0 fm 0]
>> driver: FINISHED time 76140.316
> 
> Anyone know what might be going wrong?

They may be incomplete dumps.  If there is a problem before a dump
completes you end up with a partial dump image in your holdingdisk
that never gets cleaned up. (To the developers: Since amflush seems
to know which ones are complete, can't it remove the incomplete ones?
Also, why does amflush mark a tape as used even when it doesn't write
any new data to it?)

> Anyone know how to just delete the held backup jobs?

'rm' works for me.  Just make sure they aren't complete (try a verify of the
dump image) or are older than your rotation before removing them.

Frank

> Any help would be much appreciated.
> 
> Andrew Crawford



-- 
Frank Smith                                      fsmith AT hoovers DOT com
Sr. Systems Administrator                       Voice: 512-374-4673
Hoover's Online                                   Fax: 512-374-4501


<Prev in Thread] Current Thread [Next in Thread>