Amanda-Users

Re: Comperession/Tape Size Issues

2003-07-07 18:58:53
Subject: Re: Comperession/Tape Size Issues
From: Gene Heskett <gene.heskett AT verizon DOT net>
To: "Joshua D. Bello" <josh AT nextrials DOT com>, amanda-users AT amanda DOT org
Date: Mon, 7 Jul 2003 18:54:44 -0400
On Monday 07 July 2003 18:16, Joshua D. Bello wrote:
>Thank you to everybody for their help and suggestions.  I went ahead
> and disabled hardware compression on my drive.
>
>Unfortunately, I am running into further problems in my attempts to
>clear out the previously spooled dumps with amflush.  amflush fails
>after a short time without writing anything at all to tape.
>
>Here is what happens...
>
>I rum amflush and get the following output:
>
>  Scanning /local0/ambackup/nextrials...
>    20030703: found Amanda directory.
>    20030704: found Amanda directory.
>
>  Multiple Amanda directories, please pick one by letter:
>    A. 20030703
>    B. 20030704
>  Select directories to flush [A..B]: [ALL]
>
>I get the same results, whether I pick A, B, or ALL.  The amflush
> report looks like:
>
>STATISTICS:
>
><big fat 0's for every reported time and size value>
><Avg Compressed Size, Avg Dump Rate and Avg Tp Write Rate all report
>"--">
>
>NOTES:
>  driver: WARNING: /local0/ambackup: 83886080 KB requested, but only
>  61325902 KB available
>  taper: tape NEXTRIALS026 kb 0 fm 0 [OK]
>
>DUMP SUMMARY:
>                                     DUMPER STATS            TAPER
> STATS HOSTNAME     DISK        L ORIG-KB OUT-KB COMP% MMM:SS  KB/s
> MMM:SS KB/s --------------------------
> --------------------------------- ----------- anubis       /       
>      NO FILE TO FLUSH --------------------------- anubis      
> /local0       NO FILE TO FLUSH --------------------------- anubis  
>     /usr/local    NO FILE TO FLUSH ---------------------------
> anubis       /var          NO FILE TO FLUSH
> --------------------------- athene.web   /             NO FILE TO
> FLUSH --------------------------- athene.web   /var          NO
> FILE TO FLUSH --------------------------- etc, etc, etc...
>
>While amflush is running, amstatus reports that everything is
> waiting to flush, as in:
>
>flush           :  47  7288612k
>       <0k, 0.00% values for everything else>
>wait to flush   :  47  7288612kk  7288612k (100.00%) (  0.00%)
>4 dumpers idle, 0 dumpers busy...
>
>The amanda log file reports:
>
>DISK amflush hades /
>DISK amflush hades /usr
> etc, etc...
>DISK amflush bali.web /var
>START amflush date 20030707
>START driver date 20030707
>WARNING driver WARNING: /local0/ambackup/nextrials: 104857600 KB
>requested, but
>only 61325652 KB available.
>STATS driver startup time 0.071
>START taper datestamp 20030707 label NEXTRIALS026 tape 0
>
>The amflush log file reports:
>
>amflush: datestamp 20030707
>driver: pid 58387 executable driver version 2.4.3b2
>driver: send-cmd time 0.004 to taper: START-TAPER 20030707
>FLUSH ra /local0 20030703 0
>/local0/ambackup/nextrials/20030703/ra._local0.0
>FLUSH ra /usr/local 20030703 1
>/local0/ambackup/nextrials/20030703/ra._usr_local
>.1
> etc, etc...
>ENDFLUSH
>driver: adding holding disk 0 dir /local0/ambackup/nextrials size
>61325652
>reserving 61325652 out of 61325652 for degraded-mode dumps
>driver: start time 0.071 inparallel 4 bandwidth 1410666408 diskspace
>61325652 di
>r OBSOLETE datestamp 20030707 driver: drain-ends tapeq LFFO
> big-dumpers sssS
>taper: pid 58388 executable taper version 2.4.3b2
>taper: page size is 4096
>taper: buffer size is 32768
>taper: buffer[00] at 0x30048000
>taper: buffer[01] at 0x30050000
>taper: buffer[02] at 0x30058000
>taper: buffer[03] at 0x30060000
>taper: buffer[04] at 0x30068000
>taper: buffer[05] at 0x30070000
>taper: buffer[06] at 0x30078000
>taper: buffer[07] at 0x30080000
>taper: buffer[08] at 0x30088000
>taper: buffer[09] at 0x30090000
>taper: buffer[10] at 0x30098000
>taper: buffer[11] at 0x300a0000
>taper: buffer[12] at 0x300a8000
>taper: buffer[13] at 0x300b0000
>taper: buffer[14] at 0x300b8000
>taper: buffer[15] at 0x300c0000
>taper: buffer[16] at 0x300c8000
>taper: buffer[17] at 0x300d0000
>taper: buffer[18] at 0x300d8000
>taper: buffer[19] at 0x300e0000
>taper: buffer structures at 0x300e8000 for 240 bytes
>taper: read label `NEXTRIALS026' date `20030707'
>taper: wrote label `NEXTRIALS026' date `20030707'
>driver: result time 8.932 from taper: TAPER-OK
>driver: send-cmd time 8.932 to taper: FILE-WRITE 00-00001
>/local0/ambackup/nextr
>ials/20030703/ra._local0.0 ra /local0 0 20030703
>driver: state time 8.932 free kps: 1410666408 space: 61325652 taper:
>writing idl
>e-dumpers: 4 qlen tapeq: 46 runq: 0 roomq: 0 wakeup: 86400
> driver-idle: not-idle
>driver: interface-state time 8.932 if : free 600000 if FXP0: free
>1410065408 if
>LOCAL: free 1000
>driver: hdisk-state time 8.932 hdisk 0: free 61325652 dumpers 0
>
>This is frustrating, I need to get the spool disk flushed or else
>backups will fail completely tonight due to lack of space.
>
>Amanda has been chugging along just fine for over a year, this
> problem has just now crept up as my ammount of data storage has
> grown.

>From the looks of it, you have more than a tapefull, so its not going 
to fit, no way, no how.

This goes back to the relatively huge entries in the disklist.  I'd 
bite the bullet, delete the missed backups from the holding disk, 
break the disklist up into manageable sized pieces and only expose 
about 3/4ths of a tape full per run until its all exposed, 
effectively restarting amanda with a more workable configuration.

Did you treat the tapes to a remove the compression treatment?

Something like this run over each tape should do it.
-----------------------
#!/bin/sh
# note, presently setup for a 4 tape changer, rip up to suit

if [ `whoami` != 'root' ]; then
        echo
        echo "!!!!!!!!!!!!!!!!! Warning !!!!!!!!!!!!!!!!!!"
        echo "fixtape needs to be run by the user root,"
        echo "else the perms on some commands will be denied."
        echo
        exit 1
fi

set echo on
cd /amanda
# blatently stolen from the bash manual
NUMBERS="0 1 2 3"
for number in $NUMBERS
do 
        echo "amtape DailySet1 slot "$number
        su amanda -c 'amtape DailySet1 slot '$number
        mt -f /dev/nst0 rewind
        dd if=/dev/st0 of=./scratch bs=64K count=1
        ls -l ./scratch # show the user how big the label block is
        mt -f /dev/nst0 compression off
        mt -f /dev/nst0 defcompression -1
        # now make the label a 32k block if it wasn't before
        dd if=./scratch of=/dev/nst0 bs=32K conv=sync
        mt -f /dev/nst0 tell
        # This _should_ get rid of compressed headers, adjust the write         
        # blocks to make sure it overflows the drives buffer, forcing the
        # flush and the flag update
        echo "forcing buffer flush with an 4+ meg write to tape # $number
        dd if=/dev/zero bs=32K count=130 of=/dev/st0 
        echo "Now reading the label"  # show it to the user
        dd if=/dev/st0 bs=32K
        # verify a few other things
        mt -f /dev/st0 status
done
exit 0
-----------------------------------
The above will treat 4 tapes, a magazine full here, at a time, and do 
it without wrecking the labels amanda laid down.  It may require some 
fiddling for other systems of course so don't run it blindly without 
verifying the device numbers and such are correct.  For a single 
drive, no changer, remove all the loop constructions etc.


>Thanks again, you folks are super helpful!
>
>--Joshua D. Bello
>
>On Mon, Jul 07, 2003 at 10:26:09AM -0700, Joshua D. Bello wrote:
>> I am currently experiencing problems with Amanda backup runs
>> completing successfully, supposedly due to running out of tape.  I
>> am using Amanda 2.4.3 on a FreeBSD 4.8-STABLE machine, backing up
>> to 35/70GB DLT 7000 drive.  Compression is enabled on the drive,
>> and we are also using compression within Amanda.  We are getting
>> nowhere near the expected 70GB compressed capacity of the tapes.

-- 
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz  512M
99.26% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.



<Prev in Thread] Current Thread [Next in Thread>