You are in a maze of twisty little error messages, all different.
The real problem here is that various pieces of amanda are each
making assumptions based on what they can see from their side of
the elephant, and you hit a corner case that is handled properly
but the messages are somewhere between confusing and just plain
wrong.
What you've got is, roughly this. TECH14 has all your dumps on it
except rochester/project2, which failed. This dump was going
directly to tape rather than to holding disk. (This was a 15 GB
dump and your holding disk is only 2 GB!) For some reason this
dump then failed. It's not clear to me from the messages whether
something happened on rochester and it stopped sending data, or
you ran out of tape, or you just had a tape write error, but some
error condition happened that caused this dump to fail shortly
before it was finished. When that happened one part of amanda said
"uh oh, tape problem, send all further dumps to disk", which
triggered the "you need to amflush message". But this was the last
dump, so there was nothing to flush.
-Mitch
On Wed, 23 Apr 2003, T. Blader wrote:
> Hello,
> I've been running amanda for a couple years here on aroudn
> 60 gig of data and almost 20 tapes. Backups run nightly
> and there is never very much of a problem.
>
> I got a strange mail this morning that said amanda was
> out of tape. This was aroudn 8:00am in the morning. The
> email told me to flush last nights backups to TAPE15 and
> use TAPE16 for the backups tonight.
>
> I logged into the tape server and ran amflush. TAPE15 was
> already in the drive because I had changed it this morning
> when I came in. Well, amflush said there was nothing to flush.
> Confusedly, I ran "amcheck tech" and amcheck said everything was OK.
>
> Why does the email amanda sent out say to change to TAPE16
> for tonight, and amcheck is telling me TAPE15 is OK? Also,
> where do you suppose the "DUMPS TO DISK" went? THere is nothing
> in my holding disks from last night. (email and notes are below)
>
> Thanks for the help.
>
>
> #################################################################
> ############ [ email from this morning ] ########################
> #################################################################
> *** THE DUMPS DID NOT FINISH PROPERLY!
>
> *** A TAPE ERROR OCCURRED: [[writing file: short write]].
> *** PERFORMED ALL DUMPS TO HOLDING DISK.
>
> THESE DUMPS WERE TO DISK. Flush them onto tape TECH15 or a new tape.
> Tonight's dumps should go onto 1 tape: TECH16.
>
> FAILURE AND STRANGE DUMP SUMMARY:
> rochester /project2 lev 0 FAILED [out of tape]
> taper: FATAL syncpipe_get: w: unexpected EOF
> rochester /project2 lev 0 FAILED ["data write: Broken pipe"]
> rochester /project2 lev 0 FAILED [dump to tape failed]
>
>
> STATISTICS:
> Total Full Daily
> -------- -------- --------
> Dump Time (hrs:min) 5:17 4:39 0:30 (0:08 start)
> Output Size (meg) 4879.5 3904.0 975.5
> Original Size (meg) 11234.9 9725.8 1509.1
> Avg Compressed Size (%) 35.9 35.1 44.0
> Tape Used (%) 41.6 33.3 8.4 (level:#disks...)
> Filesystems Dumped 22 5 17 (1:15 2:2)
> Avg Dump Rate (k/s) 255.9 231.5 443.2
> Avg Tp Write Rate (k/s) 269.7 239.2 551.8
>
>
> FAILED AND STRANGE DUMP DETAILS:
>
> /-- rochester /project2 lev 0 FAILED ["data write: Broken pipe"]
> sendbackup: start [rochester:/project2 level 0]
> sendbackup: info BACKUP=/usr/sbin/ufsdump
> sendbackup: info RECOVER_CMD=/usr/local/bin/gzip -dc
> |/usr/sbin/ufsrestore -f... -
> sendbackup: info COMPRESS_SUFFIX=.gz
> sendbackup: info end
> | DUMP: Writing 32 Kilobyte records
> | DUMP: Date of this level 0 dump: Tue Apr 22 00:35:04 2003
> | DUMP: Date of last level 0 dump: the epoch
> | DUMP: Dumping /dev/rdsk/c1t2d0s0 (rochester:/project2) to standard output.
> | DUMP: Mapping (Pass I) [regular files]
> | DUMP: Mapping (Pass II) [directories]
> | DUMP: Estimated 31104096 blocks (15187.55MB) on 0.23 tapes.
> | DUMP: Dumping (Pass III) [directories]
> | DUMP: Dumping (Pass IV) [regular files]
> | DUMP: 2.12% done, finished in 7:45
> | DUMP: 4.16% done, finished in 7:43
> | DUMP: 6.54% done, finished in 7:10
> | DUMP: 8.68% done, finished in 7:02
> | DUMP: 10.98% done, finished in 6:46
> | DUMP: 13.04% done, finished in 6:41
> | DUMP: 15.15% done, finished in 6:33
> | DUMP: 17.37% done, finished in 6:22
> | DUMP: 19.20% done, finished in 6:20
> | DUMP: 21.27% done, finished in 6:11
> | DUMP: 23.29% done, finished in 6:03
> | DUMP: 25.30% done, finished in 5:55
> | DUMP: 27.46% done, finished in 5:44
> | DUMP: 29.59% done, finished in 5:34
> | DUMP: 31.96% done, finished in 5:20
> | DUMP: 34.22% done, finished in 5:08
> | DUMP: 36.17% done, finished in 5:00
> | DUMP: 38.13% done, finished in 4:52
> | DUMP: 40.07% done, finished in 4:44
> | DUMP: 42.10% done, finished in 4:35
> | DUMP: 44.43% done, finished in 4:23
> | DUMP: 46.75% done, finished in 4:11
> | DUMP: 48.80% done, finished in 4:01
> | DUMP: 51.09% done, finished in 3:50
> | DUMP: 53.08% done, finished in 3:41
> | DUMP: 54.78% done, finished in 3:35
> | DUMP: 56.19% done, finished in 3:31
> | DUMP: 58.10% done, finished in 3:22
> | DUMP: 60.19% done, finished in 3:12
> | DUMP: 62.43% done, finished in 3:00
> | DUMP: 64.88% done, finished in 2:48
> | DUMP: 66.83% done, finished in 2:39
> | DUMP: 68.70% done, finished in 2:30
> | DUMP: 70.79% done, finished in 2:20
> | DUMP: 72.69% done, finished in 2:11
> | DUMP: 74.87% done, finished in 2:01
> | DUMP: 77.11% done, finished in 1:50
> | DUMP: 79.14% done, finished in 1:40
> | DUMP: 80.62% done, finished in 1:34
> | DUMP: 82.74% done, finished in 1:23
> | DUMP: 84.03% done, finished in 1:18
> | DUMP: 85.75% done, finished in 1:09
> | DUMP: 87.42% done, finished in 1:02
> \--------
>
>
> NOTES:
> driver: WARNING: /backup1/amanda/work: 1945600 KB requested, but only
> 1898904 KB available.
> planner: Incremental of rochester:/revision1 bumped to level 2.
> planner: Incremental of rochester:/project2 bumped to level 3.
> planner: Incremental of rochester:/user_files bumped to level 3.
> planner: Dump too big for tape: full dump of rochester:/ug_lib delayed.
> taper: tape TECH14 kb 10454688 fm 23 writing file: short write
>
>
> DUMP SUMMARY:
> DUMPER STATS
> TAPER STATS
> HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s
> MMM:SS KB/s
> --------------------------
> ----------------------------------------------------
> andromeda / 1 128 128 -- 0:04 29.6 0:00
> 3021.8
> andromeda /usr/home 1 6880 6880 -- 0:26 260.9 0:07
> 1030.5
> cygnus / 1 3744 3744 -- 0:13 284.4 0:01
> 4980.5
> cygnus /quote 1 543743 236256 43.4 8:06 486.0 8:06
> 485.7
> cygnus /quote2 1 544480 544480 -- 10:29 866.0 4:40
> 1945.2
> davinci -/adm/flamtech 1 160 160 -- 0:03 47.5 0:00
> 3612.1
> draco / 1 96 96 -- 0:10 10.0 0:00
> 3690.8
> draco /usr/home 0 779212 216416 27.8 25:30 141.4 25:31
> 141.4
> rochester / 1 128 128 -- 0:06 22.4 0:00
> 3246.0
> rochester /apps 0 554943 205088 37.0 17:04 200.3 17:04
> 200.2
> rochester /fixture 0 922687 414944 45.0 32:21 213.8 32:22
> 213.7
> rochester /opt 1 96 96 -- 0:05 19.4 0:00
> 3441.9
> rochester /project 1 386591 173888 45.0 14:01 206.8 14:02
> 206.7
> rochester /project1 1 5855 1664 28.4 0:54 30.7 0:55
> 30.6
> rochester /project2 0
> FAILED--------------------------------------------
> rochester /revision 1 127 32 25.2 0:17 1.8 0:20
> 3.3
> rochester /revision1 2 39231 17280 44.0 1:46 163.0 1:46
> 162.6
> rochester /revision2 0 6923711 2382592 34.4 197:58 200.6 197:59
> 200.6
> rochester /ug_lib 2 768 768 -- 0:10 75.3 0:00 3676.9
> rochester /user_files 0 778688 778688 -- 14:58 867.4 5:41
> 2285.4
> rochester /usr 1 96 96 -- 0:17 5.7 0:02
> 55.8
> rochester /usr/local 1 96 96 -- 0:04 23.2 0:00
> 3722.7
> rochester /var 1 13120 13120 -- 0:22 605.9 0:10
> 1258.8
>
> (brought to you by Amanda version 2.4.1p1)
>
>
>
>
> ###################################################################
> ############ [ amflush says nothing to flush ]####################
> ###################################################################
> [operator@apus ~]$ amflush tech
> Scanning /backup1/amanda/work...
> Could not find any Amanda directories to flush.
>
>
>
> ##################################################################
> ########### [ amcheck says all is good ] ######################
> ##################################################################
> [operator@apus ~]$ amcheck tech
> Amanda Tape Server Host Check
> -----------------------------
> WARNING: /backup1/amanda/work: only 1898905 KB free (1945600 KB
> requested).
> NOTE: skipping tape-writable test.
> Tape TECH15 label ok.
> Server check took 17.838 seconds.
>
> Amanda Backup Client Hosts Check
> --------------------------------
> Client check: 5 hosts checked in 2.152 seconds, 0 problems found.
>
> (brought to you by Amanda 2.4.1p1)
>
>
|
