I'm setting up Amanda on a machine with directly attached
disks-to-backup and jukebox. This configuration has worked
successfully for me on other machines.
When I run the dump though, it calculates what it needs to
do, gets to the "driver: hdisk-state time xxx" line, and
then just hangs for everdoing nothing. I reduced the
disklist to a single ~2G filesystem and it does the same
thing every time. I have no idea what's wrong with it or
how to figure out what's wrong, and it really has be stumped
because this config should be ok, and it's even worked
before on this very machine.
Here's what the Amanda processes are doing:
$ ps auxfww | grep ^amanda
amanda 9302 0.0 0.1 2064 968 pts/1 S 21:08 0:00 |
\_ /bin/sh /usr/sbin/amdump normal
amanda 9315 0.0 0.1 2144 824 pts/1 S 21:08 0:00 |
\_ /usr/lib/amanda/driver normal
amanda 9316 0.0 0.1 2780 784 pts/1 S 21:08 0:00 |
\_ taper normal
amanda 9320 0.0 0.1 2804 864 pts/1 S 21:09 0:00 |
| \_ taper normal
amanda 9317 0.0 0.1 2276 828 pts/1 S 21:08 0:00 |
\_ dumper0 normal
amanda 9318 0.0 0.1 2276 828 pts/1 S 21:08 0:00 |
\_ dumper1 normal
amanda 9319 0.0 0.1 2276 828 pts/1 S 21:08 0:00 |
\_ dumper2 normal
$ ps -eo user,comm,pid,wchan | grep ^amanda
amanda amdump 9302 wait4
amanda driver 9315 do_select
amanda taper 9316 unix_stream_data_wait
amanda dumper 9317 unix_stream_data_wait
amanda dumper 9318 unix_stream_data_wait
amanda dumper 9319 unix_stream_data_wait
amanda taper 9320 pipe_wait
$ sudo lsof -u amanda
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
amdump 9302 amanda cwd DIR 9,1 4096 82229
/etc/amanda/normal
amdump 9302 amanda rtd DIR 9,1 4096 2 /
amdump 9302 amanda txt REG 9,1 626124 82095 /bin/bash
amdump 9302 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
amdump 9302 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
amdump 9302 amanda mem REG 9,1 12842 131486
/lib/libdl-2.3.2.so
amdump 9302 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
amdump 9302 amanda 0r CHR 1,3 67293 /dev/null
amdump 9302 amanda 1w REG 58,3 6059 389583
/var/log/amanda/amdump
amdump 9302 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
amdump 9302 amanda 255r REG 58,2 3708 132560
/usr/sbin/amdump
driver 9315 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
driver 9315 amanda rtd DIR 9,1 4096 2 /
driver 9315 amanda txt REG 58,2 107826 376069
/usr/lib/amanda/driver
driver 9315 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
driver 9315 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
driver 9315 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
driver 9315 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
driver 9315 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
driver 9315 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
driver 9315 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
driver 9315 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
driver 9315 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
driver 9315 amanda 0r FIFO 0,5 102241558 pipe
driver 9315 amanda 1w REG 58,3 6059 389583
/var/log/amanda/amdump
driver 9315 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
driver 9315 amanda 3u unix 0xc46ac220 102241568 socket
driver 9315 amanda 4u unix 0xdc764b80 102241570 socket
driver 9315 amanda 5u unix 0xcd7700c0 102241572 socket
driver 9315 amanda 6u unix 0xda30ec20 102241574 socket
taper 9316 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
taper 9316 amanda rtd DIR 9,1 4096 2 /
taper 9316 amanda txt REG 58,2 99316 376072
/usr/lib/amanda/taper
taper 9316 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
taper 9316 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
taper 9316 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
taper 9316 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
taper 9316 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
taper 9316 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
taper 9316 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
taper 9316 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
taper 9316 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
taper 9316 amanda mem DEL 0,4 917510 /SYSV00000000
taper 9316 amanda 0u unix 0xc46acbe0 102241569 socket
taper 9316 amanda 1u unix 0xc46acbe0 102241569 socket
taper 9316 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
taper 9316 amanda 4w FIFO 0,5 102241582 pipe
taper 9316 amanda 5r FIFO 0,5 102241583 pipe
dumper 9317 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
dumper 9317 amanda rtd DIR 9,1 4096 2 /
dumper 9317 amanda txt REG 58,2 96915 376070
/usr/lib/amanda/dumper
dumper 9317 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
dumper 9317 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
dumper 9317 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
dumper 9317 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
dumper 9317 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
dumper 9317 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
dumper 9317 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
dumper 9317 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
dumper 9317 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
dumper 9317 amanda mem REG 9,1 49929 131473
/lib/libnss_files-2.3.2.so
dumper 9317 amanda 0u unix 0xde6d04a0 102241571 socket
dumper 9317 amanda 1u unix 0xde6d04a0 102241571 socket
dumper 9317 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
dumper 9317 amanda 3u IPv4 102241593 UDP *:656
dumper 9318 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
dumper 9318 amanda rtd DIR 9,1 4096 2 /
dumper 9318 amanda txt REG 58,2 96915 376070
/usr/lib/amanda/dumper
dumper 9318 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
dumper 9318 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
dumper 9318 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
dumper 9318 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
dumper 9318 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
dumper 9318 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
dumper 9318 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
dumper 9318 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
dumper 9318 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
dumper 9318 amanda mem REG 9,1 49929 131473
/lib/libnss_files-2.3.2.so
dumper 9318 amanda 0u unix 0xcd770dc0 102241573 socket
dumper 9318 amanda 1u unix 0xcd770dc0 102241573 socket
dumper 9318 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
dumper 9318 amanda 3u IPv4 102241596 UDP *:657
dumper 9319 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
dumper 9319 amanda rtd DIR 9,1 4096 2 /
dumper 9319 amanda txt REG 58,2 96915 376070
/usr/lib/amanda/dumper
dumper 9319 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
dumper 9319 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
dumper 9319 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
dumper 9319 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
dumper 9319 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
dumper 9319 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
dumper 9319 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
dumper 9319 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
dumper 9319 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
dumper 9319 amanda mem REG 9,1 49929 131473
/lib/libnss_files-2.3.2.so
dumper 9319 amanda 0u unix 0xda30f920 102241575 socket
dumper 9319 amanda 1u unix 0xda30f920 102241575 socket
dumper 9319 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
dumper 9319 amanda 3u IPv4 102241598 UDP *:658
taper 9320 amanda cwd DIR 58,1 40960 32647 /tmp/amanda
taper 9320 amanda rtd DIR 9,1 4096 2 /
taper 9320 amanda txt REG 58,2 99316 376072
/usr/lib/amanda/taper
taper 9320 amanda mem REG 9,1 90168 131383
/lib/ld-2.3.2.so
taper 9320 amanda mem REG 58,2 445193 66233
/usr/lib/libamserver-2.4.4p1.so
taper 9320 amanda mem REG 58,2 190598 65577
/usr/lib/libamtape-2.4.4p1.so
taper 9320 amanda mem REG 58,2 590579 65913
/usr/lib/libamanda-2.4.4p1.so
taper 9320 amanda mem REG 9,1 137140 131487
/lib/libm-2.3.2.so
taper 9320 amanda mem REG 58,2 208488 65563
/usr/lib/libreadline.so.4.3
taper 9320 amanda mem REG 9,1 11696 131407
/lib/libtermcap.so.2.0.8
taper 9320 amanda mem REG 9,1 90721 131465
/lib/libnsl-2.3.2.so
taper 9320 amanda mem REG 9,1 1292588 131338
/lib/libc-2.3.2.so
taper 9320 amanda mem DEL 0,4 917510 /SYSV00000000
taper 9320 amanda 0u unix 0xc46acbe0 102241569 socket
taper 9320 amanda 1u unix 0xc46acbe0 102241569 socket
taper 9320 amanda 2w REG 58,3 6059 389583
/var/log/amanda/amdump
taper 9320 amanda 3r FIFO 0,5 102241582 pipe
taper 9320 amanda 4u CHR 9,128 67161 /dev/nst0
taper 9320 amanda 6w FIFO 0,5 102241583 pipe
amanda is just stuck, doesn't timeout, doesn't do anything,
just sits there.
I'm just running:
sudo -u amanda amdump normal
this is a totally fresh run, nothing has been backed up so
far (although I've aborted some attempts when it hung and
run amcleanup).
Here are the relevant config files:
$ cat normal/amanda.conf | grep -v ^\#
org "Questra Corporation"
mailto "root"
dumpuser "amanda"
printer "shamu.roc.questra.com"
tpchanger "chg-zd-mtx"
changerdev "/dev/changer"
tapedev "/dev/nst0"
changerfile "changer"
diskfile "disklist"
tapelist "tapelist"
infofile "/var/lib/amanda/normal/curinfo"
indexdir "/var/lib/amanda/normal/index"
logdir "/var/log/amanda"
dumpcycle 14 days
runspercycle 0
tapecycle 18 tapes
runtapes 6
taperalgo largestfit
includefile "tapetype"
tapetype AIT-2
netusage inf
inparallel 3
autoflush on
amrecover_do_fsf on
amrecover_check_label on
define dumptype tardump {
comment "uses GNU tar"
compress none
holdingdisk no
ignore no
index yes
program "GNUTAR"
record no
starttime 0001
}
$ cat normal/changer.conf | grep -v ^\#
driveslot=0
firstslot=1
lastslot=18
havereader=0 # buy one of these to make my life easier
autoclean=0
offline_before_unload=1
initial_poll_delay=10
unloadpause=3
$ cat normal/disklist | grep -v ^\#
localhost /export/local tardump
$ cat normal/tapetype | grep -v ^\#
define tapetype AIT-2 {
comment "SDX-500C drive, no hw-compress, 50G AIT-2 media,
tapetype-gen"
length 48898 M
filemark 2788 K
speed 6080 KBps
}
here is the logfile from the dump (which is still hanging;
I'll leave it that way in case some diagnosis can still be
done after a reply from the list):
$ sudo -u amanda cat /var/log/amanda/amdump
amdump: start at Sat Jul 26 21:08:59 EDT 2003
amdump: datestamp 20030726
planner: pid 9314 executable /usr/lib/amanda/planner version 2.4.4p1
planner: build: VERSION="Amanda-2.4.4p1"
planner: BUILT_DATE="Wed Jul 2 04:41:52 EDT 2003"
planner: BUILT_MACH="Linux benzoyl.roc.questra.com
2.4.21-pre4-ac1+neil+trond+gibbs+netdump+acl+ea #1 Mon Jun 2 11:27:48 EDT 2003
i686 i686 i386 GNU/Linux"
planner: CC="gcc"
planner: CONFIGURE_COMMAND="'./configure' '--host=i686-pc-linux-gnu'
'--build=i686-pc-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin'
'--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share'
'--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/lib/amanda'
'--localstatedir=/var/lib' '--sharedstatedir=/usr/com'
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-shared'
'--with-index-server=localhost'
'--with-gnutar-listdir=/var/lib/amanda/gnutar-lists'
'--with-smbclient=/usr/bin/smbclient' '--with-amandahosts' '--with-user=amanda'
'--with-group=disk' '--with-gnutar=/bin/tar'"
planner: paths: bindir="/usr/bin" sbindir="/usr/sbin"
planner: libexecdir="/usr/lib/amanda" mandir="/usr/share/man"
planner: AMANDA_TMPDIR="/tmp/amanda" AMANDA_DBGDIR="/tmp/amanda"
planner: CONFIG_DIR="/etc/amanda" DEV_PREFIX="/dev/"
planner: RDEV_PREFIX="/dev/" DUMP="/sbin/dump"
planner: RESTORE="/sbin/restore" SAMBA_CLIENT="/usr/bin/smbclient"
planner: GNUTAR="/bin/tar" COMPRESS_PATH="/usr/bin/gzip"
planner: UNCOMPRESS_PATH="/usr/bin/gzip" MAILER="/usr/bin/Mail"
planner: listed_incr_dir="/var/lib/amanda/gnutar-lists"
planner: defs: DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1"
planner: DEFAULT_TAPE_SERVER="localhost"
planner: DEFAULT_TAPE_DEVICE="/dev/null" HAVE_MMAP HAVE_SYSVSHM
planner: LOCKING=POSIX_FCNTL SETPGRP_VOID DEBUG_CODE
planner: AMANDA_DEBUG_DAYS=4 BSD_SECURITY USE_AMANDAHOSTS
planner: CLIENT_LOGIN="amanda" FORCE_USERID HAVE_GZIP
planner: COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast"
planner: COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc"
planner: time 0.000: dgram_bind: socket bound to 0.0.0.0.653
READING CONF FILES...
planner: time 0.005: startup took 0.005 secs
SETTING UP FOR ESTIMATES...
planner: time 0.005: setting up estimates for localhost:/export/local
localhost:/export/local overdue 12260 days for level 0
setup_estimate: localhost:/export/local: command 0, options:
last_level -1 next_level0 -12260 level_days 0
getting estimates 0 (0) -1 (-1) -1 (-1)
planner: time 0.006: setting up estimates took 0.000 secs
GETTING ESTIMATES...
driver: pid 9315 executable /usr/lib/amanda/driver version 2.4.4p1
driver: send-cmd time 0.004 to taper: START-TAPER 20030726
driver: started dumper0 pid 9317
driver: started dumper1 pid 9318
driver: started dumper2 pid 9319
taper: pid 9316 executable taper version 2.4.4p1
taper: page size is 4096
taper: buffer size is 32768
taper: buffer[00] at 0x4020d000
taper: buffer[01] at 0x40215000
taper: buffer[02] at 0x4021d000
taper: buffer[03] at 0x40225000
taper: buffer[04] at 0x4022d000
taper: buffer[05] at 0x40235000
taper: buffer[06] at 0x4023d000
taper: buffer[07] at 0x40245000
taper: buffer[08] at 0x4024d000
taper: buffer[09] at 0x40255000
taper: buffer[10] at 0x4025d000
taper: buffer[11] at 0x40265000
taper: buffer[12] at 0x4026d000
taper: buffer[13] at 0x40275000
taper: buffer[14] at 0x4027d000
taper: buffer[15] at 0x40285000
taper: buffer[16] at 0x4028d000
taper: buffer[17] at 0x40295000
taper: buffer[18] at 0x4029d000
taper: buffer[19] at 0x402a5000
taper: buffer structures at 0x402ad000 for 240 bytes
changer: opening pipe to: /usr/lib/amanda/chg-zd-mtx -info
dumper: dgram_bind: socket bound to 0.0.0.0.656
dumper: pid 9317 executable dumper0 version 2.4.4p1, using port 656
dumper: dgram_bind: socket bound to 0.0.0.0.657
dumper: pid 9318 executable dumper1 version 2.4.4p1, using port 657
dumper: dgram_bind: socket bound to 0.0.0.0.658
dumper: pid 9319 executable dumper2 version 2.4.4p1, using port 658
changer: got exit: 0 str: 2 18 1
changer: opening pipe to: /usr/lib/amanda/chg-zd-mtx -slot current
changer: got exit: 0 str: 2 /dev/nst0
planner: time 5.029: got result for host localhost disk /export/local: 0 ->
2865750K, -1 -> -1K, -1 -> -1K
planner: time 5.030: getting estimates took 5.023 secs
FAILED QUEUE: empty
DONE QUEUE:
0: localhost /export/local
ANALYZING ESTIMATES...
pondering localhost:/export/local... next_level0 -12260 last_level -1 (due
for level 0) (new disk, can't switch to degraded mode)
curr level 0 size 2865750 total size 2874210 total_lev0 2865750
balanced-lev0size 204696
INITIAL SCHEDULE (size 2874210):
localhost /export/local pri 12261 lev 0 size 2865750
DELAYING DUMPS IF NEEDED, total_size 2874210, tape length 300429312 mark
2788
delay: Total size now 2874210.
PROMOTING DUMPS IF NEEDED, total_lev0 2865750, balanced_size 204696...
planner: time 5.030: analysis took 0.000 secs
GENERATING SCHEDULE:
--------
ENDFLUSH
DUMP localhost fffffeff9ffe0f /export/local 20030726 12261 0 1970:1:1:0:0:0
2865750 95525
--------
reserving 0 out of 0 for degraded-mode dumps
driver: start time 5.021 inparallel 3 bandwidth 1000000000 diskspace 0 dir
OBSOLETE datestamp 20030726 driver: drain-ends tapeq LARGESTFIT big-dumpers ttt
taper: slot 2: date 20030726 label amanda-normal-1 (active tape)
changer: opening pipe to: /usr/lib/amanda/chg-zd-mtx -slot next
changer: got exit: 0 str: 3 /dev/nst0
taper: slot 3: date 20030717 label amanda-normal-2 (first labelstr match)
taper: read label `amanda-normal-2' date `20030717'
taper: wrote label `amanda-normal-2' date `20030726'
driver: result time 100.388 from taper: TAPER-OK
driver: state time 100.388 free kps: 1000000000 space: 0 taper: idle
idle-dumpers: 3 qlen tapeq: 0 runq: 1 roomq: 0 wakeup: 10221 driver-idle:
start-wait
driver: interface-state time 100.388 if : free 1000000000
driver: hdisk-state time 100.388
$ sudo -u amanda cat /var/log/amanda/log
DISK planner localhost /export/local
START planner date 20030726
INFO planner Adding new disk localhost:/export/local.
START driver date 20030726
FINISH planner date 20030726
STATS driver startup time 5.021
START taper datestamp 20030726 label amanda-normal-2 tape 0
So, I have no idea how to proceed. Does anyone know what
might be causing Amanda to hang? Any more information
needed?
Thanks for any help.
|
