Amanda-Users

Re: Troubleshooting a slowdown problem?

2004-06-17 17:09:53
Subject: Re: Troubleshooting a slowdown problem?
From: KEVIN ZEMBOWER <KZEMBOWE AT jhuccp DOT org>
To: amanda-users AT amanda DOT org
Date: Thu, 17 Jun 2004 17:04:02 -0400
Eric and Frank, thank you very much for your detailed analysis of my problem.

I just noticed on the most recent amstatus I ran that centernet:sda5 completed, 
and admin:sda3 is now dumping to tape. I've pasted in the most recent amstatus 
to the end of this note. I guess the reason that it says just "dumping to tape" 
rather than centernet's "dumping  1332992k ( 73.56%) (2:02:34)" is that admin 
is the tapehost itself.

The holding disks on the tapehost are large, I thought:
amanda@admin:~ > df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             7.6G  5.2G  2.0G  72% /
/dev/sda1              22M  3.4M   17M  16% /boot
/dev/sdb1             8.3G  753M  7.1G  10% /var/amanda  #This is hd1 (holding 
disk 1)
/dev/sdc1              33G  4.9G   26G  16% /dumps2      # and this is hd2
shmfs                1009M     0 1009M   0% /dev/shm
amanda@admin:~ >

They don't seem to be in use at this late stage of this backup:
amanda@admin:/dumps2/amanda > du -sxh . /var/amanda/
8.0k    .
12k     /var/amanda
amanda@admin:/dumps2/amanda > 

Centernet is a low-volume web server, primarily. Even while running the backup, 
the load was less than 2. It's a 600MHz dual Pentium Dell PowerEdge 2450 with 
256MB, 100MHz RAM. The tapehost, in contrast, right now has a load of almost 4:
amanda@admin:/dumps2/amanda > uptime
  4:24pm  up 1 day, 23:30,  1 user,  load average: 3.77, 3.71, 3.57
amanda@admin:/dumps2/amanda > 

There's no firewall between centernet and the tapehost, admin. Both are inside 
our firewall. The network is switched 100Mbps Ethernet.

Here's a ps list of the amanda jobs currently running on the tapehost:
amanda@admin:/dumps2/amanda > ps aux|grep amanda
amanda    4342  0.0  0.2  2220 1044 ?        S    Jun16   0:00 /bin/sh -c 
/usr/local/sbin/amdump DailySet1 && /usr/bin/mt -f /dev/nst0 offline
amanda    4346  0.0  0.2  2228 1104 ?        S    Jun16   0:00 /bin/sh 
/usr/local/sbin/amdump DailySet1
amanda    4381  0.0  0.2  2204 1080 ?        S    Jun16   0:02 
/usr/local/libexec/driver DailySet1
amanda    4382  0.1  0.3  2780 1600 ?        S    Jun16   2:16 taper DailySet1
amanda    4383  0.4  0.2  2492 1216 ?        S    Jun16   5:12 dumper0 DailySet1
amanda    4384  0.0  0.2  2488 1208 ?        S    Jun16   1:07 dumper1 DailySet1
amanda    4385  0.0  0.2  2488 1208 ?        S    Jun16   0:28 dumper2 DailySet1
amanda    4386  0.0  0.1  2272  940 ?        S    Jun16   0:00 dumper3 DailySet1
amanda    4387  0.0  0.1  2272  940 ?        S    Jun16   0:00 dumper4 DailySet1
amanda    4388  0.1  0.2  2808 1536 ?        D    Jun16   1:50 taper DailySet1
amanda    4389  0.0  0.1  2272  940 ?        S    Jun16   0:00 dumper5 DailySet1
amanda    4390  0.0  0.1  2272  940 ?        S    Jun16   0:00 dumper6 DailySet1
amanda    4391  0.0  0.1  2272  940 ?        S    Jun16   0:00 dumper7 DailySet1
amanda    7185  0.0  0.1  2012  916 ?        S    15:52   0:00 
/usr/local/libexec/sendbackup
amanda    7187 57.4  0.1  1612  684 ?        S    15:52  24:33 /usr/bin/gzip 
--fast
amanda    7188  0.1  0.2  2348 1512 ?        S    15:52   0:03 dump 0usf 
1048576 - /dev/sda3
amanda    7189  0.7  0.3  2440 1640 ?        S    15:53   0:19 dump 0usf 
1048576 - /dev/sda3
amanda    7190  1.0  0.2  2348 1504 ?        S    15:53   0:27 dump 0usf 
1048576 - /dev/sda3
amanda    7191  1.1  0.2  2348 1504 ?        S    15:53   0:28 dump 0usf 
1048576 - /dev/sda3
amanda    7192  1.0  0.2  2348 1504 ?        S    15:53   0:27 dump 0usf 
1048576 - /dev/sda3
amanda    7248  0.0  0.1  2064  924 pts/0    S    16:07   0:00 su - amanda
amanda    7249  0.0  0.2  2612 1508 pts/0    S    16:07   0:00 -bash
amanda    7335  0.0  0.2  2440 1504 pts/0    R    16:35   0:00 ps aux
amanda    7336  0.0  0.1  1540  576 pts/0    S    16:35   0:00 grep amanda
amanda@admin:/dumps2/amanda > 

I'll send in the daily report as soon as I receive it. Normally, I would have 
interrupted amanda around 2:00pm by just killing all the amanda jobs on admin 
and running amcleanup. Then, I would put the next tape in and run amflush. This 
would complete before I needed to put the next tape in for the nightly run and 
go home. Tonight, I'll just let it run out. In addition, there's a thunderstorm 
rolling through Baltimore right now and all the lights are flickering. All the 
servers are on a UPS, but my workstation isn't.

The partitions on admin like "admin://db/c$" are actually Samba shares from an 
NT host.

Thanks, again, for all your suggestions. I won't make any changes right now, 
until you've had a chance to look at the daily report. I appreciate all your 
help.

It still hasn't ended and it's 5:03 and I'm hungry and tired, so I'm going 
home. I'll talk with you all again tomorrow.

-Kevin Zembower

amanda@admin:/dumps2/amanda > amstatus DailySet1
Using /var/log/amanda/DailySet1/amdump from Wed Jun 16 20:00:00 EDT 2004

admin://db/c$                       0   462720k finished (20:56:41)
admin://db/e$                       1       10k finished (20:16:16)
admin://db/f$                       1  2559660k finished (20:51:25)
admin://db/f$/inetsrv/webpub/images 1       30k finished (20:16:07)
admin:sda1                          0     3410k finished (20:17:09)
admin:sda3                          0  3683924k dumping to tape (15:52:43)
admin:sdb1                          0    24270k finished (20:17:05)
centernet:sda1                      0     4846k finished (20:06:03)
centernet:sda2                      0   715715k finished (2:15:47)
centernet:sda3                      0   110883k finished (20:58:42)
centernet:sda5                      0  1818166k finished (15:52:41)
centernet:sda6                      1       73k finished (20:03:37)
centernet:sda7                      0      564k finished (20:04:03)
centernet:sda9                      0    30391k finished (20:18:59)
mailinglists:hda1                   0     2198k finished (20:04:30)
mailinglists:hda2                   0   399339k finished (23:03:41)
mailinglists:hda7                   0   743827k finished (4:41:56)

SUMMARY          part      real  estimated
                           size       size
partition       :  17
estimated       :  17             11566387k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   1              3683924k           ( 31.85%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :  17  10560026k  11566387k ( 91.30%) ( 91.30%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :  16   6876102k   7882463k ( 87.23%) ( 59.45%)
7 dumpers idle  : not-idle
taper writing, tapeq: 0
network free kps:     25498
holding space   :  36474076k (100.00%)
 dumper0 busy   : 18:50:27  ( 95.06%)
 dumper1 busy   :  6:24:10  ( 32.31%)
 dumper2 busy   :  2:52:40  ( 14.52%)
   taper busy   :  1:40:59  (  8.49%)
 0 dumpers busy :  0:33:22  (  2.81%)             no-hold:  0:33:22  (100.00%)
 1 dumper busy  : 10:51:06  ( 54.75%)             no-hold: 10:51:06  (100.00%)
 2 dumpers busy :  7:57:50  ( 40.18%)  client-constrained:  5:31:58  ( 69.47%)
                                                  no-hold:  2:25:40  ( 30.49%)
                                               start-wait:  0:00:11  (  0.04%)
 3 dumpers busy :  0:26:50  (  2.26%)  client-constrained:  0:26:47  ( 99.77%)
                                               start-wait:  0:00:03  (  0.23%)
amanda@admin:/dumps2/amanda > date
Thu Jun 17 17:03:28 EDT 2004
amanda@admin:/dumps2/amanda >