Amanda-Users

timeouts

2009-09-21 13:05:06
Subject: timeouts
From: Brian Cuttler <brian AT wadsworth DOT org>
To: amanda-users AT amanda DOT org, Chris Knight <knight AT wadsworth DOT org>
Date: Mon, 21 Sep 2009 12:50:58 -0400
I'm running 2.6.1 on Solaris 10 for both client and server
(client != server), one is x86, the other sparc.

We have, er, 60+ DLE on the client, finsen, I don't know why
we get dumper retries or estimate timeouts, and the badly
formatted response is new behavior.

I'm not too concerned about the "strange" errors on my active
web cache servers, its sort of the price that you pay.

I will look for the client side logs, but its getting difficult
to find the logs I want given the number of DLEs involved.

I realize without logs debugging is extreemely difficult.

                                                thank you,

                                                Brian

----- Forwarded message from Amanda/Curie <amanda AT wadsworth DOT org> -----

Hostname: curie
Org     : Curie
Config  : curie
Date    : September 18, 2009

These dumps were to tape Curie21.
The next 2 tapes Amanda expects to use are: Curie22, Curie23.

FAILURE DUMP SUMMARY:
   finsen  /                      lev 0  FAILED "[disk /, all estimate timed 
out]"
   finsen  /export/samba/e260ikon lev 1  FAILED [too many dumper retry: 
"[request failed: timeout waiting for REP]"]
   finsen  /export/home/rreid     lev 1  FAILED [too many dumper retry: 
"[request failed: timeout waiting for REP]"]
   everest /images3                RESULTS MISSING
  planner: ERROR badly formatted response from finsen
  planner: ERROR badly formatted response from finsen
  planner: ERROR Request to everest failed: timeout waiting for ACK

STRANGE DUMP SUMMARY:
   squidzone2 /sqcache2/var        lev 1  STRANGE (see below)
   squidzone1 /sqcache1/squidguard lev 1  STRANGE (see below)
   squidzone1 /sqcache1/var        lev 0  STRANGE (see below)


STATISTICS:
                          Total       Full      Incr.
                        --------   --------   --------
Estimate Time (hrs:min)    0:14
Run Time (hrs:min)        13:04
Dump Time (hrs:min)       33:03      16:54      16:09
Output Size (meg)      445020.1   226794.8   218225.3
Original Size (meg)    680599.7   357597.6   323002.1
Avg Compressed Size (%)    61.4       56.8       65.9   (level:#disks ...)
Filesystems Dumped          117         23         94   (1:90 2:3 3:1)
Avg Dump Rate (k/s)      3829.8     3818.5     3841.5

Tape Time (hrs:min)       12:36       6:54       5:42
Tape Size (meg)        445020.0   226794.8   218225.2
Tape Used (%)              55.4       28.2       27.2   (level:#disks ...)
Filesystems Taped           117         23         94   (1:90 2:3 3:1)
   (level:#chunks ...)
Chunks Taped                117         23         94   (1:90 2:3 3:1)
Avg Tp Write Rate (k/s) 10048.0     9350.2    10893.0

USAGE BY TAPE:
  Label         Time      Size      %    Nb    Nc
  Curie21      12:36   445020M   55.4   117   117

NOTES:
  planner: disk finsen:/, estimate of level 0 timed out.
  planner: disk finsen:/, estimate of level 1 timed out.
  planner: Incremental of finsen:/export/samba/sciteam bumped to level 3.
  planner: Incremental of finsen:/export/home/amanda bumped to level 2.
  planner: Full dump of trel:/Users promoted from 4 days ahead.
  planner: Full dump of mailserv:/usr2 promoted from 3 days ahead.
  planner: Full dump of squidzone2:/sqcache2/squidguard promoted from 4 days 
ahead.
  planner: Full dump of squidzone1:/sqcache1/var promoted from 4 days ahead.
  planner: Full dump of muninn:/var promoted from 4 days ahead.
  planner: Full dump of h220:/opt promoted from 4 days ahead.
  planner: Full dump of ldap1:/export/home promoted from 4 days ahead.
  planner: Full dump of huginn:/ promoted from 4 days ahead.
  planner: Full dump of ngato:/ promoted from 4 days ahead.
  planner: Full dump of curie:/thump/source promoted from 5 days ahead.
  taper: tape Curie21 kb 455700511 fm 117 [OK]
  big estimate: lyra /db4 1
                est: 1019M    out 811M
  big estimate: finsen /export/home/winsrvbu 2
                est: 6252M    out 4247M
  small estimate: finsen /export/samba/iso 1
                  est: 3018M    out 6283M


DUMP SUMMARY:
                                       DUMPER STATS               TAPER STATS 
HOSTNAME     DISK        L ORIG-MB  OUT-MB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
-------------------------- ------------------------------------- -------------
c110         /           0    1165     619   53.1   10:32 1002.9   0:52 12289.2
c110         /opt        1       0       0    1.6    0:41    0.1   0:00  795.5
curie        /           0    7798    3108   39.9   13:23 3965.5   4:29 11826.5
curie        /export     1   81736   65105   79.7  127:33 8711.8  87:56 12634.9
curie        -xport/flar 1   15509   11076   71.4   28:24 6654.1  21:45 8687.9
curie        -ump/source 0   36744   30146   82.0   71:00 7246.8  41:22 12437.0
curie        -p/vmfs-bak 1       0       0    --     0:01    0.8   0:00   71.3
dnix         /dev/sda1   1    1421    1421    --     6:00 4040.9   1:57 12408.6
dnix         /dev/sda4   1   11138   11138    --    42:28 4476.2  32:27 5857.3
everest      /images3      MISSING -------------------------------------------
finsen       /             MISSING -------------------------------------------
finsen       /export     1   34548   19116   55.3  149:04 2188.5  26:01 12538.1
finsen       -xport/home 1       0       0    3.3    0:01    2.2   0:00   69.7
finsen       -me/agaupel 1       0       0    --     0:01    4.8   0:00  967.0
finsen       -home/along 1       0       0    --     0:01    4.8   0:00  968.4
finsen       -ome/amanda 2       5       5   98.8    0:02 2581.9   0:00 12675.1
finsen       -e/bhowland 1   10907    9069   83.2   41:50 3700.1  12:21 12536.6
finsen       -e/brainard 1       0       0    --     0:01    4.6   0:00  951.7
finsen       -me/cbryant 1       0       0    --     0:01    4.9   0:00  962.1
finsen       -home/cchen 1       0       0    --     0:01    4.4   0:00  985.2
finsen       -/home/dkim 1       0       0    --     0:01    4.6   0:00  961.4
finsen       -me/dmuller 1   34662   30289   87.4   97:39 5293.4  41:24 12486.3
finsen       -me/dschley 1       0       0    --     0:01    4.8   0:00  962.1
finsen       -ome/eparks 1       0       0    --     0:01    4.6   0:00  954.9
finsen       -e/gopinath 1       0       0    --     0:01    5.0   0:00  979.8
finsen       -me/hilaire 1       0       0    --     0:01    4.9   0:00  971.2
finsen       -e/hredmond 1       0       0    --     0:01    4.9   0:00  958.7
finsen       -home/jking 1       0       0    --     0:01    4.9   0:00  958.4
finsen       -me/kboulay 1       0       0    --     0:01    4.5   0:00  987.7
finsen       -me/khughes 1       0       0    --     0:01    4.8   0:00  967.3
finsen       -ome/knight 1       0       0    --     0:01    1.6   0:00   84.1
finsen       -e/ldechris 1       0       0    --     0:01    4.7   0:00  984.8
finsen       -e/mdejesus 1       0       0    --     0:01    4.7   0:00  979.8
finsen       -e/mganapat 1       0       0    --     0:01    4.7   0:00  985.6
finsen       -e/mkhvedel 1       0       0    --     0:01    4.6   0:00  989.2
finsen       -e/mravalli 1       0       0    --     0:01    4.1   0:00  980.9
finsen       -e/mzakalas 1       0       0    --     0:01    4.6   0:00  986.6
finsen       -ome/ntokas 1       0       0    --     0:01    4.4   0:00  977.3
finsen       -ome/nwatch 1       0       0    --     0:01    4.0   0:00 1027.9
finsen       -e/odonnell 1       0       0    --     0:01    4.0   0:00 1132.0
finsen       -me/rfisher 1       0       0    --     0:01    4.5   0:00  957.7
finsen       -me/rhannum 1       0       0    --     0:01    4.5   0:00  958.3
finsen       -e/robinson 1       0       0    --     0:01    2.6   0:00  954.8
finsen       -me/rpeters 0       0       0    --     0:01    4.6   0:00  980.9
finsen       -home/rreid 1 FAILED --------------------------------------------
finsen       -home/rsync 0       0       0    --     0:01    3.8   0:00  996.5
finsen       -e/sgambati 1       0       0    --     0:01    3.6   0:00  986.0
finsen       -me/shadmin 1       0       0    --     0:01    4.2   0:00  991.7
finsen       -e/shamilto 1       0       0    --     0:01    4.5   0:00  985.6
finsen       -e/shelming 1       0       0    --     0:01    3.0   0:00  942.5
finsen       -ome/shoran 1       0       0    --     0:01    3.4   0:00  913.8
finsen       -ome/skeogh 1       0       0    --     0:01    2.8   0:00  948.0
finsen       -e/spersaud 1       0       0    --     0:01    4.4   0:00  936.2
finsen       -ome/tomadm 1       0       0    --     0:01    3.5   0:00 1009.4
finsen       -e/topilina 1       0       0    --     0:01    3.1   0:00  942.9
finsen       -e/winsrvbu 2    7988    4247   53.2   19:50 3656.0   5:56 12225.1
finsen       -ome/ycheng 1       0       0    --     0:01    3.3   0:00  928.4
finsen       -port/samba 1       0       0    --     0:01    2.9   0:00  418.3
finsen       -amba/aiadm 1       1       0   23.6    0:02   74.0   0:00 2170.2
finsen       -a/bdlshare 1   11718    9506   81.1   47:10 3439.0  13:24 12111.7
finsen       -a/bdlsharw 1       0       0    --     0:01    1.7   0:00   64.5
finsen       -samba/btrp 1       0       0    --     0:01    1.7   0:00   61.5
finsen       -mba/citrix 1   18917   17651   93.3   64:52 4644.3  24:08 12479.8
finsen       -a/clepgete 1      59      30   51.6    0:16 1902.0   0:02 12528.7
finsen       -ba/dihivpt 1    1168     942   80.7    5:03 3188.1   1:17 12528.7
finsen       -a/e260ikon 1 FAILED --------------------------------------------
finsen       -a/envirbio 1       0       0    --     0:01    1.6   0:00   64.0
finsen       -a/histolab 1       0       0    --     0:01    1.6   0:00   66.4
finsen       -/samba/iso 1    6524    6283   96.3   21:05 5086.8   9:06 11772.9
finsen       -mba/lndata 1     218      58   26.6    0:25 2342.4   0:05 12556.7
finsen       -amba/lnpkg 0     265     189   71.3    0:54 3585.3   0:15 12567.0
finsen       -a/mbelfort 1       0       0    --     0:01    1.7   0:00   62.9
finsen       -/mcdonough 1       0       0    --     0:01    1.6   0:00   66.1
finsen       -ba/parasit 0       0       0   17.5    0:02   11.7   0:00 7559.8
finsen       -/resboards 1       0       0    --     0:01    1.6   0:00   63.4
finsen       -ba/schmura 1       0       0    --     0:01    1.6   0:00   61.8
finsen       -ba/sciteam 3    6719    5899   87.8   25:35 3934.6   8:50 11401.9
finsen       -a/seegimag 1   18566   13763   74.1   98:23 2387.4  20:27 11483.8
finsen       -ba/sharcor 0       0       0    --     0:01    1.7   0:00   63.1
finsen       -a/sloanpis 2       6       2   30.0    0:02  937.0   0:00 9754.8
finsen       -mba/tissue 1       0       0    --     0:01    1.6   0:00   61.8
finsen       -ba/wcspace 0       3       1   20.1    0:01  527.2   0:00 12130.8
gatem        /           1     807     807    --     5:39 2439.7   2:54 4743.5
gatem        /usr1       1       1       1    --     0:39   28.5   0:00 12523.3
h220         /           1      85       5    5.4    5:15   14.8   0:01 6747.7
h220         /opt        0    2710     993   36.6   22:23  756.8   1:25 11951.6
huginn       /           0   42050   42050    --   114:20 6276.6  58:55 12182.2
ldap1        /           1     784      38    4.8    2:15  285.3   0:04 9700.1
ldap1        -xport/home 0    6069    5168   85.2   21:39 4073.1  61:05 1444.0
ldap1        /usr1       1     553      46    8.2    2:05  372.2   0:13 3680.3
lyra         /           0   31231   15220   48.7  124:23 2088.4  20:49 12476.5
lyra         /3rdparty   1      15       1    5.0    1:58    6.3   0:00 11996.5
lyra         /db1        0   28871   10635   36.8   98:18 1846.4  14:37 12419.7
lyra         /db2        1   18888    2151   11.4   38:09  962.2   2:56 12530.3
lyra         /db3        1   22254    3504   15.7   49:34 1206.8   5:38 10625.6
lyra         /db4        1    8197     811    9.9   19:59  692.6   1:08 12139.1
lyra         -port/home0 1       2       0    8.3    0:22    6.4   0:00 10082.7
lyra         /ndevelop   1      45       5   10.7    2:59   27.8   0:00 12448.2
lyra         /space      1     659     504   76.5    9:20  922.4   0:42 12282.7
mailserv     /           1    1170     371   31.7    4:40 1354.4   3:38 1742.0
mailserv     /usr1       1    4166    1540   37.0   11:33 2276.1   7:13 3638.5
mailserv     /usr2       0  156910   88370   56.3  385:36 3911.3 120:13 12544.9
muninn       /           1     559     167   29.8   17:16  164.8   0:14 12417.2
muninn       /var        0    3170     581   18.3   14:50  668.1   1:43 5782.4
ngato        /           0    6132    2431   39.6   14:17 2903.3   4:09 10015.1
nlascar      /           1       0       0   30.7    0:12    9.5   0:00 10968.6
nlascar      /boot       1       0       0    --     0:01    1.8   0:00  449.6
nlascar      /data       0     833    1658  199.0    4:49 5872.2   5:37 5037.2
nlascar      /var        1     162      47   29.1    0:34 1413.1   0:04 12421.9
panther      /           0    2586    1051   40.7   53:52  333.1   1:26 12529.7
panther      /data       1       0       0    --     0:01    0.7   0:00  171.9
pavlov       /           0   15157    9730   64.2   26:53 6176.3  52:15 3178.0
squidone     /           1     203      25   12.1    6:09   68.1   0:02 12216.0
squidtwo     /           1      40       4   10.4    4:58   14.5   0:00 12537.1
squidzone1   -squidguard 1    1200    1200    --     2:24 8508.8   8:03 2542.9
squidzone1   -cache1/var 0    9263    9263    --    20:46 7611.2  13:23 11813.7
squidzone2   -squidguard 0    3383    3383    --     6:41 8648.2   8:16 6988.3
squidzone2   -cache2/var 1     480     480    --     0:50 9867.3   0:39 12612.7
trel         /Users      0    3258    2201   67.5    8:57 4192.9   3:08 11995.1
trel         /trelAQ     1     918     918    --     5:18 2956.6   1:16 12368.2
trel         /trelRZ     1       2       2    --     0:18  102.0   0:00 12335.2

(brought to you by Amanda version 2.6.1p1)

----- End forwarded message -----
---
   Brian R Cuttler                 brian.cuttler AT wadsworth DOT org
   Computer Systems Support        (v) 518 486-1697
   Wadsworth Center                (f) 518 473-6384
   NYS Department of Health        Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure.  It
is intended only for the addressee.  If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments.  Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.



<Prev in Thread] Current Thread [Next in Thread>