Hello list,
I have been working on a performance problem for about a month and I am
out of ideas. I hope someone can help me. What is happening is that
backups of large DLE that start very early seem to dump at a very slow
rate. Where as dumps of DLE on the same host, physical drive, etc. that
start hours latter dump at an expected rate. The amstatus snapshot
below was taken at 9:30:00. The backups start at midnight and notice
that the dump rate for mbthome:/home/public is about 830K/sec. But DLE
mbthome:/home_a_i which has been running for 21 mins. is at a dump rate
of about 9500K/sec.
-------------------------------------------------
mbthome:/ 0 51860k finished (0:32:54)
mbthome:/boot 0 8431k finished (0:29:27)
mbthome:/home/public 0 46747710k dumping 26966780k
( 57.69%) (0:28:31)
mbthome:/home_a_i 0 20529050k dumping 11973280k
( 58.32%) (9:09:34)
mbthome:/home_j_k 0 39011140k writing to tape (9:09:34)
mbthome:/home_l_o 0 13542210k wait for dumping
mbthome:/home_p 0 781310k finished (1:01:03)
mbthome:/home_q_z 0 10639260k wait for dumping
mbthome:/home_rest 0 10k finished (0:28:59)
mbthome:/usr 0 606581k finished (0:58:04)
mbthome:/var 0 330491k finished (0:38:43)
-------------------------------------------------
I have changed many things and not found the solution yet. I have moved
all the data from an hardware RAID5 to a single large drive on a
different SCSI controller. Played with maxdumps parameter it is set at
two for the above snap shot, for the data below the maxdump was set 1
(default). Top utility shows very high CPU sys levels which settles
when dumps are finished, are mine higher than expected when amdump is
running?
-------------------------------------------------
top - 09:48:04 up 2 days, 18:43, 2 users, load average: 5.89, 5.49,
5.37
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
Cpu0 : 9.3% us, 61.3% sy, 2.3% ni, 0.0% id, 20.9% wa, 0.0% hi,
6.3% si
Cpu1 : 1.0% us, 91.1% sy, 0.7% ni, 0.0% id, 6.0% wa, 0.0% hi,
1.3% si
Mem: 2075988k total, 2069940k used, 6048k free, 14444k buffers
Swap: 1020116k total, 0k used, 1020116k free, 1934516k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21343 amanda 16 0 2716 948 1780 S 89.8 0.0 197:24.39 dumper
21344 amanda 16 0 2976 1096 1868 S 18.5 0.1 76:27.43 dumper
28982 amanda 16 0 3028 804 1664 S 13.7 0.0 0:18.33 sendbackup
21341 amanda 15 0 2728 1536 2432 D 12.0 0.1 20:17.89 taper
28984 root 16 0 6660 1900 4304 D 10.1 0.1 0:14.84 tar
21342 amanda 15 0 2720 1532 2432 S 7.5 0.1 10:57.67 taper
28985 amanda 16 0 2532 420 1372 S 4.6 0.0 0:06.96 tar
-------------------------------------------------
Finally, so you can see the data for a full backup run I have included
below the report from last night's backup run. Notice that most DLE
that have long dump times are dumped at a very slow rate. I am sure I
didn't include something that will be need to solve this problem, please
email me back and I will provide the requested info.
-------------------------------------------------
DUMP SUMMARY:
DUMPER STATS TAPER
STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS
KB/s
-------------------------- ---------------------------------
------------
mbtas00 / 0 2177990 816539 37.5 8:121659.0
1:0313057.1
mbtas00 /back01 0 138367104785625 34.6 28:372786.4
6:2612392.8
mbtas00 /boot 0 10880 6346 58.3 0:023898.6
0:022919.1
mbtas00 /opt 0 182746208971682 49.1 50:242966.9
6:1423968.4
mbtas00 /var 0 165760 22406 13.5 0:35 637.0
0:0210429.8
mbtas01 / 0 34362501438229 41.9 7:043392.3
1:5912131.7
mbtas01 /boot 0 10880 6346 58.3 0:024023.7
0:041659.7
mbtas01 /opt 0 3140260015084957 48.0 64:583869.4
15:1716454.1
mbtas01 /u 0 10 1 10.0 0:00 0.0 0:01
0.9
mbtas01 /var 0 221740 29565 13.3 0:201488.1
0:038748.2
mbtdb01 / 0 2090070 806698 38.6 4:432848.7
0:1748362.3
mbtdb01 /boot 0 10880 6344 58.3 0:015897.3
0:023488.9
mbtdb01 /opt 0 235088507524669 32.0 29:324246.4
2:5343470.9
mbtdb01 -ata/arch01 0 268880407923108 29.5 30:494285.1
3:0243647.0
mbtdb01 -ata/arch02 0 268880407923108 29.5 30:534275.3
3:0343404.1
mbtdb01 -ata/back01 0 7335320819585192 26.7 394:16 827.9
24:0613543.6
mbtdb01 -ata/back02 0 7054363217685384 25.1 70:474163.9
5:5549858.2
mbtdb01 -ata/back03 0 7045140018625938 26.4 190:151631.7
12:5923901.9
mbtdb01 -ata/back04 0 7047288018959600 26.9 164:061925.6
8:1838060.7
mbtdb01 -ata/data24 0 225760 79360 35.2 0:164897.4
0:0157126.9
mbtdb01 -ata/data25 0 2627960 801797 30.5 2:574527.8
0:2040786.2
mbtdb01 /var 0 178450 22837 12.8 0:141618.9
0:0115461.3
mbtdbmon / 0 2933423014565741 49.7 329:34 736.6
14:3916575.8
mbtdbmon /boot 0 26240 8218 31.3 0:024734.5
0:023558.7
mbtddb01 / 0 124253807549686 60.8 207:27 606.6
9:4412921.1
mbtddb01 /boot 0 32340 18843 58.3 0:082280.6
0:028575.3
mbtdev / 0 200110 96922 48.4 0:273527.7
0:0426065.5
mbtdev /boot 0 6100 3814 62.5 0:016704.0
0:012595.2
mbtdev /opt 0 91523407164046 78.3 19:386080.9
9:0513156.2
mbtdev /usr 0 1861820 695789 37.4 3:383187.3
0:5512582.7
mbtdev /var 0 286040 66504 23.2 0:213224.4
0:0232729.4
mbthome / 0 162160 51489 31.8 0:381345.5
0:0228477.5
mbthome /boot 0 9020 8432 93.5 0:023672.4
0:023553.0
mbthome -ome/public 0 4674511237080424 79.3 437:571411.1
25:5423858.1
mbthome /home_a_i 0 2096223015266312 72.8 63:244013.7
10:1824709.6
mbthome /home_j_k 0 3901114021733976 55.7 106:023416.2
14:4224635.9
mbthome /home_l_o 0 1354221010397536 76.8 41:004226.0
6:5325175.6
mbthome /home_p 0 781120 358604 45.9 1:553108.2
0:1133516.1
mbthome /home_q_z 0 106384606287909 59.1 29:233565.9
3:0534043.1
mbthome /home_rest 0 10 1 10.0 0:00 0.0 0:01
0.8
mbthome /usr 0 1628430 606585 37.2 5:371800.7
0:1832936.3
mbthome /var 0 589220 312805 53.1 1:462951.8
0:1128095.3
mbtns1 / 0 109530 41446 37.8 0:331261.4
0:0220104.2
mbtns1 /boot 0 5640 3368 59.7 0:021926.8
0:021594.6
mbtns1 /usr 0 1377340 520768 37.8 17:16 502.5
0:4212254.3
mbtns1 /var 0 432520 284235 65.7 5:54 802.2
0:1716970.1
mbtns2 / 0 244140 132484 54.3 0:552427.6
0:1111976.6
mbtns2 /boot 0 5050 4060 80.4 0:015025.0
0:021927.8
mbtns2 /opt 0 103040 34858 33.8 0:122896.9
0:0214753.5
mbtns2 /usr 0 1462440 595361 40.7 20:42 479.4
0:4712767.9
mbtns2 /var 0 1250850 603571 48.3 28:12 356.7
0:4114751.6
mbtpas02 / 0 40305402190711 54.4 79:56 456.8
2:2714904.4
mbtpas02 /boot 0 10260 8524 83.1 0:032582.8
0:023824.2
mbttdb02 / 0 30578601398209 45.7 11:342014.6
1:0023217.1
mbttdb02 /boot 0 26950 22452 83.3 0:073279.7
0:038598.5
mbttdb03 / 0 39120702074878 53.0 16:512053.0
2:3013841.5
mbttdb03 /boot 0 20200 16808 83.2 0:053122.1
0:036549.9
mbtts00 / 0 1962845010877320 55.4 96:321878.0
15:1111940.7
mbtts00 /boot 0 10780 6243 57.9 0:022998.0
0:022886.2
mbtts01 / 0 2344288011531924 49.2 113:191696.1
7:4025089.6
mbtts01 /boot 0 10780 6243 57.9 0:07 881.1
0:032456.7
mbtvm01 / 0 1268570 495797 39.1 2:263406.7
1:107090.0
mbtvm01 /boot 0 25150 22218 88.3 0:046238.0
0:029008.5
(brought to you by Amanda version 2.4.4p2)
-------------------------------------------------
--
Kenneth Berry
Systems Engineer / System Administrator
Micro Beef Technologies
P.O. Box 9262
Amarillo, Texas 79105
806-372-2369
kberry AT microbeef DOT com
The information contained in or attached to this e-mail message is
intended only for the confidential use of the named individual(s) above.
If you are not the named recipient of this message you are hereby
notified that you have received this document and its attachments in
error and that review, dissemination, or copying of this message is
prohibited. If you have received this message in error, please notify
Micro Beef Technologies immediately. Thank you.
|
