Thanks for all the great ideas. I have some comments and questions.
>>From Gene's message:
The dump's are not going straight to tape which really does slow things
down. One of the way I can tell is that the final DLE dump that has been
running since midnight is spooling to disk and the tape is idle. Thank
you so much for the formating hints. I will review the compression
options, actually the dump for mbthome is without compression on today's
run. I know the dump report I included in the original post compression
was on.
>>From Jon's message:
I don't have any spindle values defined. I am not sure how best to set
them if I have multiple DLE defined all residing on a single RAID5 array
with many drives. At any rate the night before last dump run was with
maxdumps at 1 (see performance on dump statistics report), and last
night's dump run maxdumps set at 2 (see the amstatus report). I have
had maxdumps set as high as 3. and overall performance is actually
better than at 1 or 2. /home/public is seperated out in an attempt to
control the size of the dump. The /home/public is just another logical
volume in the same set of physical disk as the rest of /home. I do have
the /home fs exported for other computers to mount!
>>From Gene's message:
The performance bottle neck seems to be during dumps not while taping.
I have changed onto diffent SCSI controller and diffent disks though.
>>From Joshua's message:
The tape drive is not "shoe-shinning" now in the past is was but I have
resolve that. So I should expect to see tape speed about 60GB/sec? Do
comments about block size apply to the client or server, or taper? The
main thing that puzzles me is that the exact same contoller and physical
drive can have such diffent rates (even with only one dump per host
allowed.) This is a production machine will the bonnie results be
accurate enough with users hitting it (it is usally lightly loaded if
amanda is not running)? Will you comment on what type of values I
should expect from bonnie? Sorry about the formating problems below, I
have not had time to figure out how to fix it in Evolution.
1.)
Below are bonnie results on mbthome:/home/public this is on a percraid
controller handled as a single 300GB HDD with LVM allocating the space
to the following filesystems /home, /home/public/,
and /home/public/Ausk... The dump rate last night for this single drive
varied from 1030.7KB/sec to 9257.3 KB/sec depending on the DLE.
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %
CPU /sec %CPU
1000 31610 98.7 62640 33.1 52888 21.3 37878 100.0 933421 100.0
44261.0 192.5
2.)
Below are the bonnie results on mbthome:/amanda2 this one of the active
amanda holding disk locations. This is the same percraid controller
card as above but the file system is a LVM built on a RAID5 consisting
of a set of 4 disks.
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %
CPU /sec %CPU
1000 32104 100.0 80989 42.6 53707 23.9 18077 54.0 523912 60.7
36115.8 174.2
3.)
Below are bonnie results for mbthome:/amanda one of the holding disk
that the amanda.conf file is using. On a megaraid scsi controller
connected to external single disk volume (/dev/sdd1);
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %
CPU /sec %CPU
1000 32305 100.0 102545 50.9 39105 15.4 37431 100.0 939585
100.1 42895.4 182.3
4.)
This is for completeness only this drive is not being backed up or used
in any manner by amanda. It is on the same megaraid controller as above
but is a RAID5 with 7 physical drives connected. Performance seems much
less! Something is very wrong here!
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %
CPU /sec %CPU
1000 28147 85.9 7503 3.7 3380 1.5 37520 100.0 924322 100.0
44097.1 192.9
Regards,
Kenneth Berry
On Wed, 2007-03-07 at 13:29 -0500, Joshua Baker-LePain wrote:
> On Wed, 7 Mar 2007 at 11:45am, Kenneth Berry wrote
>
> > The computer mbthome is both the amanda server and client, it is a Dell
> > PE2650 dual Xeon processor, 2GB Ram. Tape unit is LTO3 on a dedicated
> > SCSI controller. Internal HDD's are four SCSI ultra320 10K configured
> > as hardware RAID5. On top of this 100GB RAID5 disk is LVM Volume00
> > allocated as shown below. One additional internal SCSI ultra320 10K
> > drive of 300GB capacity is allocated to LVM Volume02 and usage can be
> > seen below. All the internal HDD share a common percraid controller. I
> > have an external RAID5 set of drives on an third SCSI controller which
> > LVM Volume01 resides. It should not be a factor currently it is unused.
> > Over the past weekend I migrated all the /home data from the external
> > set of drives to the single large internal drive, thinking the external
> > drives were the problem. But this reorganization made not difference.
>
> Some quick things to look into:
>
> 1) Benchmark your various volumes with both bonnie++ (single thread
> streaming) and tiobench (multithreaded). This will give you some baseline
> numbers. You can also play with multiple instances of dd. I have no idea
> how good those perc controllers are, as Dull tends to obfuscate them
> compared to their LSI underpinnings.
>
> 2) LTO3 is *fast*. Reading from a 4 disk RAID5 should be able to keep it
> streaming, but simultaneously writing to that same RAID5 is likely to be
> *very* slow... Actually, looking at your tape write speeds in the
> original mail, they're barely where they should be. LTO3 can throttle to
> 1/2 its native speed of 80MB/s before it has to resort to
> drive/tape-lifetime-degrading stutter-stop-restart behavior. One thing
> that significantly helps is increasing your blocksize from amanda's
> standard 32KB -- I use 2MB. This requires recompiling amanda.
>
> > /dev/sdd1 141003764 36925740 96915448 28% /amanda
>
> Erm, where does sdd fit in in terms of the LVM volumes?
>
> Also, both of Jon's suggestions (spindle numbers and filesystem profile on
> /home) are good ones to look into. Again, you can do your own
> benchmarking outside of amanda to get a more controlled look at what's
> going on.
>
--
Kenneth Berry
Systems Engineer / System Administrator
Micro Beef Technologies
P.O. Box 9262
Amarillo, Texas 79105
806-372-2369
kberry AT microbeef DOT com
The information contained in or attached to this e-mail message is
intended only for the confidential use of the named individual(s) above.
If you are not the named recipient of this message you are hereby
notified that you have received this document and its attachments in
error and that review, dissemination, or copying of this message is
prohibited. If you have received this message in error, please notify
Micro Beef Technologies immediately. Thank you.
|
