Evan,
On Thu, Dec 07, 2006 at 04:52:52PM -0600, Evan Harris wrote:
>
> On Thu, 7 Dec 2006, Brian Cuttler wrote:
>
> >I've assumed that the spindle the amanda work area is on was dedicated
> >to the work area and not split, I know nothing about the architecture
>
> That is correct. That disk is only used by amanda.
>
> >of the specific system bus. If you are able to get through put with
> >amdump than I'd guess its not a bus issue to either the tape nor the
> >drive. That leaves perhaps CPU or sharing the bus access to the disk
> >drive ?
>
> It is almost certainly an issue (as I think I stated previously) with the
> dumper writes and drive seeks between reading from and writing to the same
> spindle degrading the holding disk throughput enough to cause the disk to
> be unable to have sufficient bandwidth to keep the tape drive streaming.
>
> >But I'm going to have to step back at this point, since it sounds like
> >a problem specific to a system that I'm not readily able to visualize.
>
> Here's an attempt to explain the issue better:
>
> Let's say my holding disk has a throughput of about 20MB/sec. I think
> everyone can agree that with 20MB/sec of throuhput from the drive and
> 11MB/sec to the tape, there should be no problem keeping the tape drive
> streaming, assuming there isn't a CPU or bus bottleneck, and that there is
> no additional disk contention for the same spindle. We've got 9MB/sec of
> left-over bandwidth, which is plenty.
>
> Now, the tech specs on drive used for my holding disk say that the average
> seek time is 9ms. That means for every seek done (on average), I lose
> 184KB of bandwidth.
>
> Ok, so my disk has a total of 20MB/sec bandwidth. For the purposes of this
> discussion, lets assume that the read and write bandwidths are equal. Lets
> also say that the dumper (for the purposes of this discussion, we have
> limited amanda to only allow one dumper at a time, not several running
> concurrently, in order to avoid additional complications) is consuming
> about 6MB/sec on average (reasonable assumption for dumps coming from a
> remote machine on a 100mbit network backbone). That leaves 14MB/sec left.
> To keep the tape drive streaming (from your own specs for uncompressed
> writes) requres 11MB/sec. That leaves us with 3MB/sec.
>
> Now say that interleaving the reads and writes to the disk is causing 25
> seeks per second. 25 seeks * 184KB/sec is 4.6MB/sec of throughput lost to
> seek overhead, and you can see that we are now have a deficit of 1.6MB/sec.
> From this example, you should be able to see that it is impossible to keep
> the tape drive streaming unless either the number of seeks or the amount of
> data being written by the dumper (or both) is reduced.
>
> Does that make the issue clear? The only way to reliably fix the
> shoe-shine problem (absent replacement of hardware) is to keep the dumper
> from using the disk while taper is using it, or being able to throttle the
> dumper's use of the disk drive to leave enough bandwidth to keep the taper
> from starving. And it is appears that neither method of limiting is
> currently supported by Amanda, which seems very surprising to me.
Are there performance enhancements available by using a different
type of file system on the amanda work area drive ?
Anything else I can think of involves new hardware, mirrors, multiple
amanda work areas... I have no idea how to set (active dumpers + tapers) = 1
> Evan
>
> >>Evan
> >>
> >>On Thu, 7 Dec 2006, Brian Cuttler wrote:
> >>
> >>>On Wed, Dec 06, 2006 at 04:33:16PM -0600, Evan Harris wrote:
> >>>>
> >>>>Debian testing, SDLT 110/220, Intel P4 2.8Ghz 3gig RAM. Amanda
> >>>>2.5.1p1-2.
> >>>>
> >>>>The holding disk is a dedicated PATA IDE drive (master) on its own
> >>>>cable/bus (no slave). The tape drive is on its own SCSI bus (no other
> >>>>devices). Bonnie tests on the IDE drive give roughly 20MB/sec, and I
> >>>>have
> >>>>no trouble keeping the tape drive streaming using dd from the IDE drive
> >>>>to
> >>>>the tape drive. Amanda tapetest speed on the tapedrive came in right at
> >>>>10MB/sec.
> >>>
> >>>I have an SDLT 220 also, mine being set for high density but without
> >>>HW compression, I specifically perform SW compression (client side though
> >>>in this case the client==server). I am also peaking about 10MB/sec per
> >>>the amdump report.
> >>>
> >>>I have to look at the tape specs... Sun online docs show a sustained
> >>>transfer rate of 11 MB/Sec, so you and I are both doing pretty well
> >>>in that dept.
> >>>
> >>>Tell me again why you feel your drive is shoe-shining ?
> >>>Are the specs for your particular drive substantially different ?
> >>>
> >>>
> >>>>I'm currently testing on a standalone SDLT drive first. But the final
> >>>>config will be the same type of drive in an ADIC changer.
> >>>>
> >>>>Evan
> >>>>
> >>>>On Wed, 6 Dec 2006, Brian Cuttler wrote:
> >>>>
> >>>>>Evan,
> >>>>>
> >>>>>What OS, type of tape, type of drive, HW platform ?
> >>>>>
> >>>>>I'm unfamiliar with a parameter to do what your asking
> >>>>>for for good measure, what version of Amanda ?
> >>>>>
> >>>>>Actually, how did you determine that a drive within a changer
> >>>>>was shoe-shining ? What else shares the bus with the tape and
> >>>>>with the amanda work area drive(s) ?
> >>>>>
> >>>>>On Wed, Dec 06, 2006 at 02:34:31PM -0600, Evan Harris wrote:
> >>>>>>
> >>>>>>I'm having a problem with my tape drive shoe-shining because the
> >>>>>>holding
> >>>>>>disk can't keep up with the tape drive if it is also being written to
> >>>>>>by a
> >>>>>>dumper. Without the extra disk seek overhead of dumpers writing to
> >>>>>>the
> >>>>>>holding disk at the same time, the holding disk should be plenty fast
> >>>>>>enough to keep the tape drive streaming.
> >>>>>>
> >>>>>>Is there any way I can force amanda to serialize the dumper/taper so
> >>>>>>that
> >>>>>>they are never run concurrently? I've already set inparallel to 1,
> >>>>>>but
> >>>>>>that only affects how many dumpers can run, not the taper.
> >>>>>>
> >>>>>>I've also tried increasing the tapebufs parameter to 8000 (256MiB) to
> >>>>>>see
> >>>>>>if that would at least let the drive stay streaming for longer
> >>>>>>periods,
> >>>>>>but
> >>>>>>if it made any difference, it wasn't significant. What thresholds
> >>>>>>does
> >>>>>>the
> >>>>>>taper use to decide when the tapebufs are filled enough to start tape
> >>>>>>motion? There doesn't seem to be any docs on that, or settings to
> >>>>>>customize.
> >>>>>>
> >>>>>>I did get a suggestion that I should just leave the tape out of the
> >>>>>>drive,
> >>>>>>let the dumpers fill the holding disk and then load the tape and run
> >>>>>>amflush, but that doesn't really work when using a changer, plus the
> >>>>>>holding disk isn't large enough for the total size of all the backups,
> >>>>>>though it can fit them one-by-one.
> >>>>>>
> >>>>>>Seems like there should be a "speed" config option for holding disks
> >>>>>>like
> >>>>>>there is for network interfaces and tape drives, so that amanda could
> >>>>>>test
> >>>>>>to see if the holding disk can't handle dumpers using the holding disk
> >>>>>>at
> >>>>>>the same time a taper is running. That seems like it'd solve the
> >>>>>>problem
> >>>>>>nicely, and even seems to fit with the scheme amanda uses for network
> >>>>>>interfaces.
> >>>>>>
> >>>>>>Thanks.
> >>>>>>
> >>>>>>Evan
> >>>>>---
> >>>>>Brian R Cuttler brian.cuttler AT wadsworth DOT org
> >>>>>Computer Systems Support (v) 518 486-1697
> >>>>>Wadsworth Center (f) 518 473-6384
> >>>>>NYS Department of Health Help Desk 518 473-0773
> >>>>>
> >>>---
> >>> Brian R Cuttler brian.cuttler AT wadsworth DOT org
> >>> Computer Systems Support (v) 518 486-1697
> >>> Wadsworth Center (f) 518 473-6384
> >>> NYS Department of Health Help Desk 518 473-0773
> >>>
> >---
> > Brian R Cuttler brian.cuttler AT wadsworth DOT org
> > Computer Systems Support (v) 518 486-1697
> > Wadsworth Center (f) 518 473-6384
> > NYS Department of Health Help Desk 518 473-0773
> >
---
Brian R Cuttler brian.cuttler AT wadsworth DOT org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773
|
