On 2006-07-11 14:13, Jerlique Bahn wrote:

Just a thought: Were you sitting next to the tape drive while running the
above test? Did you see it stopping and re-starting over and over...?

Yes it is starting and stopping at frequent intervals. I recompiled amanda
with --with-buffered-dump to try and fix this but it didn't work.

First make really sure you are using a holdingdisk.

If you have dumps that bypass the holdingdisk, because it is too small
for that DLE, be sure to specify a "split_diskbuffer" directory and a
tapesplit_size of 1 or 2 Gigabyte. Making it larger can result in mmap()
failing for that large file, resulting in a small buffer sized by
the "fallback_splitsize", by default 10 Mbyte -- increase that too!

For measuring how fast your disk can write:
    gendata -v > somefile
    ...  watch the speed ...
    ^C  when seen enough

Over the network:

    first on the receiving host:
    $ nc -l -p 1234 > /dev/null

    Then on the sending host:
    $ gendata -v | nc somehost.example.com 1234

    or together with gzip:
    $ gendata -v -c | gzip | somehost.example.com 1234

etc.

Or writing to tape (attention:  this destroys all data on the tape)

    $ gendata -v | dd obs=32k of=/dev/st0
    ... watch it...
    ^C

Or writing to tape, which is using hardware compression, to notice
the speed difference in bytes we have to feed the tape:

    $ gendata -v -C | dd obs=32k of=/dev/st0

--
Paul Bijnens, xplanation Technology Services        Tel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM    Fax  +32 16 397.512
http://www.xplanation.com/          email:  Paul.Bijnens AT xplanation DOT com
***********************************************************************
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out          *
***********************************************************************

#!/usr/bin/perl

# (c) 1996 Paul Bijnens
# This program may be freely distributed.

# generate stream of random characters that can not be compressed
#       (compress will do about -32% on the outputstream)
# give feedback


# bug: even without opt_n it will stop after MAXINT Mbyte

# When using perl 4, yes this program is that old :-)
#   do "getopts.pl" || die "Can't include getopts.pl";
#   do Getopts('qvcCn:b:')  ||  die <<'EOS';


use Getopt::Std;


getopts('qvcCZn:b:')  ||  die <<'EOS';
Usage: gendata [-vcCq] [-b #] [-n #]
  -q    be quiet, no summary at end
  -v    verbose - give feedback every 10MB (time & bytes)
  -c    generate compresseable data (compress: about 42% compression)
  -C    generate very compresseable data (compress: about 85% compression)
  -Z    generate a sequence of ascii zero only
  -b #u output block size (def. 5k, max 1Mbyte); unit can be empty, "k" or "M"
  -n #  stop after # Mbyte outputbytes
EOS

binmode(STDOUT, ":raw")  if $[ >= v5.8.0;

$lagspan = 5;

$n = int($opt_n);

$obs = 0;
if ($opt_b) {
    ($obs, $unit) = ($opt_b =~ /^([\d.]+)(.*)/);
    if ($unit eq "") {
        ; # ok
    } elsif ($unit =~ /^[kK]$/) {
        $obs *= 1024;
    } elsif ($unit =~ /^[mM]$/) {
        $obs *= 1024 * 1024;
    } else {
        die "$0: Bad unit in output block size\n";
    }
}

$obs = 5 * 1024  if ($obs <= 0);

#
# generate some data in $buffer
#

srand();

for ($i = 0; $i < 32*1024; $i++) {
    $buf1 .= chr(rand(256));
    $buf2 .= chr(rand(256)) unless ($opt_C);
    $buf3 .= chr(rand(256)) unless ($opt_C);
    $buf4 .= chr(rand(256)) unless ($opt_C);
}

if ($opt_Z) {
    $buf = "\x00" x (8*32*1024);
} elsif ($opt_C) {
    $buf = unpack("b*", $buf1);
} elsif ($opt_c) {
    $buf = unpack("h*", $buf1 . $buf3 . $buf2 . $buf4);
} else {
    $buf = $buf1 . $buf3 . $buf2 . $buf4;
    $buf .= $buf;
}

undef $buf1;
undef $buf2;
undef $buf3;
undef $buf4;

$buf .= $buf;
$buf .= $buf;

$obs = length($buf)  if ($obs > length($buf));  # Max 1 Mbyte now

$buf .= substr($buf, 0, $obs);  # make sure we can write complete blocks
                                # in 1 syswrite call

#
# look at the clock and start emitting data as fast as we can
#

$starttime = time;

unless ($opt_q) {
    $SIG{'INT'} = 'atend';
    $SIG{'PIPE'} = 'atend';
    $SIG{__DIE__} = 'atend';
}

# loop 10MB at a time

$s = 1024 * 1024;
while (1) {
    shift(@times)  if (scalar(@times) >= $lagspan);
    push(@times, time);
    &progress  if ($opt_v);
   
    for (1..10) {
        $s -= 1024 * 1024;
        do {
            syswrite(STDOUT, $buf, $obs, $s) || die("$!\n");
        } while (($s += $obs) < 1024 * 1024);
        if (--$n == 0) {
            &atend  unless($opt_q);
            exit 0;
        }
    }
}

&atend  unless $opt_q;
exit;


#######################################################################


#
# show some progress on the screen
#
sub progress {
    $lag = $times[$#times] - $times[0];
    # ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
    ($sec,$min,$hour,,,,,,) =
                localtime(time);
    printf STDERR "%.2d:%.2d:%.2d %6dM  -  %6.3f MB/s\n",
                $hour,$min,$sec, ($opt_n - $n),
                ($lag == 0) ? 0 : 10 * ($#times) / $lag ;
                #($lag == 0) ? 0 : 10 * (scalar(@times)-1) / $lag ;
}


#
# show final statistics
#
sub atend {

    &progress  if ($opt_v);

    $elapsedtime = time - $starttime;

    printf STDERR "Total bytes output: %dMB%s\n",
        ($opt_n - $n),
        $opt_C ? " (high compressable)": ($opt_c?" (compressable)":"");
    printf STDERR "Elapsed time: %dh %dm %ds - average throughput %.3f MB/s\n",
        int($elapsedtime / 3600),
        int(($elapsedtime / 60) % 60),
        int($elapsedtime % 60),
        ($elapsedtime == 0) ? 0 : ($opt_n - $n) / $elapsedtime;
    exit;
}