Steve Polyack wrote:
> On 08/11/10 11:34, Hugo Silva wrote:
>> Christian Gaul wrote:
>>> Am 11.08.2010 16:49, schrieb Hugo Silva:
>>>> Thomas Mueller wrote:
>>>>
>>>>> Am Tue, 10 Aug 2010 15:13:07 +0100 schrieb Hugo Silva:
>>>>>
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm backing up a server in Germany from a director in The
>>>>>> Netherlands.
>>>>>> Using bacula, I can't seem to get past ~3000KB/s.
>>>>>>
>>>>>> Here's an iperf result:
>>>>>> [ 3] local [fd-addr] port 16625 connected with [dir-addr] port
>>>>>> 5001 [
>>>>>> ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 110
>>>>>> MBytes 91.2 Mbits/sec
>>>>>>
>>>>> you speak of a server in germany and director in netherlands. the
>>>>> sd is
>>>>> also on the director machine. fd sends data to sd directly - could
>>>>> also be
>>>>> a routing issue.
>>>>>
>>>>> and: as in many other threads mentioned, backing up a filesystem with
>>>>> thounds or millions of files can't be compared to a sequential read
>>>>> with
>>>>> dd.
>>>>>
>>>>> and: did you ran the btape tests on the sd to check the performance?
>>>>>
>>>>>
>>>>> - Thomas
>>>>>
>>>> Hi,
>>>>
>>>> Thank you for your input.
>>>>
>>>> The SD is also in the director machine, indeed. I don't think it's a
>>>> routing issue - the iperf test was done between these two machines with
>>>> excellent results.
>>>>
>>>> I'm using disk storage; btape doesn't seem to be of help:
>>>> btape: btape.c:302 btape only works with tape storage.
>>>>
>>>> I am aware that a dd test vs many small files isn't comparable - but at
>>>> least it rules out the SD storage. (and see below)
>>>>
>>>> My interest is in knowing if there are known ways people use to
>>>> speed up
>>>> the backup process when done over the internet. This is my first bacula
>>>> configuration backing up FDs in remote countries.
>>>>
>>>>
>>>> Consider the following:
>>>> # zfs create storage/test
>>>> # zfs set mountpoint=/test storage/test
>>>> # zfs set compression=off storage/test
>>>>
>>>>
>>>> # dd if=/dev/urandom of=/test/testfile bs=128k count=4096
>>>> 4096+0 records in
>>>> 4096+0 records out
>>>> 536870912 bytes transferred in 7.020243 secs (76474691 bytes/sec)
>>>>
>>>>
>>>>
>>>> Now at the director, I create a FileSet backing up this one file.
>>>> To aid bacula even more, I'll first put it in the OS cache:
>>>>
>>>> # dd if=/test/testfile of=/dev/null bs=128k
>>>> 4222+0 records in
>>>> 4222+0 records out
>>>> 553385984 bytes transferred in 2.910288 secs (190148180 bytes/sec)
>>>>
>>>> And finally, the backup job, using this FileSet:
>>>> FileSet {
>>>> Name = "TestFileSet"
>>>> Include {
>>>> Options {
>>>> #Compression=gzip
>>>> Signature=SHA1
>>>> Onefs=yes
>>>> Honor nodump flag=yes
>>>> Noatime=yes
>>>> }
>>>>
>>>> File = /test/testfile
>>>> }
>>>> }
>>>>
>>>>
>>>> Notice the read bytes/sec on the second dd.
>>>>
>>>> At this point, consider that:
>>>>
>>>> * An iperf test used the link at ~93%.
>>>> * The SD hdd is capable of writing at least 70MB/s.
>>>> * The FD hdd (ok, zfs cache) is capable of reading at least 180MB/s.
>>>>
>>>> It follows, I believe, that this test should show transfer rates close
>>>> to 100mbits. This is one big file, and the hdd is perfectly capable of
>>>> sustaining 12.5MB/s sequential read (far more, as demonstrated)
>>>>
>>>> However..
>>>>
>>>> Traffic Peak Total
>>>> em0 in 4.863MB/s 4.863MB/s 16.461GB
>>>> out 137.977KB/s 137.977 KB/s 495.591MB
>>>>
>>>> To the three points made above, consider that:
>>>>
>>>> * Bacula is using the network link at ~38.4% during this test.
>>>>
>>>>
>>>>
>>>> I had to disable the Maximum Network Buffer Size in the mean time,
>>>> coincidence or not the director started throwing out "unknown errors"
>>>> while connecting to storage, so this test is run with default buffer
>>>> sizes (which shouldn't be a problem - I got 91-93% of the max link
>>>> speed with iperf using default buffer sizes)
>>>>
>>>> This test:
>>>> * Uses TLS encryption [encrypted comms]
>>>> * Uses PKI encryption [encrypted backup data]
>>>> * Does not use compression
>>>>
>>>> I don't think TLS/PKI is the cause - there's plenty of CPU% while it's
>>>> running. Could investigate this further.
>>>>
>>> On how many cores? AFAIK the FD only uses one thread for TLS / PKI /
>>> compression.
>>> (At least it never goes over 100% CPU for me, even when running
>>> concurrent jobs)
>>>
>>>
>>>> Not sure what to try next. Any suggestions?
>>>>
>>>> Thanks for reading.
>>>>
>>>> Hugo
>>>>
>>>>
>>>
>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>> 11 root 171 ki31 0K 128K CPU7 7 5127.9 100.00%
>> {idle: cpu7}
>> 11 root 171 ki31 0K 128K CPU5 5 5126.0 100.00%
>> {idle: cpu5}
>> 11 root 171 ki31 0K 128K CPU3 3 5122.7 100.00%
>> {idle: cpu3}
>> 11 root 171 ki31 0K 128K CPU4 4 5114.2 100.00%
>> {idle: cpu4}
>> 11 root 171 ki31 0K 128K CPU1 1 5113.7 100.00%
>> {idle: cpu1}
>> 11 root 171 ki31 0K 128K RUN 0 5101.2 100.00%
>> {idle: cpu0}
>> 11 root 171 ki31 0K 128K CPU2 2 5101.5 98.39% {idle:
>> cpu2}
>> 11 root 171 ki31 0K 128K CPU6 6 5123.7 90.58% {idle:
>> cpu6}
>> 61122 root 58 0 27848K 5900K select 7 0:05 9.28% {bacula-fd}
>>
>>
>> FreeBSD will move bacula-fd to another CPU now and then, but as you see
>> it's using only about 10% CPU during this test. Core #7, where it was at
>> the time of this snapshot, was 100% idle (this is actually a top
>> discrepancy - the process was on CPU#6 before, and you can see that one
>> is 90.58% idle, which sounds about right)
>>
> Regardless of the usage you should try disabling PKI encryption on the
> FD you are backing up from. I've seen it really slow things down on
> even very new CPUs.
>
Steve,
Thank you for your suggestion (and sorry for not replying to the
off-list mail sooner - forgot about it); with PKI off:
em0 in 5.189 MB/s 5.206 MB/s 16.899 GB
out 148.275 KB/s 151.429 KB/s 508.401 MB
Or - 41.44% vs 38.4%.
Can anyone try running this test and see if the numbers agree?
Hugo
------------------------------------------------------------------------------
This SF.net email is sponsored by
Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
