Bacula-users

Re: [Bacula-users] Finding performance issues

2009-02-16 16:01:19
Subject: Re: [Bacula-users] Finding performance issues
From: "(private) HKS" <hks.private AT gmail DOT com>
To: bacula-users <Bacula-users AT lists.sourceforge DOT net>
Date: Mon, 16 Feb 2009 15:58:01 -0500
On Fri, Feb 13, 2009 at 4:00 PM, (private) HKS <hks.private AT gmail DOT com> 
wrote:
> On Thu, Feb 12, 2009 at 6:03 PM, (private) HKS <hks.private AT gmail DOT com> 
> wrote:
>> On Tue, Feb 10, 2009 at 3:56 PM, (private) HKS <hks.private AT gmail DOT 
>> com> wrote:
>>> On Tue, Feb 10, 2009 at 1:04 PM, Steve Polyack <korvus AT comcast DOT net> 
>>> wrote:
>>>> (private) HKS wrote:
>>>>>
>>>>> My server's network performance seems all right. Testing basic TCP
>>>>> throughput with iperf, I'm showing an average of 880Mbps or so. FTP
>>>>> downloads to this server hum along at about 85MB/s.
>>>>>
>>>>>
>>>>
>>>> You may want to expirement with the "Maximum Network Buffer Size" parameter
>>>> which is available in both the file-daemon and storage-daemon configuration
>>>> files. As far as the documentation explains, the SD's default is 32768 
>>>> bytes
>>>> while the FD default is 65536 bytes.  I'm not sure what the reason is for
>>>> the difference, but I would try setting them both to either 32768 or 65536.
>>>>  Perhaps try larger values, but stay under the limits of your OS's TCP
>>>> send/recv buffers (256k s/r on FreeBSD 7).
>>>>
>>>
>>> Thanks for the recommendation. I've changed this around a bit, but
>>> haven't seen any change. I also haven't been able to replicate this
>>> poor performance through any other method of writing to disk,
>>> transferring across the network, or some combination of the two. I'm
>>> certainly open to doing tests on this if only I knew what to do.
>>>
>>> -HKS
>>>
>>
>>
>> I have discovered (what appears to be) a bug in OpenBSD's bnx(4)
>> driver that limits tx performance.
>>
>> I don't *think* it's related to this problem: rx traffic performs
>> beautifully and the threshold (109Mb/s) is far above what I'm getting
>> with Bacula. I mention it in the interest of full disclosure.
>>
>> -HKS
>>
>
>
> I rebuilt one of my servers on FreeBSD 7.1 which is unaffected by this
> bug and the performance issue persists. This pretty much eliminates
> the OS as a problem here, so I guess we're looking at a hardware
> oddity, some tuning knobs that need adjustment, or some combination of
> the two.
>
> -HKS
>

I know this is turning into a long-running monologue, but this
performance issues is the last thing standing between me and a Backup
Exec-free environment, so it's important to me.

I believe I've eliminated the disks as the performance bottleneck.
Through various tuning knobs (sector size, softdeps, stripe size),
I've doubled my disk throughput on a simple dd test, but bacula
performance has remained unchanged. This was also true of multiple
RAID configurations.

It seems more and more likely to me that this is a Bacula-specific
issue. What else can I dig into to try to resolve this?

-HKS

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users