First off thank you everyone in the group for your useful and insightful
suggestions. I was just about to throw in the towel when the solution
presented itself.
For all you 3ware users out there this might be of help to you.
Steve you gave me the nudge I needed the problem was with the high
Iowait time. I went looking through many other forums and about IO wait
time when I came across the following line
"[sda] Write cache: enabled, read cache: enabled, doesn't support DPO or
FUA"
The person who posted it was also complaining of slow performance but
his he thought was related to the DPO/FUA error. I remember seeing this
same line and wanted to check his error against mine. This is where I
found the smoking gun.
"[sdc] Write cache: enabled, read cache: disabled, doesn't support DPO
or FUA"
After finding this line it was a quick search of the 3ware forum to find
out how to enable the read caching.
1. Upgrade 3DM2 web tool and tw_cli
This was necessary to see if my current firmware supported the read
cache setting. (of course it did not)
2. Upgrade the firmware on the raid card
I would call 3ware just in case as you have to make sure that you
current firmware can be upgraded in one single upgrade or if you need to
patch to previous releases first. Also you need to make sure that your
current driver will support the new firmware and finally backup backup
backup. They 3ware team could not stress this one enough.
3. Enable the read cache via either 3dm2 or tw_cli.
After upgrading the firmware and enabling read caching I followed some
of these links to further increase performance.
http://www.3ware.com/kb/article.aspx?id=11050
http://www.3ware.com/KB/article.aspx?id=15244
http://www.3ware.com/KB/article.aspx?id=10889
I am now getting between 30 and 50 megs/s. The slower speed is due the
sheer number of files being backed up (2.1M files in 1.7TB)
Thanks again.
-H
-----Original Message-----
From: Steve Ellis [mailto:ellis AT brouhaha DOT com]
Sent: Thursday, December 10, 2009 10:21 AM
To: bacula-users AT lists.sourceforge DOT net
Subject: Re: [Bacula-users] Network transfer Speed
On 12/10/2009 9:33 AM, Hayden Katzenellenbogen wrote:
> Steve,
>
> Here is a quick snap of my top during a full backup.
>
> top - 09:32:35 up 1 day, 18:38, 1 user, load average: 11.41, 11.60,
> 10.75
> Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 0.0%id, 99.4%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu1 : 13.0%us, 1.4%sy, 0.0%ni, 18.1%id, 66.9%wa, 0.0%hi, 0.6%si,
> 0.0%st
> Cpu2 : 1.9%us, 0.3%sy, 0.0%ni, 0.0%id, 97.8%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu3 : 0.6%us, 1.3%sy, 0.0%ni, 22.7%id, 74.8%wa, 0.0%hi, 0.6%si,
> 0.0%st
> Cpu4 : 0.6%us, 0.3%sy, 0.0%ni, 0.0%id, 99.1%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu5 : 0.6%us, 0.0%sy, 0.0%ni, 20.8%id, 78.3%wa, 0.0%hi, 0.3%si,
> 0.0%st
> Cpu6 : 0.4%us, 0.7%sy, 0.0%ni, 52.5%id, 46.5%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu7 : 0.6%us, 0.3%sy, 0.0%ni, 7.6%id, 91.5%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 4057152k total, 4035996k used, 21156k free, 14372k
buffers
> Swap: 8000360k total, 1536k used, 7998824k free, 3852936k
cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 6118 root 20 0 60388 2212 1296 S 24 0.1 102:35.41
bacula-fd
>
> 6240 root 20 0 130m 2092 1348 S 3 0.1 227:19.75
bacula-sd
>
> 23214 hayden 20 0 18992 1308 932 R 1 0.0 0:00.24 top
>
>
> There are many occasions where the bacula-sd and bacula-fd have
between
> 10-50% and all the wait percentages are through the roof as you see
> above. Also as you can see the load average is also through the roof.
>
> -H
Hayden-
I'm not sure what else you might be running on this box, but I've
noticed on mine that my memory usage recently went up significantly, and
I started seeing high load average peaks when only bacula (or the db
behind it) should have been generating high load. It didn't seem that
the bacula daemons themselves were using significantly more memory than
previously, and I hadn't changed my bacula version anyway. Further, I
had not changed any configuration from what had been running smoothly
before. In my case, after detuning my MySQL config to use less memory,
my load averages improved, and so did bacula performance.
I have not dug into this very far yet, but my guess is that either the
kernel behavior has changed recently with some update I applied (perhaps
significantly increased kernel buffering??), or some library used by
lots of programs now uses more memory than previously. Certainly, my
box, with only 2 processors shouldn't need as much physical RAM as
yours, but both your 8-core box and my 2-core box have 4Gigs of RAM.
I have no silver bullets here, but if see load average spike
dramatically during a backup, and see very high wait percentages on your
CPUs during backups, then you might look to see what programs are using
the most VM, and seeing if they can either be temporarily turned off or
detuned to use less memory. An even better answer might be to try
adding RAM to your machine (assuming its running a 64-bit OS), but the
first approach could be done without taking the box down and without any
expense at least just to test the theory.
Hope that helps,
-se
------------------------------------------------------------------------
------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
