On 12/10/2009 9:33 AM, Hayden Katzenellenbogen wrote:
> Steve,
>
> Here is a quick snap of my top during a full backup.
>
> top - 09:32:35 up 1 day, 18:38, 1 user, load average: 11.41, 11.60,
> 10.75
> Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 0.0%id, 99.4%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu1 : 13.0%us, 1.4%sy, 0.0%ni, 18.1%id, 66.9%wa, 0.0%hi, 0.6%si,
> 0.0%st
> Cpu2 : 1.9%us, 0.3%sy, 0.0%ni, 0.0%id, 97.8%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu3 : 0.6%us, 1.3%sy, 0.0%ni, 22.7%id, 74.8%wa, 0.0%hi, 0.6%si,
> 0.0%st
> Cpu4 : 0.6%us, 0.3%sy, 0.0%ni, 0.0%id, 99.1%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu5 : 0.6%us, 0.0%sy, 0.0%ni, 20.8%id, 78.3%wa, 0.0%hi, 0.3%si,
> 0.0%st
> Cpu6 : 0.4%us, 0.7%sy, 0.0%ni, 52.5%id, 46.5%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu7 : 0.6%us, 0.3%sy, 0.0%ni, 7.6%id, 91.5%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 4057152k total, 4035996k used, 21156k free, 14372k buffers
> Swap: 8000360k total, 1536k used, 7998824k free, 3852936k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 6118 root 20 0 60388 2212 1296 S 24 0.1 102:35.41 bacula-fd
>
> 6240 root 20 0 130m 2092 1348 S 3 0.1 227:19.75 bacula-sd
>
> 23214 hayden 20 0 18992 1308 932 R 1 0.0 0:00.24 top
>
>
> There are many occasions where the bacula-sd and bacula-fd have between
> 10-50% and all the wait percentages are through the roof as you see
> above. Also as you can see the load average is also through the roof.
>
> -H
Hayden-
I'm not sure what else you might be running on this box, but I've
noticed on mine that my memory usage recently went up significantly, and
I started seeing high load average peaks when only bacula (or the db
behind it) should have been generating high load. It didn't seem that
the bacula daemons themselves were using significantly more memory than
previously, and I hadn't changed my bacula version anyway. Further, I
had not changed any configuration from what had been running smoothly
before. In my case, after detuning my MySQL config to use less memory,
my load averages improved, and so did bacula performance.
I have not dug into this very far yet, but my guess is that either the
kernel behavior has changed recently with some update I applied (perhaps
significantly increased kernel buffering??), or some library used by
lots of programs now uses more memory than previously. Certainly, my
box, with only 2 processors shouldn't need as much physical RAM as
yours, but both your 8-core box and my 2-core box have 4Gigs of RAM.
I have no silver bullets here, but if see load average spike
dramatically during a backup, and see very high wait percentages on your
CPUs during backups, then you might look to see what programs are using
the most VM, and seeing if they can either be temporarily turned off or
detuned to use less memory. An even better answer might be to try
adding RAM to your machine (assuming its running a 64-bit OS), but the
first approach could be done without taking the box down and without any
expense at least just to test the theory.
Hope that helps,
-se
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
