Hi,
Thanks to all who responded and offered advice. I didn't want to get into
technical details of the server and software in this thread and I do
appreciate your time. I was just trying to get a "reality-check" that I
wasn't trying to do something silly. The message I've gotten is that
Bacula can work with the requirements I have, on the hardware that I have,
and that the performance issues I'm seeing are because there is something
currently wrong with my implementation.
Thanks,
Rich.
On Wed, 19 Feb 2014, rfox AT mbl DOT edu wrote:
> Hi,
>
> I'm using accurate mode on my system and the mysql queries crawl along
> causing the whole backup to run very slowly.
>
> I have 25 million files on a filesystem and I'm expected to be able to
> restore any file that existed on the filesystem on any day for a period of
> 1.5 years.
>
> Based on the performance of previous backup systems run on this server, I
> split the filesystem up into four groups and created bacula jobs for each
> group. Each group has its own dedicated logical library but all four jobs
> use the same MySQL database. I've been using Bacula in accurate mode for
> this system since Dec 20. While it was delightfully speedy at first over
> time it has become quite slow. The MySQL queries used during the run seem
> to be the bottleneck and I am worried the problem is insurmountable.
>
> I have been going through rounds of mysql tuning to try to increase
> performance but it's a powerful computer with 24 cores (2 Xeon E5645s @
> 2.4GHz), mysql is using over 32G RAM, the database is on a SSD and MySQL
> tmp is on a pair of striped SSDs. The query that hangs everything up
> during the process takes more than 24 hours to get MySQL to 'explain' and
> jobs take about 1.7 days to run at this point. (A job is running now and
> the same query has been executing for over 62000 seconds.) I am using
> MySQL 5.5 though, so I may be able to benefit from a more recent version.
>
> I've been running the MySQL sql-bench benchmarks on a number of different
> servers I have available and this server seems to be quite fast matching
> performance we get from our dedicated database servers with high speed
> enterprise disk arrays and 128G RAM. (Except when dropping tables, the
> dedicated systems are much faster than this server, but I don't believe
> that is related to the slow query problem I am seeing.)
>
> I would chalk it up to a simple overestimation on my part of the ability
> of accurate mode on a system this size but in my experience I have run
> across quite a few statements about big bacula implementations that have
> billions of rows in the File table. Mine currently only has 110 million or
> so. I can't help but wonder if these references to large tables are on
> systems that run in non-accurate mode and consequently don't perform the
> complex queries that I'm seeing.
>
> I expected a accurate mode to be difficult to implement without incurring
> a performance hit but I wonder if I'm not exceeding realistic expectations
> of the file selection algorithm with this size filesystem.
>
> Thanks,
> Rich.
>
>
>
--
Rich Fox
Systems Administrator
JBPC - Marine Biological Laboratory
http://www.mbl.edu/jbpc
508-289-7669 - mbl-at-richfox.org
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
