On Tue, Mar 23, 2010 at 04:07:17PM +0000, Alan Brown wrote:
> Matija Nalis wrote:
>> It is probably not hung, but just very, very slow.
>
> Yes. You probably need a LOT more ram and to tune mysql's parameters.
Or maybe someone should tune the SQL queries (no, I'm not
volunteering, it's not my forte) and/or the way bacula stores the
catalog ?
The main issue is that for us 3.0.3 was about 20-100 *times* faster
for approximately same dataset than 5.0.1. I do understand that 5.0.0
added BaseFiles support, but IMHO such is a speed drop is not an
acceptable tradeoff (especially if one cannot turn it off in order to
get faster queries again. I'd gladly compile with "--disable-basefiles"
if that gave me 2 orders of magnitude speedup).
Queries that never took more than 3-5 *minutes* with 3.0.3 have
started taking more that several *hours* with 5.0.0.
(Before you ask, we've had to upgrade because there were [and still
are in 5.0.1, although somewhat rarer] bugs with director stopping
working -- otherwise we would've downgraded back to 3.0.x)
We'd try tweaking key_buffer (and converting to InnoDB and tweakings
innodb_buffer_pool_size), join_buffer_size, max_heap_table_size,
tmp_table_size, sort_buffer_size, read_buffer_size, read_rnd_buffer_size
but in the end we've had to reduce retention to just a few weeks in order
to make the restoration happen in reasonable times (ie. getting the file
selection in less than 10 minutes).
> Even with 48Gb ram, a few restores on our system (~255 million File
> records: up to 4 million files on some full backups but nost are under
> 100k entries) could take an hour to get past the "building directory
> tree" stage.
That is really terrible, I really think the developers should look
into it.
We've used Legato networker before (we still do, as we're not yet
successfully completed the migration; and it's looking more and more
grim prospect by the day), and on approximately the same dataset (of
about 500 million records spread over 100 servers) and somewhat
weaker hardware, it would allow user to start selecting files to
restore in matter of *seconds* (and it was using it's simple db6
files, no server/database tuning required at all)
Now with bacula 5.0.1, we have to wait several *hours* before we can
start selecting files to restore, and it is considered "normal" ?!
Several minutes might be tolerated by our users (although even that
is almost hundred times slower than they were used to !), but several
hours most certainly isn't (and a retention drop from several months
to several weeks as alternative is also isn't making them extremely
happy)
> It's a _lot_ faster with postgresql and moderate tuning (My other gripes
> about the changeover notwithstanding, those are annoyances, not
> showstoppers)
Waiting several hours to choose file for restoring might not be an
issue for you; but we have users which were used to waiting just
several seconds to select files to restore (and a few more minutes
for restore to happen), and they are not impressed at all with
bacula.
As it is, it is *much faster* for us if we need to restore one file
to do a complete restore of whole server and then delete 99.999% of
the files, than to use the file catalog to select few files to
restore. That is ridiculous situation.
> The lesson for us was that mysql doesn't scale to huge datasets well and
> we should have switched to postgres much earlier.
That might be, and we'll try converting to PostgreSQL (there are
issues with moving bacula data from MySQL to PostgreSQL), it seems.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
