Bacula-users

Re: [Bacula-users] Tuning for large (millions of files) backups?

2010-11-08 17:23:34
Subject: Re: [Bacula-users] Tuning for large (millions of files) backups?
From: Gavin McCullagh <gavin.mccullagh AT gcd DOT ie>
To: Alan Brown <ajb2 AT mssl.ucl.ac DOT uk>
Date: Mon, 8 Nov 2010 22:21:39 +0000
Hi Alan,

On Mon, 08 Nov 2010, Alan Brown wrote:

> >When we do restores, building the tree takes a considerable time now.  I
> >haven't had a lot of time to look at it, but suspected it might be down to
> >this issue.
> 
> That's a classic symptom of not having the right indexes on the File table.
> 
> This is in the FAQ somewhere and it's the issue you mentioned as
> affecting mysql but not postgresql (Their index handling is slightly
> different).

Right you are....

http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog

There is still an element of "move to postgresql" though

  "Moving from MySQL to PostgreSQL should make it work much better due to
   different (more optimized) queries and different SQL engine. "
   
http://bugs.bacula.org/view.php?id=1472http://bugs.bacula.org/view.php?id=1472

We seem to have the correct indexes on the file table.  I've run optimize table
and it still takes 14 minutes to build the tree on one of our bigger clients.
We have 51 million entries in the file table.

> The other big contributor to long tree builds is insufficient ram.
> Keep an eye on swap and the bacula-dir process whilst building the
> tree. If it starts thrashing, you need more memory.

I think we should be okay on that score but it's something to watch out
for.

Thanks,
Gavin


------------------------------------------------------------------------------
The Next 800 Companies to Lead America's Growth: New Video Whitepaper
David G. Thomson, author of the best-selling book "Blueprint to a 
Billion" shares his insights and actions to help propel your 
business during the next growth cycle. Listen Now!
http://p.sf.net/sfu/SAP-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users