On 06/24/2011 02:56 PM, Gavin McCullagh wrote:
> On Fri, 24 Jun 2011, ted wrote:
>
>> I have an issue when trying a restore of a large data set consisting of
>> about 6.5TB and 35,000,000 files the console takes an extremely long time
>> to build the directory tree, over an hr. After the tree is built I typed
>> "mark *" and this command ran for about 18 hours before I hit ctrl-c; the
>> mySQL server showed no activity and neither bconsole nor bacula-dir was
>> not using any CPU, so I believe the process just petered out. I was
>> wondering if anyone has been successful in restoring large TB data sets.
>> Either ones containing a large number of individual files or ones that
>> are 6TB or larger with small file lists and contain small number of large
>> files? 
> You are not alone.  I have one client in particular for whom building the
> file tree takes about 25-30 minutes.  Like you, I'm using MySQL.  There
> have been suggestions from a number of quarters which suggest that
> Postgresql doesn't have this issue.
>
> This may not necessarily be a criticism of MySQL, so much as that the
> developers tend to spend more time working with Postgresql and therefore
> Bacula is better optimised toward Postgresql.
>
> There is a discussion here:
>
> http://old.nabble.com/Re%3A-Tuning-for-large-%28millions-of-files%29-backups--tt30099042.html#a30259098
>
> We plan migrating to Postgresql as that's the only solution I know of right
> now.
>
> Gavin
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a 
> definitive record of customers, application performance, security 
> threats, fraudulent activity and more. Splunk takes this data and makes 
> sense of it. Business sense. IT sense. Common sense.. 
> http://p.sf.net/sfu/splunk-d2d-c1
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
We are running Bacula 5.0.3 (from Ports) on FreeBSD 8.2, with
PostgreSQL. Building the directory tree for a backup that was around
1.1TB w/25million files took about 12 minutes.

"mark"-ing all the files in that job took around 18 minutes. That is
handled entirely by the bacula-dir process.

I hope my data point has helped, I don't know why it stopped responding,
but have a similar environment and its been working out for us.

Mike C

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a 
definitive record of customers, application performance, security 
threats, fraudulent activity and more. Splunk takes this data and makes 
sense of it. Business sense. IT sense. Common sense.. 
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users