Re: [Bacula-users] Tuning for large (millions of files) backups?

'Ondrej PLANKA (Ignum profile)' wrote:
>Thanks :)
>Which type of MySQL storage engine are you using? MyISAM or InnoDB for
>large Bacula system?
>Can you please copy/paste your MySQL configuration? I mean my.cnf file

Please re-read this thread and you should find what you are looking for.

>Thanks, Ondrej.
>
>
>Henrik Johansen napsal(a):
>> 'Ondrej PLANKA (Ignum profile)' wrote:
>>
>>> Hello Henrik,
>>>
>>> what are you using? MySQL?
>>>
>>
>> Yes - all our catalog servers run MySQL.
>>
>> I forgot to mention this in my last post - we are Bacula System
>> customers and they have proved to very supportive and competent.
>>
>> If you are thinking about doing large scale backups with Bacula I can
>> only encourage you to get a support subscription - it is worth every
>> penny.
>>
>>
>>
>>> Thanks, Ondrej.
>>>
>>> 'Mingus Dew' wrote:
>>>
>>>> Henrik,
>>>> Have you had any problems with slow queries during backup or restore
>>>> jobs? I'm thinking about http://bugs.bacula.org/view.php?id=1472
>>>> specifically, and considering that the bacula.File table already has 73
>>>> million rows in it and I haven't even successfully ran the "big" job
>>>> yet.
>>>>
>>> Not really.
>>>
>>> We have several 10+ million file jobs - all run without problem (backup
>>> and restore).
>>>
>>> I am aware of the fact that a lot of Bacula users run PG  ( Bacula
>>> Systems also does recommend PG for larger setups ) but nevertheless
>>> MySQL has served us very well so far.
>>>
>>>
>>>> Just curious as a fellow Solaris deployer...
>>>>
>>>> Thanks,
>>>> Shon
>>>>
>>>> On Fri, Oct 8, 2010 at 3:30 PM, Henrik Johansen
>>>> <henrik AT scannet DOT dk<mailto:henrik AT scannet DOT dk <mailto:henrik 
>>>> AT scannet DOT dk%3Cmailto:henrik AT scannet DOT dk>>> wrote:
>>>> 'Mingus Dew' wrote:
>>>> All,
>>>> I am running Bacula 5.0.1 on Solaris 10 x86. I'm currently running
>>>> MySQL 4.1.22 for the database server. I do plan on upgrading to a
>>>> compatible version of MySQL 5, but migrating to PostgreSQL isn't an
>>>> option at this time.
>>>>
>>>> I am trying to backup to tape a very large number of files for a
>>>> client. While the data size is manageable at around 2TB, the number of
>>>> files is incredibly large.
>>>> The first of the jobs had 27 million files and initially failed because
>>>> the batch table became "Full". I changed the myisam_data_pointer size
>>>> to a value of 6 in the config.
>>>>
>>>> This job was then able to run successfully and did not take too long.
>>>>
>>>> I have another job which has 42 million files. I'm not sure what that
>>>> equates to in rows that need to be inserted, but I can say that I've
>>>> not been able to successfully run the job, as it seems to hang for
>>>> over 30 hours in a "Dir inserting attributes" status. This causes
>>>> other jobs to backup in the queue and once canceled I have to restart
>>>> Bacula.
>>>>
>>>> I'm looking for way to boost performance of MySQL or Bacula (or both)
>>>> to get this job completed.
>>>>
>>>> You *really* need to upgrade to MySQL 5 and change to InnoDB - there is no
>>>> way in hell that MySQL 4 + MyISAM is going to perform decent in your
>>>> situation.
>>>> Solaris 10 is a Tier 1 platform for MySQL so the latest versions are
>>>> always available from http://www.mysql.com in the native pkg format so 
>>>> there really
>>>> is no excuse.
>>>>
>>>> We run our Bacula Catalog MySQl servers on Solaris (OpenSolaris) so
>>>> perhaps I can give you some pointers.
>>>>
>>>> Our smallest Bacula DB is currently ~70 GB (381,230,610 rows).
>>>>
>>>> Since you are using Solaris 10 I assume that you are going to run MySQL
>>>> off ZFS - in that case you need to adjust the ZFS recordsize for the
>>>> filesystem that is going to hold your InnoDB datafiles to match the
>>>> InnoDB block size.
>>>>
>>>> If you are using ZFS you should also consider getting yourself a fast
>>>> SSD as a SLOG (or to disable the ZIL entirely if you dare) - all InnoDB
>>>> writes to datafiles are O_SYNC and benefit *greatly* from an SSD in
>>>> terms of write / transaction speed.
>>>>
>>>> If you have enough CPU power to spare you should try turning on
>>>> compression for the ZFS filesystem holding the datafiles - it also can
>>>> accelerate DB writes / reads but YMMV.
>>>>
>>>> Lastly, our InnoDB related configuration from my.cnf :
>>>>
>>>> # InnoDB options skip-innodb_doublewrite
>>>> innodb_data_home_dir = /tank/db/
>>>> innodb_log_group_home_dir = /tank/logs/
>>>> innodb_support_xa = false
>>>> innodb_file_per_table = true
>>>> innodb_buffer_pool_size = 20G
>>>> innodb_flush_log_at_trx_commit = 2
>>>> innodb_log_buffer_size = 128M
>>>> innodb_log_file_size = 512M
>>>> innodb_log_files_in_group = 2
>>>> innodb_max_dirty_pages_pct = 90
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Shon
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Beautiful is writing same markup. Internet Explorer 9 supports
>>>> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
>>>> Spend less time writing and  rewriting code and more time creating great
>>>> experiences on the web. Be a part of the beta today.
>>>> http://p.sf.net/sfu/beautyoftheweb
>>>>
>>>> _______________________________________________
>>>> Bacula-users mailing list
>>>> Bacula-users AT lists.sourceforge DOT net<mailto:Bacula-users AT 
>>>> lists.sourceforge DOT net <mailto:Bacula-users AT lists.sourceforge DOT 
>>>> net%3Cmailto:Bacula-users AT lists.sourceforge DOT net>>
>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>>>
>>>> --
>>>> Med venlig hilsen / Best Regards
>>>>
>>>> Henrik Johansen
>>>> henrik AT scannet DOT dk<mailto:henrik AT scannet DOT dk <mailto:henrik AT 
>>>> scannet DOT dk%3Cmailto:henrik AT scannet DOT dk>>
>>>> Tlf. 75 53 35 00
>>>>
>>>> ScanNet Group
>>>> A/S ScanNet
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Beautiful is writing same markup. Internet Explorer 9 supports
>>>> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
>>>> Spend less time writing and  rewriting code and more time creating great
>>>> experiences on the web. Be a part of the beta today.
>>>> http://p.sf.net/sfu/beautyoftheweb
>>>>
>>>> _______________________________________________
>>>> Bacula-users mailing list
>>>> Bacula-users AT lists.sourceforge DOT net <mailto:Bacula-users AT 
>>>> lists.sourceforge DOT net>
>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>> --
>>> Med venlig hilsen / Best Regards
>>>
>>> Henrik Johansen
>>> henrik AT scannet DOT dk <mailto:henrik AT scannet DOT dk>
>>> Tlf. 75 53 35 00
>>>
>>> ScanNet Group
>>> A/S ScanNet
>>>
>>>
>>>
>>>
>>>
>>
>>

-- 
Med venlig hilsen / Best Regards

Henrik Johansen
henrik AT scannet DOT dk
Tlf. 75 53 35 00

ScanNet Group
A/S ScanNet 

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store 
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users