Ben Roberts

Hi Kern,

Thanks for updating the docs regarding auto prune. I disabled this last week and it has immediately fixed a long-running performance problem we were having when Bacula was looking for appendable volumes for a running job. I bring this up not only to report success but also because our issue was slightly different than initially described, so worth having on the record in this thread in case it affects other people in future.

We could experience this issue even when only 2-3 jobs were running at once. If one job ran out of volumes, the console would become slow to respond to any queries that touched the catalog (~30-60 seconds). If two jobs ran out at the same time the console might block for several minutes at a time. When three or more jobs all ran out the console would become practically unusable, blocking for so long that you'd have to go away and do something else in the meantime. I think Bacula rescans for appendable volumes every ~5minutes, so this process would compound if there was more than 5mins work to do each cycle. It made resolving an out-of-volumes issue an absolute pain and as a result it was frequently ignored causing our already long-running jobs to hit the 6 day limit and fail. This had gotten increasingly worse over the last few months which is likely because the number of volumes we consume grows over time (~+50/month). While the end-of-job pruning scans would also have caused this

The reason for this is that while bacula is scanning for appendable volumes it appears to issue queries against the catalog regarding individual volumes in sequence (I guess this is to find the list of jobs on each volume one at a time, then check the job retention periods to see if any can be deleted?) We have several thousand volumes, so executing these queries takes a while and I get the feeling bacula will only do these in sequence, due to either a lock or other single-threaded behaviour. So when there were several of these operations queued up because multiple jobs were seeking volumes at the same time, it's understandable why the window to slip a bconsole query in and have a timely response was very narrow and why most bconsole commands would take a very long time to return.

Full disclosure, I did also upgrade to 7 at the same time as making the pruning changes and the separate catalog connection may also have helped with this, however both together certainly give the desirable performance we used to experience.

Ben Roberts


> -----Original Message-----
> From: Kern Sibbald [mailto:kern AT sibbald DOT com]
> Sent: 25 November 2014 13:42
> To: bacula-users; bacula-devel
> Subject: [Bacula-users] Mini-Status report
>
> Hello,
>
> This is a mini-status report just to let you know that someone on one of
> these lists said that the new white papers advise using:
>
> Maximum Concurrent Jobs = 1
>
> In the bacula-sd.conf Device resource. I have reviewed the three white
> papers listed on the bacula.org site and I find no such mention. Almost
> all the examples use MaximumConcurrentJobs=5 and several set it to 10.
>
> If someone can point me to the place (if it exists) where 1 in mentioned,
> I will be happy to fix it.
>
> In addition, another person pointed out that manual_prune.pl does not seem
> to be available. That was an oversight on my part, and that file is now
> included as a download item under White Papers. The essential elements of
> what manual_prune.pl does has been integrated directly into the Bacula
> source code, and in the next few days I will update the documentation to
> refer to the new way of doing "manual" pruning (i.e.
> pruning once a day rather than at the end of every Job).
>
> For those of you who are not aware of the pruning issues, you need to know
> that when one has many jobs (say 1000 per day) and many Volumes, the
> normal Job pruning that Bacula does at the end of each Job can become a
> performance problem (not likely with less than 50 jobs per day). To
> resolve it, it is best to turn off automatic pruning but then to schedule
> an Admin job once a day that will do the equivalent. The program (now
> integrated into the code) manual_prune.pl was written by Bacula Systems
> for the Enterprise customers, but is published for your use with the
> community version.
>
> Finally, I have made a few very minor changes to the Best Practices for
> Disk Based Backup document.
>
> Best regards,
> Kern
>
> --------------------------------------------------------------------------
> ----
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from
> Actuate! Instantly Supercharge Your Business Reports and Dashboards with
> Interactivity, Sharing, Native Excel Exports, App Integration & more Get
> technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clkt
> rk
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users

---

This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient. If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.