Re: [Bacula-users] Mini-Status report
2014-12-06 05:54:18
Hello Ben,
Thanks for your feedback on pruning. While I was reading your
email, I kept wondering what version of Bacula you were running,
because slowing down of the console with version 5.2.x and before
was a typical symptom, while in 7.0.x this should largely be fixed
even without delaying the pruning. At the end of your email, you
pointed out that you also switched to 7.0.x. Previously Bacula
used a single thread process at a time that was accessing the
database, with version 7.0.x (if I remember right), we separated
the console into a second DB connection so it runs in its own
thread and generally does not need to wait for pruning or
attribute spooling that can go on at the same time.
If you are currently using MySQL as the backend database and you
still find that either pruning or inserting file attributes takes
a long time with Bacula version 7.0.x, I would recommend you to
move to PostgreSQL. Out of the box, PostgreSQL is a bit harder to
get to work with Bacula for two reasons: 1. Getting the login
permissions setup correctly for Bacula to access and modify the
database is more complex than MySQL which works out of the box. 2.
The default installation of PostgreSQL is very inefficient for
medium to large Bacula installations. It requires a little bit of
tuning of about 10 of the parameters, then PostgreSQL will perform
significantly better than MySQL. I have heard some good comments
about MariaDB, but we have not yet tried it and thus don't
officially support it.
Final note: the current versions of Bacula are designed to handle
about 1,000 volumes, if you start getting 20,000 or 100,000
volumes, you will very likely run into severe performance
problems. This is currently on my radar screen, and I now think I
know how to fix it (not trivial) but haven't yet found the time.
Best regards,
Kern
On 12/06/2014 11:33 AM, Roberts, Ben wrote:
Hi Kern,
Thanks for updating the docs regarding auto prune. I disabled this
last week and it has immediately fixed a long-running performance
problem we were having when Bacula was looking for appendable
volumes for a running job. I bring this up not only to report
success but also because our issue was slightly different than
initially described, so worth having on the record in this thread
in case it affects other people in future.
We could experience this issue even when only 2-3 jobs were
running at once. If one job ran out of volumes, the console would
become slow to respond to any queries that touched the catalog
(~30-60 seconds). If two jobs ran out at the same time the console
might block for several minutes at a time. When three or more jobs
all ran out the console would become practically unusable,
blocking for so long that you'd have to go away and do something
else in the meantime. I think Bacula rescans for appendable
volumes every ~5minutes, so this process would compound if there
was more than 5mins work to do each cycle. It made resolving an
out-of-volumes issue an absolute pain and as a result it was
frequently ignored causing our already long-running jobs to hit
the 6 day limit and fail. This had gotten increasingly worse over
the last few months which is likely because the number of volumes
we consume grows over time (~+50/month). While the end-of-job
pruning scans would also have caused this
The reason for this is that while bacula is scanning for
appendable volumes it appears to issue queries against the catalog
regarding individual volumes in sequence (I guess this is to find
the list of jobs on each volume one at a time, then check the job
retention periods to see if any can be deleted?) We have several
thousand volumes, so executing these queries takes a while and I
get the feeling bacula will only do these in sequence, due to
either a lock or other single-threaded behaviour. So when there
were several of these operations queued up because multiple jobs
were seeking volumes at the same time, it's understandable why the
window to slip a bconsole query in and have a timely response was
very narrow and why most bconsole commands would take a very long
time to return.
Full disclosure, I did also upgrade to 7 at the same time as
making the pruning changes and the separate catalog connection may
also have helped with this, however both together certainly give
the desirable performance we used to experience.
Ben Roberts
> -----Original Message-----
> From: Kern Sibbald [mailto:kern AT sibbald DOT com]
> Sent: 25 November 2014 13:42
> To: bacula-users; bacula-devel
> Subject: [Bacula-users] Mini-Status report
>
> Hello,
>
> This is a mini-status report just to let you know that
someone on one of
> these lists said that the new white papers advise using:
>
> Maximum Concurrent Jobs = 1
>
> In the bacula-sd.conf Device resource. I have reviewed the
three white
> papers listed on the bacula.org site and I find no such
mention. Almost
> all the examples use MaximumConcurrentJobs=5 and several set
it to 10.
>
> If someone can point me to the place (if it exists) where 1
in mentioned,
> I will be happy to fix it.
>
> In addition, another person pointed out that manual_prune.pl
does not seem
> to be available. That was an oversight on my part, and that
file is now
> included as a download item under White Papers. The essential
elements of
> what manual_prune.pl does has been integrated directly into
the Bacula
> source code, and in the next few days I will update the
documentation to
> refer to the new way of doing "manual" pruning (i.e.
> pruning once a day rather than at the end of every Job).
>
> For those of you who are not aware of the pruning issues, you
need to know
> that when one has many jobs (say 1000 per day) and many
Volumes, the
> normal Job pruning that Bacula does at the end of each Job
can become a
> performance problem (not likely with less than 50 jobs per
day). To
> resolve it, it is best to turn off automatic pruning but then
to schedule
> an Admin job once a day that will do the equivalent. The
program (now
> integrated into the code) manual_prune.pl was written by
Bacula Systems
> for the Enterprise customers, but is published for your use
with the
> community version.
>
> Finally, I have made a few very minor changes to the Best
Practices for
> Disk Based Backup document.
>
> Best regards,
> Kern
>
>
--------------------------------------------------------------------------
> ----
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT
Server from
> Actuate! Instantly Supercharge Your Business Reports and
Dashboards with
> Interactivity, Sharing, Native Excel Exports, App Integration
& more Get
> technology previously reserved for billion-dollar
corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clkt
> rk
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
This email and any files
transmitted with it contain confidential and proprietary
information and is solely for the use of the intended recipient.
If you are not the intended recipient please return the email to
the sender and delete it from your computer and you must not
use, disclose, distribute, copy, print or rely on this email or
its contents. This communication is for informational purposes
only. It is not intended as an offer or solicitation for the
purchase or sale of any financial instrument or as an official
confirmation of any transaction. Any comments or statements made
herein do not necessarily reflect those of GSA Capital. GSA
Capital Partners LLP is authorised and regulated by the
Financial Conduct Authority and is registered in England and
Wales at Stratton House, 5 Stratton Street, London W1J 8LA,
number OC309261. GSA Capital Services Limited is registered in
England and Wales at the same address, number 5320529.
|
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk _______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
|
|