Re: [Bacula-users] LTO speed optimisation

Hi all,

Thanks for your input and apologies for my much delayed reply. Backups only forms part of my daily routine and for the past few weeks I have been focussed on other things, as often happens.

> Hmm, I do not believe in this. I think that this is too expensive and too
> inefective solution. Too many mechanical disks. Which volume organization
> do you use? ZFS: zpool list -v?
>
> 46 mechanical disks can provide really big aggregate speed for many
> parallel data streams, but if you have just one or two 200 MB/s read
> streams and several slow write streams, it is very hard work for
> mechanical disks.

As a correction to my earlier statement, there are actually 44 disks in this array. These are setup with ZFS using 4x raidz2 vdevs each consisting of 11 disks splayed across all the internal I/O controllers.

We do track disk usage stats and I've just checked them historically. There are occasional spikes but for the majority of the time the disks are operating significantly below their maximums. The attached graph depicts the iostat percent_b and percent_w stats per disk and there is very little in the upper half of the graph where all the data points would be if the disks were maxed out.

For what it's worth this machine was not spec'd specifically for Bacula. It was a former fileserver and has been repurposed which is why it seems a bit overpowered in terms of number of spindles for what it's now doing.

> However in any case, observe CPU utilization too. And do not forget to
> compile Bacula with compiler optimizations, e. g. gcc -O3.
> And I expect that you have Spool Data = "" and you do not have Spool
> Attributes = no.

The orange line is the 1min load average. This is a 12-core box, so a load average of 2-3.5 while running backups seems entirely reasonable. Bacula was built with gcc -O2. Spool Data and Spool Attributes are both explicitly enabled, although from the documentation I believe enabling data spooling automatically forces attribute spooling on anyway.

I spool attributes for all jobs (disk and tape), however in this case it's pretty irrelevant as I'm using zfs send/receive with bpipe so there is only one file entry involved for the whole job.

As I mentioned below while two backup jobs were running to different tape drives (so with 2 parallel write streams), I was still able to use zfs send/receive to read/write another parallel datastream at 300mB/s so I'm certain there's additional capacity not being utilised by Bacula. I also observe the same despool throughput whether there's one job or two running concurrently so there seems to be a component other than disk I/O causing the bottleneck.

> You can try to switch off tape drive compression. There is some
> possibility, that write speed could increase. With tape drive compression,
> you have to have bigger throughput, atleast 200 MB/s.

I can give this a go and see if it has any effect.

> Check "Maximum Network Buffer Size" in your configs.

I don't currently have this configured anywhere; I will see what I can change it to and what effect this might have.

> I currently have a project to ease the problems of changing block sizes for
> tapes in Bacula (hopefully by year end)

I look forward to the announcement when this is ready. I anticipate raising the block size will help, though I shall heed your warning about raising it too far. The drives are relatively new and I'm not regularly writing to anything other than brand new tape media so old/dirty tapes/drives is less of a concern. I also do restore from a subset of all tapes written every month and thanks to ZFS checksums, if the restore completes successfully I can be confident the backup media is bit-perfect so I would detect media errors quickly.

> I think (and hope) that the only problem you will get into, is that if you
> write a volume with 512K blocks, currently, you will not be able to read it
> back with a Bacula configured for 64K block sizes.

My main concern is reconfiguring Bacula to write larger block sizes, then not be able to read older media with smaller block sizes if I have to restore old data. If you don't think this will be a problem, then I'm happy to try and report back. The warnings in the documentation have scared me off doing this so far.

Thanks again for your input,
Ben Roberts

This email and any files transmitted with it contain confidential and proprietary information and is solely for the use of the intended recipient. If you are not the intended recipient please return the email to the sender and delete it from your computer and you must not use, disclose, distribute, copy, print or rely on this email or its contents. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. Any comments or statements made herein do not necessarily reflect those of GSA Capital. GSA Capital Partners LLP is authorised and regulated by the Financial Conduct Authority and is registered in England and Wales at Stratton House, 5 Stratton Street, London W1J 8LA, number OC309261. GSA Capital Services Limited is registered in England and Wales at the same address, number 5320529.

disk-usage.png
Description: disk-usage.png

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users