Re: [Bacula-users] slow despool to tape speed?

>-----Original Message-----
>From: Steve Ellis [mailto:ellis AT brouhaha DOT com]

>Your hardware is _much_ better than mine (spool in non-RAID, no SSD,
>LTO3, 4GB RAM, etc), yet I get tape throughput not a lot lower. At the
>despool rates you are seeing, I'm guessing you may be shoeshining the
>tape on LTO4. Here's the results I saw from my most recent 3 full
>backup jobs (presumably tape despool for incrementals is similar, but
>mine are so small as to include way too much start/stop overhead):

yes, I am which is why I'm posting to get some more ideas.

>
>I have a few questions about your setup:
>
>* do you have "Spool Attributes" enabled? -- I think this may help.
>

Yes, I am spooling data which automatically also turns on spooling of attributes.

>* have you increased the "Maximum File Size" in your tape config? --My
>tape throughput went up somewhat when I went to 5G for the file size (on
>LTO4 you might even want it larger, my drive is LTO3).

Yes, my -sd.conf is set up with:
Maximum Block Size = 4194304 (tried increments from 512K up to 16,000,000bytes but did not see much of a difference after 2M really)

Maximum File Size = 10G (tried 5G, 8G, 10G)

Maximum Job Spool Size = 800G (size of a single tape)

>* why are you using sqlite? I was under the impression it was
>non-preferred for several reasons (I'm using mysql, but I believe
>postgres is somewhat preferred).

This is a backup of a single host, did not see any real need for larger/more complex databases. Also since the last post I have just tried re-compiling bacula with
SQLITE3_INIT_QUERY "PRAGMA synchronous = OFF" as well as moving the database file to a ram disk for testing, no improvements so think that can be ruled out as an issue.

>* have you increased the "Maximum Network Buffer Size" (both SD and FD)
>and "Maximum Block Size" (SD only)? I have both at 256K, as I recall
>"Network Buffer Size" can't be larger (but I'm not sure that this will
>help your despool anyway), I think the block size change _may_ make
>previously written tapes incompatible, I don't remember why I went to
>256K on block size, but I believe the default is 64K.

Yes, this is set to 256K, though as I mentioned this is single-host so there is no external ethernet network involved here.

>If I were in your situation, I would see if "Spool Attributes" and
>"Maximum File Size" helped (these are easy to change, after all). If
>that didn't help, I would move to a different database backend (this is
>probably a lot harder). Then at that point, maybe "Maximum Block Size"
>(more reluctant to change if it does cause tape incompatibility).
>
>-se

Is there any memory prefetching of the spool file by the SD before it gets written to tape (i.e. is there a way to have the SD to say read read in data into a buffer or is that what the Maximum Block Size is supposed to do? (if so that seems kind of small being limited to just 20MB)?

Otherwise this may be something with the file system (getting esoteric here) even though I can read/write a data stream to the drives at well over 500MB/s there may be some interaction between how bacula's SD reads this against ZFS opposed to say a DD against the same file. (zfetch (zfs's version of a file prefetch) dumping data too fast or something?) Will have to try differnet file systems and/or different caching schemas or instead of having a seperate pool of 16 drives/raid-10 use the main pool (96 drives) and see what effect that has on performance.

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users