Reading the mail archive, it seems I’m not alone in wanting to see concurrent spooling/despooling.
Could we hand off the job to a standalone spooler like mbuffer (http://www.maier-komor.de/mbuffer.html)? For a simple tar to LTO-4 job it makes a tremendous improvement.
The Bacuala developers would need to modify the storage daemon to allow the datastream to be diverted out and then back in through a pair of FIFOs before it gets [properly re-blocked? and] written to tape. The user would need to tie the
FIFOs together with an external spooler. It might look like this:
------/etc/bacula-sd.conf------
<snip>
Device {
…
PushToFIFOSpooler = /path/to/bufreceive # bufreceive is a FIFO on this machine
PullFromFIFOSpooler = /path/to/bufsend # bufsend is also a FIFO
…
}
</snip>
------/etc/bacula-dir.conf------
<snip>
Job {
…
RunBeforeJob = "mbuffer -i /path/to/bufreceive -o /path/to/bufsend" # default puny 2M RAM buffer; can be set to multi-gig RAM or disk
# Alternatively: RunBeforeJob = "mbuffer < /path/to/bufreceive > /path/to/bufsend"
…
}
</snip>
This would
-
be a lot easier than asking the developers to reinvent the wheel (insert your own ring-buffer pun here…)
-
let the developers all but ignore spooling & focus on real backup stuff
-
give users access to many more options, such as using RAM or disk spool, high/low-watermarks, and rate-limiting
-
let folks cleverer than me figure out other cool things to do with the datastream
Thanks,
Dan Stieneke
IT Specialist
USDA - ARS - NWISRL
3793 N 3600 E
Kimberly, ID 83341
208/423-6519
