I have to say that so far I've been very impressed with Bacula, my biggest struggle has been finding the time to dedicate to it, and not trying to do everything using the product we currently use as a reference point, which is hard when you've used it for 10 years every day.
I think I get the "centralise" argument, but we may see that Linux file server grow to be 30-40TB over the next few years and even whilst it won't all need backing up I would be concerned if the folks using it do a job that generates a huge amount of data as we just don't have 10GbE everywhere yet so I can't help but think that being able to throw a £2.5k autoloader on that server directly just to deal with that one box may be simpler but I'm very open to reasons for and against.
The SSD speed delta makes sense, with our current backup product there is a similar database and their spinning disk requirement was insane vs. a single SSD (their database is disposable worst case, though it's still backed up daily to avoid grief). We'd backup the database with Bacula too.
In our case the SSD is Intel S3500 which is "enterprise" grade by most definitions, we have one free 2.5" bay in the server should another be needed to RAID up, or I guess we have internal options - the box wasn't purchased with Bacula specifically in mind, plans have changed a little.
I'm going to do some reading but any clarity around the "spool" function where tape is concerned would be good as I'm not entirely clear in my head if it relates to backing up directly to tape, or if you're doing D2D2T for the 2T part?
Thanks again - appreciate the time you took there.
-------- Original Message --------
Subject: Re: [Bacula-users] Hardware Specs & Sizing for Bacula?
Local Time: February 19, 2016 10:02 pm
UTC Time: February 19, 2016 10:02 PM
From: ajb2 AT mssl.ucl.ac DOT uk
To: paul.hutchings AT protonmail DOT com
CC: bacula-users AT lists.sourceforge DOT net
On 19/02/16 19:10, paul.hutchings
wrote:
Alan thanks, I omitted that we have a Spectra LTO6 library
which would be SAS attached to the server in question but I
didn't mention it as my initial query was more about the
hardware specs.
It all ties together.
The rough plan would be D2D2T and we'd probably run one of
our fileservers (linux) directly to a local directly attached
small LTO6/7 library as it's not data where we need a long
retention and it feels dumb to be running it over the network
just to send it to tape to keep for a week.
You're better off centralising it. Seriously. Even if you're only
keeping the backups a few days.
The hardware we have happens to have an 800GB SSD in it by a lucky
coincidence,which I thought could be
used for the Postgres database (not used Bacula enough to know
how big the database may grow)
800Gb is big enough, but is it fast enough? ("There are SSDs and
there are SSDs"),
With the kind of use it's getting you need to know the speed of
garbage collection as this is going to be the driving factor far
more than trim commands. On top of that you need to know the
endurance of the drives so you can calculate when they'll need
replacing (consumer SSDs run about 1000 times capacity, enterprise
will generally run to 100 times more than that.)
By way of comparison, 2million file full backups were taking hours
to insert attributes to the database on a raid 6 6-drive spinning
set. That came down to "5 minutes" when I moved to a raid1 pair of
samsung 840pro 500GB, but became "an hour" after a while. Flushing
and trimming the disks brought the speed back down, but as the same
blocks are being repeatedly written there's no trim sent in normal
operation and they're gradually slowing down again even though
they're now on a controller which supports trim commands. Whilst
840s are fairly notorious for their GC speed they're faster than
most consumer drives and the plan is to replace them with a pair of
SM843s as 500Gb isn't large enough anyway.
Raid is a must - you really don't want to try a restore without an
intact database. This is worst -case disaster scenario material and
you need to treat the database as business-critical - which it is
when things go wrong. (The database can also function as an IDS
(intrusion detection system). Bacula manuals have details on how to
use it for that)
> which I image would benefit from it immensely but I'm not clear
what the "spool" is that you're referring to - a quick dig suggests
it could be attribute spooling or a spool area for data that's going
to tape?
Correct on both counts - and if you're feeding LTO-anything you MUST
spool or you'll shoeshine the tapes and you MUST use SSDs for
concurrent backups on anything faster than LTO3 as the raw speed of
the tapes is faster than the sequential read speed of even 15krpm
drives.
The moment you start randomly seeking on spinning media your
throughput and iops will plummet, raid or no raid. On top of that,
spinning drives used for spool will self-destruct regularly (even
HGST 7k4s) due to the cumulative seek load, which translates to
unnecessary downtime and hassle.
On the current back up box I'm using a raid0 5-disk set of 64GB
Intel E25 drives. At the time they were $800 each and the spool area
is really only about half the size it needs to be. Spending that
kind of money now will get you an extremely nice, blisteringly fast
PCIe SSD. You need at least 600MB/s (sustained) so stay away from
SAS/SATA for the spool.
Sounds like we're good on the hardware but if necessary throw
in some RAM.
If you keep with the plan of spinning drives, you'll regret it very
quickly.
More ram is a must, as is proper database tuning. (postgres is good
but you need to tell it how much ram there is available to use and
give it optimisations for ssds. Mysql is tuning hell)
Use separate SSDs for the database and spool. Consider SSD for the
OS
Use software RAID and dump the PERCs unless you can switch them to
IT (initiator-target) mode from IR (Initiator-raid), as they'll slow
you down (PERCs are mpt2sas based - this is a low end SAS chipset
with significant RAID performance limits)
The spool device is disposable. Everything else is not. Your backup
system needs to be treated as business-critical and built
accordingly, along with the tape storage (a filing cabinet or
shelves is nowhere near good enough). When things go bang you need
it to work first time in order to be up and running as quickly as
possible.
Some of the more paranoid people I know use 3-4 way raid1 mirroring
on the OS and database disksets, specifically so they can keep one
disk from each raidset in the datasafe at all times.
With regard to safes: We use 2 of the large ones pictured at
http://www.phoenixsafeusa.com/primary-designation/media-safes -
these hold ~800 LTOs apiece and they should be positioned close to
your tape library - which in turn should be in a temperature/humidty
controlled dust-free environment _out_ of your main server areas
(The last thing you want if the server rooms catch fire is to lose
your backup system too and server rooms always end up dusty, which
kills tape drives)
If you buy a lot of tapes, consider a LTO cleaner from mptapes.com -
most tape-drive related contamination incidents we've seen have been
the result of new media with contamination on it contaminating the
drives, which in turnm crosscontaminated a lot of other tapes. This
shows as drives requesting excessive cleaning cycles and tapes
showing as "full" at significantly less than their raw capacity (in
the worst cases tapes were only holding 100Gb of data, the rest was
taken up by rewrites)
We're
so new to Bacula that I'll be blunt and admit there's lots I
simply haven't got my head around yet if we do go with it so
apologies if some of this is dumb/obvious to most of you :)
Bacula installations range from home systems to major banks. There's
no "one size fits all" but there are some fairly important
guidelines you need to adhere to in order to ensure that your
backups are there and usable when you need them (which is always a
high-stress event no matter if it's "I just deleted XYZ important
file and I need it back NOW" or "the main fileserver caught fire and
we need to rebuild it", so plan ahead)
As a rule of thumb for LTO - try not to let individual backup sets
go much over 1TB. The bigger they are, the greater the chances of
something going wrong during the backup/restore procedure and you
don't want full backups going over 24 hours in any case as this
starts interfering with daily backups of the backup server itself.
If someone tells you they need a 12TB filesystem, its quite likely
they don't and they haven't thought through what happens if it needs
fscking (which is another good reason for keeping backed-up filesets
under 1TB. Beyond that fscks at startup can eat a lot of time even
when parallelised. One such machine here gets rebooted every 6
months and usually spends a day in fsck before it's ready for use.)
-------- Original Message --------
Subject: Re: [Bacula-users] Hardware Specs & Sizing for
Bacula?
Local Time: February 19, 2016 6:58 pm
UTC Time: February 19, 2016 6:58 PM
On 19/02/16 18:12, paul.hutchings
wrote:
We're new to Bacula and are still considering if it's viable for us.
Our test environment is quite small (it is a test environment) and when I read the docs I'm not sure how recent they are when they relate to hardware specs.
For example if I were to suggest box with dual 8 core E5 CPUs, hardware PERC RAID card with 1GB cache, 48TB of 7.2k SATA in RAID6 and 32GB (or more) of RAM running as a SD would people be thinking "hmmm may need more horsepower" or would people be thinking "that should handle hundred/thousands of clients"?
It depends. For SD-only use, your CPU is overkill and even
16Gb of ram would be overkill
Ram requirements are for the director and database. These
can be on the same box and probably should be to avoid
networking penalties. You don't need VMs - and really
shouldn't play that game on backup-dedicated hardware as VMs
come with performance penalties ranging from noticeable to
major.
Even with the DB and DIR on the box, your CPUs are more
than adequate.
Assuming SD + DIR + Postgres (don't mess with Mysql for
million+file installations, it doesn't scale well) then, I'd
add more ram. It's cheap enough these days that you should
think about running at least 96GB if you're backing up tens of
TB and tens of millions of files (even more if you can afford
it)
The real issue if you're running backups at this scale:
Disk is a liability. It's too slow and drives will end up
shaking themselves to pieces, making the backup pool your
Single point of failure. You _need_ tape - A decent robot and
several drives along with a suitably sized data safe.
We currently back up about 250 million files over 400TB
and I'm currently using a Quantum i500 with 14U extension and
6 SAS LTO6 drives, previously we had a Overland Neo8000 with 7
FC LTO5 drives.
Once you bite the bullet and use tape, dump the sata
spinning disks. Use something like a raid1 pair of 500GB
SM843s for your OS, put in second dedicated 1TB raid1 pair for
the database and use a _fast_ 200-800GB PCIe flash drive for
spool.
10GB networking is an absolute must. Don't try to play
games with 1Gb/s bonding. Any given data stream will only run
at 1Gb/s maximum.
On the other hand, the setup above would be an expensive
waste of time for backing up 10TB of data - although for that
size you could keep the spinning media and keep the rest - but
bear in mind that 48TB is only going to allow 3 full backups
of 15TB (any fewer than 3 full backups is asking for trouble),
without taking differentials or incrementals into account.
For 20TB+ you may want to look at a single-drive tape
autochanger capable of holding at least 10 tapes. The last
thing you want to be doing is feeding new LTO6/7s into it
every 2-3 hours when a full backup is running (yes, they will
fill up that quickly)
Director could be on the same physical box but would ideally be a VM with a couple of CPU cores and as much RAM as is needed to handle a couple dozen clients, though the largest two clients are around 10TB and each have millions of files.
Impression I get is that network and disk will be a bottleneck way before RAM and CPU should be?
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users