Bacula-users
[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

[Bacula-users] Jobs are not running concurrently as expected (otherwise multiplexing not working ?)

2011-04-04 14:54:32
Subject: [Bacula-users] Jobs are not running concurrently as expected (otherwise multiplexing not working ?)
From: Sarder Kamal <KamalS AT wmcollege.ac DOT uk>
To: "'bacula-users AT lists.sourceforge DOT net'" <bacula-users AT lists.sourceforge DOT net>
Date: Mon, 4 Apr 2011 18:39:03 +0000
Dear List Members
 
I am trying to configure bacula to back up nearly 1.5TB of data, which is always timing out since the backup does not complete before the next backup. Ideally, I would prefer the backup to begin after eveyone is out of office and finish before anyone is in -- which gives me a 12 hour (different people finishing at different time) window for the task, and somehow it is not working out.
 
Below is my present setup
 
1 bacula server running all dir, fd and sd. The server has 4TB internal drive dedicated for storage, and all data is acquired via smb mounts
 
In order to make life simple, I have broken down the total storage to small groups like group1, group2, group3, etc.
 
Therefore, the dir.conf looks like this
 
<code>
Director {                            # define myself
  Name = bkula.localhost-dir
  DirAddress = 192.168.0.21
  DIRport = 9101                # where we listen for UA connections
  QueryFile = "/usr/local/share/bacula/query.sql"
  WorkingDirectory = "/var/db/bacula"
  PidDirectory = "/var/run"
  Maximum Concurrent Jobs = 13
  Password = "<SecRet>"
  Messages = Daemon
}
 
Job {
  Name = "My-BackupCatalog"
  Pool = "CatalogueBackup"
  JobDefs = "MyDefaultJob02"
  Level = Full
  FileSet="Catalog"
  Schedule = "MyWeeklyCycleAfterBackup"
  # This creates an ASCII copy of the catalog
  # Arguments to make_catalog_backup.pl are:
  #  make_catalog_backup.pl <catalog-name>
  RunBeforeJob = "/usr/local/share/bacula/make_catalog_backup.pl MyCatalog"
  # This deletes the copy of the catalog
  RunAfterJob  = "/usr/local/share/bacula/delete_catalog_backup"
  Write Bootstrap = "/StdBkup/BSR/%n.bsr"
  Priority = 11                   # run after main backup
  Max Start Delay = 43200
}
 
#
 
Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=bkula.localhost-fd                
  FileSet="Full Set"                 
  Storage = My-NAS-02                 
  Pool = Default
  Messages = Standard
  Where = /mnt/NAS/StdntBackup000
  Priority = 7
}
 
Storage {
  Name = My-NAS-02
# Do not use "localhost" here   
  Address = bkula                # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "<SecRet>"
#   Device = FileStorage
  Device = My-NAS-02
  Media Type = File
  Maximum Concurrent Jobs = 13
}
 
 
 
# Client (File Services) to backup
Client {
  Name = bkula.localhost-fd
  FDAddress = 192.168.0.21
  Address = bkula
  FDPort = 9102
  Catalog = MyCatalog
  Password = "<SecRet>"          # password for FileDaemon
  File Retention = 15 days            # 15 NOT 30 days
  Job Retention = 1 months            # one NOT six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}
 
 
# Definition of file storage device
Storage {
  Name = File00
# Do not use "localhost" here   
  Address = bkula                # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "<SecRet>"
  Device = StoreCube00
  Media Type = File
  Maximum Concurrent Jobs = 13
}
 
 
 
 
# Definition of file storage device
Storage {
  Name = File01
  Address = bkula                # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "<SecRet>"
  Device = StoreCube01
  Media Type = File
  Maximum Concurrent Jobs = 13
}
 
 
 
# Definition of file storage device
Storage {
  Name = File02
  Address = bkula                # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "<SecRet>"
  Device = StoreCube02
  Media Type = File
  Maximum Concurrent Jobs = 13
}
 
 
 
# SiteSpecific Catalogue Service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:mysql"; dbaddress = 127.0.0.1; dbport = 
#  dbdriver = "dbi:mysql"; dbaddress = localhost; dbport = /tmp/mysql.sock
#  dbdriver = "dbi:mysql"; dbaddress = localhost; dbport = 3306
  dbdriver = "dbi:mysql"; dbaddress = localhost; dbsocket = /tmp/mysql.sock
  dbname = "bacula"; dbuser = "bacula"; dbpassword = "<secRet>"
}
 
 
 
# Reasonable message delivery -- send most everything to email address
#  and to the console
Messages {
  Name = Standard
#
  mailcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
  mail = root@localhost = all, !skipped           
  operator = root@localhost = mount
  console = all, !skipped, !saved
#
# WARNING! the following will create a file that you must cycle from
#          time to time as it will grow indefinitely. However, it will
#          also keep all your messages if they scroll off the console.
#
  append = "/var/db/bacula/log" = all, !skipped
  catalog = all
}
 
 
#
# Message delivery for daemon messages (no job).
Messages {
  Name = Daemon
  mailcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r"
  mail = root@localhost = all, !skipped           
  console = all, !skipped, !saved
  append = "/var/db/bacula/log" = all, !skipped
}
 
# Default pool definition
Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365 days         # one year
}
 
# File Pool definition
Pool {
  Name = File
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365 days         # one year
  Maximum Volume Bytes = 50G          # Limit Volume size to something reasonable
  Maximum Volumes = 100               # Limit number of Volumes in Pool
}
 
 
# My Pool definition
Pool {
  Name = MyPool
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 30 days         # one year
  Maximum Volume Bytes = 100M         # Limit Volume size to something reasonable
  Maximum Volumes = 100               # Limit number of Volumes in Pool
}
 
 
# Scratch pool definition
# Scratch pool definition
Pool {
  Name = Scratch
  Pool Type = Backup
}
 
#
# Restricted console used by tray-monitor to get the status of the director
#
Console {
  Name = bkula.localhost-mon
  Password = "<SecRet>"
  CommandACL = status, .status
}
 
 
 
 
 
#########################
### further definitions begins here
#########################
 
 
JobDefs {
  Name = "MyDefaultJob01"
  Type = Backup
  Level = Incremental
  Client = bkula.localhost-fd
  FileSet = "Full Set"
  Schedule = "MyWeeklyCycle"
  Storage = File00
  Messages = Standard
  Pool = Default
  Priority = 10
  Max Start Delay = 46800
  Maximum Concurrent Jobs = 13
}
 
JobDefs {
  Name = "MyDefaultJob02"
  Type = Backup
  Level = Incremental
  Client = bkula.localhost-fd
  FileSet = "Full Set"
  Schedule = "MyWeeklyCycle"
  Storage = File00
  Messages = Standard
  Pool = Default
  Priority = 10
  Max Start Delay = 46800
  Maximum Concurrent Jobs = 13
}
 
JobDefs {
  Name = "MyDefaultJob03"
  Type = Backup
  Level = Incremental
  Client = bkula.localhost-fd
  FileSet = "Full Set"
  Schedule = "MyWeeklyCycleWin"
  Storage = File00
  Messages = Standard
  Pool = Default
  Priority = 10
  Max Start Delay = 46800
  Maximum Concurrent Jobs = 13
}
 
 
Pool {
  Name = CatalogueBackup
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 65 days         # one year
  Maximum Volumes = 100               # Limit number of Volumes in Pool
}
 
 
 
 
Schedule {
  Name = "MyWeeklyCycle"
  Run = Full sat at 17:00
  Run = Incremental mon-fri at 22:00
}
 
Schedule {
  Name = "MyWeeklyCycleWin"
  Run = Full sat at 17:00
  Run = Incremental mon-fri at 22:00
#  Run = Full sun at 00:03
#  Run = Incremental tue-sat at 00:03
}
 
 
# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
  Name = "MyWeeklyCycleAfterBackup"
  Run = Full sun-sat at 08:00
}
 
 
 
#######
### Main Backup Filesets and Jobs begin here
#######
 
 
#####################################################################
### Lacie AccessHE_ArtDesign             ### 200907081207
#####################################################################
 
FileSet {
  Name = "AccessHE_ArtDesign"
  Include {
    Options {
    #compression = GZIP
      signature = SHA1
      strippath = 2
        recurse = yes
        exclude = yes
       wildfile = "*._*"
       wildfile = "*.exe*"
       wildfile = "*.bat*"
       wildfile = "*.com*"
       wildfile = "*.sh*"
       wildfile = "*.pl*"
       wildfile = "*.cgi*"
       wildfile = "*.mp3*"
       wildfile = "*.mp4*"
       wildfile = "*.wav*"
       wildfile = "*.wma*"
       wildfile = "*.mov*"
       wildfile = "*.lnk*"
        wilddir = "*.Trashes*"
        wilddir = "*.TemporaryItems*"
        wilddir = "*.DS_Store*"
    }
 
    File = /mnt/Lacie/AccessHE_ArtDesign
  }
 
  Exclude {
    File = "/mnt/Lacie/AccessHE_ArtDesign/.TemporaryItems*"
    File = "/mnt/Lacie/AccessHE_ArtDesign/.Trashes*"
  }
 
}
 
Pool {
  Name = AccessHE_ArtDesign
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 15 days         # two weeks
}
 
Job {
  Name = "AccessHE_ArtDesign"
  JobDefs = "MyDefaultJob01"
  Storage = File06
  FileSet = "AccessHE_ArtDesign"
  Enabled = "Yes"
  Type = "Backup"
  Level = "Incremental"
  Client = bkula.localhost-fd
  Pool = "AccessHE_ArtDesign"
  Priority = 10
  Maximum Concurrent Jobs = 13
  Write Bootstrap = "/StdBkup/BSR/%n_%v_%j.bsr"
}
 
 
 
#####################################################################
### Lacie AccessHE_Music                ### 200907081207
#####################################################################
 
FileSet {
  Name = "AccessHE_Music"
  Include {
    Options {
    #compression = GZIP
      signature = SHA1
      strippath = 2
        recurse = yes
        exclude = yes
       wildfile = "*._*"
       wildfile = "*.exe*"
       wildfile = "*.bat*"
       wildfile = "*.com*"
       wildfile = "*.sh*"
       wildfile = "*.pl*"
       wildfile = "*.cgi*"
       wildfile = "*.mp3*"
       wildfile = "*.mp4*"
       wildfile = "*.wav*"
       wildfile = "*.wma*"
       wildfile = "*.mov*"
       wildfile = "*.lnk*"
        wilddir = "*.Trashes*"
        wilddir = "*.TemporaryItems*"
        wilddir = "*.DS_Store*"
    }
 
    File = /mnt/Lacie/AccessHE_Music
  }
 
  Exclude {
    File = "/mnt/Lacie/AccessHE_Music/.TemporaryItems*"
    File = "/mnt/Lacie/AccessHE_Music/.Trashes*"
  }
 
}
 
 
 
Pool {
  Name = AccessHE_Music
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 15 days         # two weeks
}
 
 
 
Job {
  Name = "AccessHE_Music"
  JobDefs = "MyDefaultJob01"
  Storage = File07
  FileSet = "AccessHE_Music"
  Enabled = "Yes"
  Type = "Backup"
  Level = "Incremental"
  Client = bkula.localhost-fd
  Pool = "AccessHE_Music"
  Priority = 10
  Maximum Concurrent Jobs = 13
  Write Bootstrap = "/StdBkup/BSR/%n_%v_%j.bsr"
}
#####################################################################
### Lacie DW_Mac                        ### 200907081207
#####################################################################
 
FileSet {
  Name = "DW_Mac"
  Include {
    Options {
    #compression = GZIP
      signature = SHA1
      strippath = 2
        recurse = yes
        exclude = yes
       wildfile = "*._*"
       wildfile = "*.exe*"
       wildfile = "*.bat*"
       wildfile = "*.com*"
       wildfile = "*.sh*"
       wildfile = "*.pl*"
       wildfile = "*.cgi*"
       wildfile = "*.mp3*"
       wildfile = "*.mp4*"
       wildfile = "*.wav*"
       wildfile = "*.wma*"
       wildfile = "*.mov*"
       wildfile = "*.lnk*"
        wilddir = "*.Trashes*"
        wilddir = "*.TemporaryItems*"
        wilddir = "*.DS_Store*"
    }
 
    File = /mnt/Lacie/DW_Mac
  }
 
  Exclude {
    File = "/mnt/Lacie/DW_Mac/.TemporaryItems*"
    File = "/mnt/Lacie/DW_Mac/.Trashes*"
  }
 
}
 
 
 
Pool {
  Name = DW_Mac
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 15 days         # two weeks
}
 
 
 
Job {
  Name = "DW_Mac"
  JobDefs = "MyDefaultJob01"
  Storage = File08
  FileSet = "DW_Mac"
  Enabled = "Yes"
  Type = "Backup"
  Level = "Incremental"
  Client = bkula.localhost-fd
  Pool = "DW_Mac"
  Priority = 10
  Maximum Concurrent Jobs = 13
  Write Bootstrap = "/StdBkup/BSR/%n_%v_%j.bsr"
}
 
 
 
 
#####################################################################
### Main Windows User Accounts          ### 200909241950
#####################################################################
 
FileSet {
  Name = "WindowsUsers"
  Include {
    Options {
    #compression = GZIP
      signature = SHA1
      strippath = 2
        recurse = yes
        exclude = yes
       wildfile = "*._*"
       wildfile = "*.exe*"
       wildfile = "*.bat*"
       wildfile = "*.com*"
       wildfile = "*.sh*"
       wildfile = "*.pl*"
       wildfile = "*.cgi*"
       wildfile = "*.mp3*"
       wildfile = "*.mp4*"
       wildfile = "*.wav*"
       wildfile = "*.wma*"
       wildfile = "*.mov*"
       wildfile = "*.jpg*"
       wildfile = "*.tif*"
       wildfile = "*.gif*"
       wildfile = "*.png*"
       wildfile = "*.psd*"
       wildfile = "*.lnk*"
        wilddir = "*.Trashes*"
        wilddir = "*.TemporaryItems*"
        wilddir = "*.DS_Store*"
        wilddir = "*Recycler*"
        wilddir = "*Temp*"
        wilddir = "*TEMP*"
        wilddir = "*temp*"
    }
 
    File = /mnt/NAS/StudentHomeS
  }
 
  Exclude {
    File = "/mnt/NAS/StudentHomeS/.Recycler*"
    File = "/mnt/NAS/StudentHomeS/Recycler*"
  }
 
}
 
 
 
Pool {
  Name = WindowsHomeDrives
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 15 days         # two weeks
}
 
 
 
Job {
  Name = "WindowsHomeDrives"
  JobDefs = "MyDefaultJob03"
  Storage = File20
  FileSet = "WindowsUsers"
  Enabled = "Yes"
  Type = "Backup"
  Level = "Incremental"
  Client = bkula.localhost-fd
  Pool = "WindowsHomeDrives"
  Priority = 10
  Maximum Concurrent Jobs = 13
  Write Bootstrap = "/StdBkup/BSR/%n_%v_%j.bsr"
}
 
</code>
 
 
 
 
============================================
 
Now for the bacula-fd.conf (this is short :D)
 
<code>
#
Director {
  Name = bkula.localhost-dir
  Password = "<SecRet>"
}
 
#
# Restricted Director, used by tray-monitor to get the
#   status of the file daemon
#
Director {
  Name = bkula.localhost-mon
  Password = "<SecRet>"
  Monitor = yes
}
 
#
# "Global" File daemon configuration specifications
#
FileDaemon {                          # this is me
  Name = bkula.localhost-fd
  FDAddress = 192.168.0.21
  FDport = 9102                  # where we listen for the director
  WorkingDirectory = /var/db/bacula
  Pid Directory = /var/run
  Maximum Concurrent Jobs = 21
}
 
# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = bkula.localhost-dir = all, !skipped, !restored
}
 
</code>
 
=================================================
 
and finally the SD
 
<code>
#
# Default Bacula Storage Daemon Configuration file
#
#  For Bacula release 5.0.3 (04 August 2010) -- freebsd 8.2-PRERELEASE
#
# You may need to change the name of your tape drive
#   on the "Archive Device" directive in the Device
#   resource.  If you change the Name and/or the
#   "Media Type" in the Device resource, please ensure
#   that dird.conf has corresponding changes.
#
 
Storage {                             # definition of myself
  Name = bkula.localhost-sd
  SDAddress = 192.168.0.21
  SDPort = 9103                  # Director's port     
  WorkingDirectory = "/var/db/bacula"
  Pid Directory = "/var/run"
  Maximum Concurrent Jobs = 21
}
 
#
# List Directors who are permitted to contact Storage daemon
#
Director {
  Name = bkula.localhost-dir
  Password = "<SecRet>"
}
 
#
# Restricted Director, used by tray-monitor to get the
#   status of the storage daemon
#
Director {
  Name = bkula.localhost-mon
  Password = "<SecRet>"
  Monitor = yes
}
 
#
# Note, for a list of additional Device templates please
#  see the directory <bacula-source>/examples/devices
# Or follow the following link:
#
 
#
# Devices supported by this Storage daemon
# To connect, the Director's bacula-dir.conf must have the
#  same Name and MediaType.
#
 
Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /tmp
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
 
Device {
  Name = StoreCube00
  Media Type = File
  Archive Device = /StdBkup
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
 
 
Device {
  Name = StoreCube01
  Media Type = File
  Archive Device = /StdBkup
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
 
 
Device {
  Name = StoreCube02
  Media Type = File
  Archive Device = /StdBkup
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
 
 
Device {
  Name = StoreCube03
  Media Type = File
  Archive Device = /StdBkup
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
 
Device {
  Name = WMC-NAS-02
  Media Type = File
  Archive Device = /mnt/NAS/StdntBackup000
  LabelMedia = Yes;
  Random Access = Yes;
  AutomaticMount = Yes;
  RemovableMedia = no;
  AlwaysOpen = no;
}
 
#
Messages {
  Name = Standard
  director = bkula.localhost-dir = all
}
</code>
 
----------------
I have stripped much of dir and sd file but tried to leave enough to give you an idea of what the setup looks like. My understanding was, if I put the jobs in different pool (as they are), along with different storages (even though they are pointing back to same /StdBkup folder, they are different definition in SD file), I would be able to run concurrent job. Unfortunately, that is not happening. The jobs are waiting for ONE running job to finish before they begin.
 
Therefore, I am a little lost. Is that not possible? or am I doing something stupid? If latter, then what should I be doing?
 
Any pointer/suggestion is highly appreciated.
 
Thank you.
Sarder
 
 
------------------------------------------------------------------------------
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself; 
WebMatrix provides all the features you need to develop and 
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  Re: [Bacula-users] what do you do when bacula stops working for no good reason?, Martin Simmons
Next by Date:  Re: [Bacula-users] Jobs are not running concurrently as expected (otherwise multiplexing not working ?), John Drescher
Previous by Thread:  [Bacula-users] can't create postgresql bacula database, contenidos2000-bacula
Next by Thread:  Re: [Bacula-users] Jobs are not running concurrently as expected (otherwise multiplexing not working ?), John Drescher
Indexes:  [Date] [Thread] [Top] [All Lists]