Bacula-users

Re: [Bacula-users] jbd2 at 99.99% usage with very low speed

2012-02-10 09:14:28
Subject: Re: [Bacula-users] jbd2 at 99.99% usage with very low speed
From: Cristóbal Sabroe Yde <cris AT uti.unrc.edu DOT ar>
To: bacula-users AT lists.sourceforge DOT net
Date: Fri, 10 Feb 2012 11:11:49 -0300
El 09/02/12 13:46, Cristóbal Sabroe Yde escribió:
Hi, I've just installed a new backup system consisting of An IBM TS3100 ( LTO5) with bacula 5.2.5 running on an openSUSE 12.1 x86_64 server.

I'm having VERY low backup speed with 99.99% io usage from jbd2 and don't know what could be causing it. I see this behaviour only when running a job in bacula (a local backup or network with spool and without spool). I've tried intensive disk usage, intensive mysql usage and tape testing (tar and btape) and the problem doesn't occur.

The root fs is ext4 w/journal.
bacula is from this repository: http://download.opensuse.org/repositories/Archiving:/Backup:/Factory/openSUSE_12.1

This is what I see in iotop when backing up:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 433.97 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                     
  350 be/3 root        0.00 B/s    3.91 K/s  0.00 % 99.99 % [jbd2/sda2-8]
 2367 be/4 mysql       0.00 B/s   74.28 K/s  0.00 %  0.02 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/~/var/run/mysql/mysqld.pid --socket=/var/run/mysql/mysql.sock --port=3306
  [...]

Client status when running a backup:

listas Version: 5.0.2 (28 April 2010)  x86_64-unknown-linux-gnu suse 5.x
Daemon started 08-Feb-12 15:16, 3 Jobs run since started.
 Heap: heap=1,794,048 smbytes=903,251 max_bytes=920,534 bufs=1,320 max_bufs=1,944
 Sizeof: boffset_t=8 size_t=8 debug=0 trace=0

Running Jobs:
JobId 13 Job listas.2012-02-09_12.23.40_04 is running.
    Full Backup Job started: 09-Feb-12 11:01
*   Files=65,405 Bytes=2,513,983,277 Bytes/sec=689,896 Errors=0*
    Files Examined=65,439
    Processing file: /usr/share/fonts/misc/gb24st.pcf.gz
    SDReadSeqNo=5 fd=5
Director connected at: 09-Feb-12 12:01
====

These are my Device and Autochanger definition:
Device {
        Name = lto5
        Device Type = Tape
        Media Type = lto5
        Archive Device = /dev/nst0
        AutomaticMount = yes;
        AlwaysOpen = yes;
        RemovableMedia = yes;
        RandomAccess = no;
        Maximum File Size = 5GB
        AutoChanger = yes
        Alert Command = "sh -c 'smartctl -H -l error %c'" 
        Spool Directory = /spool            # fs (ext3) mounted from a dedicated hdd
        Maximum Spool Size = 268435456000       # 250 GB ( 60*1073741824)
}
Autochanger {
        Name = Autochanger
        Device = lto5
        Changer Command = "/usr/lib64/bacula/mtx-changer %c %o %S %a %d"
        Changer Device = /dev/sg3
}

Does anybody had this issue?


--
Logo
          UTI .Cris.
This is killing my brain.
I've manage to improve backup the speed of network backups (although not as much as our production backup system (SCSI LTO4 + openSUSE 11.2 + bacula 5.0.3)

Network backup on new server:

  Elapsed time:           8 mins 32 secs
  Priority:               90
  FD Files Written:       85,786
  SD Files Written:       85,786
  FD Bytes Written:       4,242,555,207 (4.242 GB)
  SD Bytes Written:       4,253,503,233 (4.253 GB)
  Rate:                   8286.2 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no

Network backup on production:
  Elapsed time:           9 mins 1 sec
  Priority:               90
  FD Files Written:       95,052
  SD Files Written:       95,052
  FD Bytes Written:       5,506,906,966 (5.506 GB)
  SD Bytes Written:       5,519,304,638 (5.519 GB)
  Rate:                   10179.1 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no


I believe the problem is in the jounaling of the ext4 filesystem. I've set 'nobarrier' as a mounting option of my spooling device and got a big improvement.
Still have problems backing up the local server.

Setting the same option on the root fs didn't solve the complete issue.

Has anyone experienced problems like this on openSUSE 12.1?
Which filesystem type do you recommend for the bacula server?

Thanks

--
Logo
        UTI .Cris.
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>