Bacula-users

Re: [Bacula-users] how to debug a job

2015-01-21 18:17:53
Subject: Re: [Bacula-users] how to debug a job
From: Heitor Faria <heitor AT bacula.com DOT br>
To: Dimitri Maziuk <dmaziuk AT bmrb.wisc DOT edu>
Date: Wed, 21 Jan 2015 20:12:25 -0300 (BRT)
> I've a client with ~316GB to back up. Currently the backup's been
> running for 5 days and wrote 33GB to the spool file. Previous runs
> failed with
> 
> > User specified Job spool size reached: JobSpoolSize=49,807,365,050
> > MaxJobSpoolSize=49,807,360,000
> > Writing spooled data to Volume. Despooling 49,807,365,050 bytes ...
> > Error: Watchdog sending kill after 518401 secs to thread stalled reading
> > File daemon.
> 
> Why is it taking 5 days to write 33GB?
> 
> Load avg on the client is 0.9%. Iperf clocks the connection at 110MB/s.
> Iostat shows zero wait and .25MB/s read on the client's disk. every few
> seconds bacula-fd shows up in iotop w/ read speed around 200-300K/s.
> This is a healthy standard sata drive capable of 100MB/s, with ext4
> filesystem.

Hey Mr. Dimitri: do you have Attribute Spooling on for this job (Job resource, 
Spool Attributes=yes)? It usually improves the performance if backing up lots 
of files, witch maybe causing this bottleneck.
We have been discussing in this list also some extreme tuning (e.g.: postgresql 
synchronous_commit = on | mysql innodb_flush_log_at_trx_commit = 0 / 
innodb_flush_method = O_DIRECT) for Bacula databases and other performance 
elements. Please read (machine translation) http://www.bacula.com.br/?p=2526 
and PVT me if you have any insights.
 
> It's a linux (centos 6) x64 client v. 5.0 and server v. 5.2.13 from
> slaanesh repo.

I'ts really advisable to upgrade your client to a 5.2.* version.  
 
> How do I find out what's taking so long? What's the debug level I should
> give to bacula-fd? Where do debug messages go? Anyone knows?

"setdebug

This command is used to set the debug level in each daemon. The form of this 
command is:
setdebug level=nn [trace=0/1 client=<client-name> | dir | director | 
storage=<storage-name> | all]

If trace=1 is set, then tracing will be enabled, and the daemon will be placed 
in trace mode, which means that all debug output as set by the debug level will 
be directed to the file bacula.trace in the current directory of the daemon. 
Normally, tracing is needed only for Win32 clients where the debug output 
cannot be written to a terminal or redirected to a file. When tracing, each 
debug output message is appended to the trace file. You must explicitly delete 
the file when you are done."

Source: http://www.bacula.org/2.4.x-manuals/en/main/Bacula_Console.html

Regards,
============================================================================== 
Heitor Medrado de Faria - LPIC-III | ITIL-F 
12 a 23 de janeiro de 2015 - Treinamento Telepresencial Bacula: 
http://www.bacula.com.br/?p=2174 
61 2021-8260 | 8268-4220 
Site: www.bacula.com.br | Facebook: heitor.faria | Gtalk: heitorfaria AT gmail 
DOT com 
=============================================================================== 

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users