Bacula-users

[Bacula-users] storage daemon does not releases resources

2011-06-29 02:36:09
Subject: [Bacula-users] storage daemon does not releases resources
From: Vadim Zotov <z0termann AT mail DOT ru>
To: bacula-users AT lists.sourceforge DOT net
Date: Wed, 29 Jun 2011 10:33:11 +0400
Hello everybody

from some time I have an annoying problem:

- from time to time firewall drops connection for some hosts during
backup process, as a consequence backup jobs failed. this is normal.
- but storage daemon believes that such jobs are stiil active. when
amount of such failed jobs is greater than "maximum concurrent jobs"
in bacula-sd.conf all backup jobs becomes blocked.

I did not found any hints in bacula manual how to avoid this problem.
Probably I missed something. May be somebody met this problem or have
some ideas.

thank you in advance.

bacula version 5.0.3


Here below you can find bacula-sd.conf and output from
stat stor & stat dir commands.

$ sudo cat /etc/bacula/bacula-sd.conf

Storage {
Name = amanda-sd
SDPort = 9103
WorkingDirectory = "/var/bacula"
Pid Directory = "/var/run"
Maximum Concurrent Jobs = 5;
Heartbeat Interval = 29;
}
Director {
Name = amanda-dir
Password = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
Device {
Name = EON1
MediaType = File
Device Type = File
Archive Device = /mnt/backup
Label Media = Yes
Random Access = Yes
AutomaticMount = Yes
Always Open = Yes
RemovableMedia = No
Requires Mount = No
# MountPoint =
# MountCommand =
Maximum Open Wait = 300s
Maximum Concurrent Jobs = 4
}

Messages {
Name = Standard
director = amanda-dir = all
}

-------------------------------------------------------------------------------
**stat stor*
Automatically selected Storage: EonStor1
Connecting to Storage daemon EonStor1 at amanda:9103

amanda-sd Version: 5.0.3 (04 August 2010) x86_64-redhat-linux-gnu redhat
Daemon started 06-Jun-11 10:17. Jobs: run=274, running=8.
Heap: heap=2,383,872 smbytes=2,325,542 max_bytes=2,325,542 bufs=1,334
max_bufs=1,335
Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8

Running Jobs:
Writing: Incremental Backup job netinv JobId=26987 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDReadSeqNo=5 in_msg=4 out_msg=4 fd=6
Writing: Incremental Backup job netinv JobId=27066 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDReadSeqNo=5 in_msg=4 out_msg=4 fd=10
Writing: Incremental Backup job netinv JobId=27138 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDReadSeqNo=5 in_msg=4 out_msg=4 fd=12
Writing: Incremental Backup job netinv JobId=27210 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDReadSeqNo=5 in_msg=4 out_msg=4 fd=16
Writing: Incremental Backup job netinv JobId=27210 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
Writing: Full Backup job grkc JobId=27244 Volume="Daily-0005"
pool="MonthlyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
Writing: Full Backup job telex JobId=27245 Volume="Daily-0005"
pool="MonthlyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
Writing: Full Backup job rtelex JobId=27246 Volume="Daily-0005"
pool="MonthlyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
Writing: Incremental Backup job shepot-allsa JobId=27247 Volume="Daily-0005"
pool="DailyPool" device="EON1" (/mnt/backup)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
====

Jobs waiting to reserve a drive:
3609 JobId=27210 Max concurrent jobs exceeded on drive "EON1"
(/mnt/backup).
3609 JobId=27244 Max concurrent jobs exceeded on drive "EON1"
(/mnt/backup).
3609 JobId=27245 Max concurrent jobs exceeded on drive "EON1"
(/mnt/backup).
3609 JobId=27246 Max concurrent jobs exceeded on drive "EON1"
(/mnt/backup).
3609 JobId=27247 Max concurrent jobs exceeded on drive "EON1"
(/mnt/backup).
====

Terminated Jobs:
JobId Level Files Bytes Status Finished Name
===================================================================
27229 Incr 2 552.2 K OK 10-Jun-11 02:58 samba-ldap
27230 Incr 178 323.4 M OK 10-Jun-11 03:00 shepot
27231 Incr 70 5.016 M OK 10-Jun-11 03:00 sovintel-gw
27233 Incr 485 8.402 G OK 10-Jun-11 03:13 agbar
27234 Incr 76 23.13 M OK 10-Jun-11 03:15 zinger
27235 Incr 105 932.7 M OK 10-Jun-11 03:16 zip1
27236 Incr 13 8.174 K OK 10-Jun-11 03:16 zip2
27232 Full 0 0 Cancel 10-Jun-11 05:08 swist
27221 Full 0 0 Cancel 10-Jun-11 05:08 pnts1-way4
27220 Full 0 0 Cancel 10-Jun-11 05:08 pnts1
====

Device status:
Device "EON1" (/mnt/backup) is not open.
====

Used Volume status:
Daily-0005 on device "EON1" (/mnt/backup)
Reader=0 writers=0 devres=4 volinuse=1
====

Data spooling: 0 active jobs, 0 bytes; 17 total jobs, 9,123,795,343 max
bytes/job.
Attr spooling: 0 active jobs, 208,909,429 bytes; 255 total jobs,
208,909,429 max bytes.
====
-------------------------------------------------------------------------------
**stat dir*
amanda-dir Version: 5.0.3 (04 August 2010) x86_64-redhat-linux-gnu redhat
Daemon started 06-Jun-11 10:17, 297 Jobs run since started.
Heap: heap=3,080,192 smbytes=806,491 max_bytes=6,366,712 bufs=2,393
max_bufs=38,603

Scheduled Jobs:
Level Type Pri Scheduled Name Volume
===================================================================================
Incremental Backup 10 10-Jun-11 21:30 shepot-allsa Daily-0005
Differential Backup 9 11-Jun-11 00:15 pdb1-oracle Daily-0005
Differential Backup 9 11-Jun-11 00:15 rea-oracle Daily-0005
Differential Backup 9 11-Jun-11 00:15 pdb2-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 agate10 Daily-0005
Differential Backup 10 11-Jun-11 00:15 agate20 Daily-0005
Differential Backup 10 11-Jun-11 00:15 amanda Daily-0005
Differential Backup 10 11-Jun-11 00:15 amanda-winimg Daily-0005
Differential Backup 10 11-Jun-11 00:15 debt Daily-0005
Differential Backup 10 11-Jun-11 00:15 debt-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 equant-gw Daily-0005
Differential Backup 10 11-Jun-11 00:15 fagot Daily-0005
Differential Backup 10 11-Jun-11 00:15 fagot-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 forger Daily-0005
Differential Backup 10 11-Jun-11 00:15 forger-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 forger1-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 helpdesk-otrs Daily-0005
Differential Backup 10 11-Jun-11 00:15 io Daily-0005
Differential Backup 10 11-Jun-11 00:15 io-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-dump Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-home Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-alliance Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-ldap Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-mysql Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-postgres Daily-0005
Differential Backup 10 11-Jun-11 00:15 kappa-space Daily-0005
Differential Backup 10 11-Jun-11 00:15 kc Daily-0005
Differential Backup 10 11-Jun-11 00:15 lists Daily-0005
Differential Backup 10 11-Jun-11 00:15 lists-www Daily-0005
Differential Backup 10 11-Jun-11 00:15 swist Daily-0005
Differential Backup 10 11-Jun-11 00:15 zinger Daily-0005
Differential Backup 10 11-Jun-11 00:15 zip2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 zip1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 agbar Daily-0005
Differential Backup 10 11-Jun-11 00:15 sovintel-gw Daily-0005
Differential Backup 10 11-Jun-11 00:15 lists-mysql Daily-0005
Differential Backup 10 11-Jun-11 00:15 miracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 miracle-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 nbs1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 nbs1-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 nbs2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 nbs2-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 netinv Daily-0005
Differential Backup 10 11-Jun-11 00:15 netinva Daily-0005
Differential Backup 10 11-Jun-11 00:15 olh1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 olh2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 omega Daily-0005
Differential Backup 10 11-Jun-11 00:15 omega-ldap Daily-0005
Differential Backup 10 11-Jun-11 00:15 petrol Daily-0005
Differential Backup 10 11-Jun-11 00:15 petrol-oracle Daily-0005
Differential Backup 10 11-Jun-11 00:15 pdb1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 pdb2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 pnts1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 pnts1-way4 Daily-0005
Differential Backup 10 11-Jun-11 00:15 pnts2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 pnts2-way4 Daily-0005
Differential Backup 10 11-Jun-11 00:15 q3 Daily-0005
Differential Backup 10 11-Jun-11 00:15 rea Daily-0005
Differential Backup 10 11-Jun-11 00:15 sad1 Daily-0005
Differential Backup 10 11-Jun-11 00:15 sad2 Daily-0005
Differential Backup 10 11-Jun-11 00:15 samba Daily-0005
Differential Backup 10 11-Jun-11 00:15 samba-ldap Daily-0005
Differential Backup 10 11-Jun-11 00:15 shepot Daily-0005
Admin 12 11-Jun-11 03:35 BackupCatalog
Incremental Backup 10 11-Jun-11 13:10 shepot-allsa Daily-0005
====

Running Jobs:
Console connected at 06-Jun-11 16:23
JobId Level Name Status
======================================================================
27244 Full grkc.2011-06-10_12.30.00_16 is waiting on Storage EonStor1
27245 Full telex.2011-06-10_12.30.01_17 is waiting on Storage EonStor1
27246 Full rtelex.2011-06-10_12.30.01_18 is waiting on Storage EonStor1
27247 Increme shepot-allsa.2011-06-10_13.10.00_19 is waiting on
Storage EonStor1
====

Terminated Jobs:
JobId Level Files Bytes Status Finished Name
====================================================================
27221 Full 0 0 Cancel 10-Jun-11 05:07 pnts1-way4
27232 Full 0 0 Cancel 10-Jun-11 05:07 swist
27240 Cata 32,734 0 Diffs 10-Jun-11 05:17 verify-e-gw
27239 Cata 70,737 0 Diffs 10-Jun-11 05:28 verify-agate20
27241 Cata 45,253 0 Diffs 10-Jun-11 05:30 verify-lists
27243 Cata 34,532 0 Diffs 10-Jun-11 05:37 verify-s-gw
27242 Cata 71,609 0 Diffs 10-Jun-11 05:49 verify-omega
27238 Cata 42,067 0 Diffs 10-Jun-11 05:52 verify-agate10
27210 Incr 0 0 Cancel 10-Jun-11 07:44 netinv
27237 0 0 OK 10-Jun-11 08:06 BackupCatalog

====
*quit
-------------------------------------------------------------------------------

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>
  • [Bacula-users] storage daemon does not releases resources, Vadim Zotov <=