Bacula-users

Re: [Bacula-users] Catastrophic overflow block problems

2013-01-17 12:47:22
Subject: Re: [Bacula-users] Catastrophic overflow block problems
From: Josh Fisher <jfisher AT pvct DOT com>
To: bacula-users AT lists.sourceforge DOT net
Date: Thu, 17 Jan 2013 12:43:14 -0500
On 1/17/2013 11:06 AM, Ruth Ivimey-Cook wrote:
Hi,

I am sometimes getting these errors in my bacula backups:
Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" 
and it is more likely on the larger volume backups. It seemingly results from bacula trying to write an additional block to a disk drive that is already 100% full. How can I stop bacula from believing this is a valid thing to do?


You don't. The trick is to define a maximum volume size and number of volumes on the drive so that it is impossible to reach 100% of the physical drive's capacity. This will prevent the i/o error, and Bacula will instead hit end of volume and seek another volume. Of course, if no existing volumes can be recycled yet, then there simply isn't enough space on the drive. In that case, it is easy to add another drive to an existing autochanger, since vchanger allows for multiple simultaneous "magazine" drives.

Background:  I have bacula setup on my local network to backup a file server and a number of workstations. The file server is also the bacula director and is running Fedora 15 and "bacula-common-5.0.3-28.fc15.x86_64". Bacula is writing backups to an iSCSI disk group (not array) over ethernet; there are 6 disks of 1TB to 2TB size and these are managed using "vchanger" 0.8.6, with 6 magazines each with 24 virtual volumes. The file server has 3.5TB of files and other workstations add about another 1TB.

More-complete log:
17-Jan 14:49 helva-sd JobId 3417: Recycled volume "DiskPool1_0006_0017" on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0), all previous data lost.
17-Jan 14:49 helva-sd JobId 3417: New volume "DiskPool1_0006_0017" mounted on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0) at 17-Jan-2013 14:49.
17-Jan 14:49 helva-sd JobId 3417: End of Volume "DiskPool1_0006_0017" at 0:216 on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). Write of 64512 bytes got 3879.
17-Jan 14:49 helva-sd JobId 3417: End of medium on Volume "DiskPool1_0006_0017" Bytes=217 Blocks=0 at 17-Jan-2013 14:49.
17-Jan 14:49 helva-sd JobId 3417: 3307 Issuing autochanger "unload slot 89, drive 0" command.
17-Jan 14:49 helva-dir JobId 3417: Using Volume "DiskPool1_0006_0018" from 'Scratch' pool.
17-Jan 14:49 helva-sd JobId 3417: 3301 Issuing autochanger "loaded? drive 0" command.
17-Jan 14:49 helva-sd JobId 3417: 3302 Autochanger "loaded? drive 0", result: nothing loaded.
17-Jan 14:49 helva-sd JobId 3417: 3304 Issuing autochanger "load slot 90, drive 0" command.
17-Jan 14:49 helva-sd JobId 3417: 3305 Autochanger "load slot 90, drive 0", status is OK.
17-Jan 14:49 helva-sd JobId 3417: Recycled volume "DiskPool1_0006_0018" on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0), all previous data lost.
17-Jan 14:49 helva-sd JobId 3417: New volume "DiskPool1_0006_0018" mounted on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0) at 17-Jan-2013 14:49.
17-Jan 14:49 helva-sd JobId 3417: End of Volume "DiskPool1_0006_0018" at 0:216 on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). Write of 64512 bytes got 3879.
17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-fd JobId 3417: Error: bsock.c:393 Write error sending 65562 bytes to Storage daemon:helva.cam.ivimey.org:9103: ERR=Connection reset by peer
17-Jan 14:49 helva-fd JobId 3417: Fatal error: backup.c:1024 Network send error to SD. ERR=Connection reset by peer
17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Job write elapsed time = 14:30:55, Transfer rate = 12.06 M Bytes/second
17-Jan 14:49 helva-dir JobId 3417: Error: Bacula helva-dir 5.0.3 (04Aug10): 17-Jan-2013 14:49:32
  Build OS:               x86_64-redhat-linux-gnu redhat 
  JobId:                  3417
  Job:                    Helva_Home.2013-01-17_00.17.26_23
  Backup Level:           Full
  Client:                 "helva-fd" 5.0.3 (04Aug10) x86_64-redhat-linux-gnu,redhat,
  FileSet:                "Home" 2010-12-07 13:37:32
  Pool:                   "Normal-Full-18w" (From Job FullPool override)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "DiskStorage" (From command line)
  Scheduled time:         17-Jan-2013 00:17:26
  Start time:             17-Jan-2013 00:17:28
  End time:               17-Jan-2013 14:49:32
  Elapsed time:           14 hours 32 mins 4 secs
  Priority:               12
  FD Files Written:       162,621
  SD Files Written:       162,621
  FD Bytes Written:       630,336,973,872 (630.3 GB)
  SD Bytes Written:       630,355,165,137 (630.3 GB)
  Rate:                   12046.8 KB/s
  Software Compression:   7.7 %
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         DiskPool1_0006_0003|DiskPool1_0001_0020|DiskPool1_0004_0013|DiskPool1_0004_0014|DiskPool1_0004_0023|DiskPool1_0004_0024|DiskPool1_0006_0017|DiskPool1_0006_0018
  Volume Session Id:      417
  Volume Session Time:    1351720132
  Last Volume Bytes:      217 (217 B)
  Non-fatal FD errors:    1
  SD Errors:              1
  FD termination status:  Error
  SD termination status:  Error
  Termination:            *** Backup Error ***


-- 
Software Manager & Engineer
Blog: http://www.ivimey.org/blog
LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/ 


------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, _javascript_ and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712


_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>