Bacula-users

Re: [Bacula-users] Antwort: Re: Antwort: Re: Bacula didn't start anymore

2009-05-27 15:31:36
Subject: Re: [Bacula-users] Antwort: Re: Antwort: Re: Bacula didn't start anymore
From: Arno Lehmann <al AT its-lehmann DOT de>
To: bacula-users AT lists.sourceforge DOT net
Date: Wed, 27 May 2009 21:25:00 +0200
Hi,

27.05.2009 16:46, ElmarRath AT systemcompetence DOT de wrote:
> 
> Hi Reiner,
> 
> I didn't change the bacula-dir.conf. Bacula is running since the last 
> two month with out a problem. All of sudden last monday my backup 
> failed. I tried then to access bacula and got that failure, after I 
> restarted bacula.
> 
> I first checked, if the configuration was corrupted, after that I 
> checked the working folders for missing files etc. The databasefile 
> bacula.db is in the working dir /root/bacula/working/
> and has a size of nearly 2 GB.

That sounds like SQLite has a problem with databases of that size, or 
the file system you use has a 2GiB file size limit. I don't know which 
file system you use, and I don't know anything about SQLite, so I 
can't tell you which of that it might be...

> My problem is that I also have no experience with sqlite, but I wasn't 
> aloud to use an other one like mysql.

Tell whoever not allowed that that using Bacula with SQLite in 
production is, for good reasons, severely deprecated. You might just 
have found one of those reasons.

Hint: You can use a database running on another, dedicated machine... 
a simple server as a database host might be cheaper than working 
around your problem.

Cheers,

Arno

> Mit freundlichen Gruessen
> 
> 
> Wolf-Elmar Rath
> 
> SYSTEM COMPETENCE GmbH
> Ohlendorfer Strasse 40
> 21220 Seevetal
> 
> Telefon: 04185-77-0
> Telefax: 04185-77-440
> Email: elmar.rath AT systemcompetence DOT de
> 
> Sitz der Gesellschaft: Seevetal
> Geschäftsführer: Sven Hannemann, Martin Pranzner, Rigo Rothenbröker
> Handelsregister: HRB 111223, Amtsgericht Lüneburg
> 
> 
> *SystemManagement <rdsys AT fs.wettzell DOT de>*
> 
> 27.05.2009 15:26
> Bitte antworten an
> dassing AT fs.wettzell DOT de
> 
> 
>       
> An
>       ElmarRath AT systemcompetence DOT de
> Kopie
>       dassing AT fs.wettzell DOT de, bacula-users AT lists.sourceforge DOT net
> Thema
>       Re: [Bacula-users] Antwort: Re:  Bacula didn't start anymore
> 
> 
>       
> 
> 
> 
> 
> 
> Hi,
> starting from
> 
> bacula-dir: dird.c:883 sqlite.c:210 Unable to open 
> Database=/root/bacula/working/bacula.db. ERR=out of memory
> 26-May 16:54 bacula-dir ERROR TERMINATION
> Please correct configuration file: bacula-dir.conf
> 
> there are different subjects to consider:
> - bacula-dir.conf
> - SQLite
> - bacula-dir
> 
> Did you change bacula-dir.conf before the problem arises?
> As I remember there is a tool to check the syntax of this file.
> (Please, ask the docs)
> 
> 
> Does file  "/root/bacula/working/bacula.db" exists?
> Did you made sure that SQLite causes the problem?
> I have no experiences with SQLite but you shold use or write a test
> application to
> access the database without using bacula
> Is there a tool to check the conistency of the SQLite database?
> 
> That are my ideas for further steps.
> 
> Regards,
> Reiner
> 
> 
> 
> ElmarRath AT systemcompetence DOT de schrieb:
>  > Hi,
>  >
>  > I checked the raid and the disks twice if there is any problem. Gpart
>  > didn't found anything.
>  > My major problem is that bacula-dir isn't starting, because I 've got he
>  > out of memory message,
>  > that I posted on the end of my mail.
>  >
>  > Mit freundlichen Gruessen
>  >
>  >
>  > Wolf-Elmar Rath
>  >
>  > SYSTEM COMPETENCE GmbH
>  > Ohlendorfer Strasse 40
>  > 21220 Seevetal
>  >
>  > Telefon: 04185-77-0
>  > Telefax: 04185-77-440
>  > Email: elmar.rath AT systemcompetence DOT de
>  >
>  > Sitz der Gesellschaft: Seevetal
>  > Geschäftsführer: Sven Hannemann, Martin Pranzner, Rigo Rothenbröker
>  > Handelsregister: HRB 111223, Amtsgericht Lüneburg
>  >
>  >
>  >
>  > SystemManagement <rdsys AT fs.wettzell DOT de>
>  > 26.05.2009 17:41
>  > Bitte antworten an
>  > dassing AT fs.wettzell DOT de
>  >
>  >
>  > An
>  > ElmarRath AT systemcompetence DOT de
>  > Kopie
>  > bacula-users AT lists.sourceforge DOT net
>  > Thema
>  > Re: [Bacula-users] Bacula didn't start anymore
>  >
>  >
>  >
>  >
>  >
>  >
>  > Hallo,
>  >
>  > the messages point to a disc problem.
>  >
>  > I see: ERR=disk I/O error
>  >
>  > Check your discs!
>  >
>  > Reiner
>  >
>  >
>  > ElmarRath AT systemcompetence DOT de schrieb:
>  >  
>  >> Haloo,
>  >>
>  >> I hope someone could help me wirth my problem. Monday morning faild my
>  >> backup with the following log.
>  >>
>  >>
>  >>
>  >> 25-May 02:01 bocsles-dir JobId 124: Start Backup JobId 124,
>  >> Job=Nightly-Save-bocsles.2009-05-25_02.01.00.02
>  >> 25-May 02:01 bocsles-dir JobId 124: Using Device "LTO4"
>  >> 25-May 02:01 bocsles-sd JobId 124: Error: 25-May 02:01 bocsles-sd JobId
>  >> 124: Volume "LIN-TAG-22" previously written, moving to end of data.
>  >> 25-May 02:01 bocsles-sd JobId 124: Ready to append to end of Volume
>  >> "LIN-TAG-22" at file=136.
>  >> bocsles-fd JobId 124:      /dev is a different filesystem. Will not
>  >> descend from / into /dev
>  >> bocsles-fd JobId 124:      /opt is a different filesystem. Will not
>  >> descend from / into /opt
>  >> bocsles-fd JobId 124:      /var/lib/ntp/proc is a different filesystem.
>  >> Will not descend from / into /var/lib/ntp/proc
>  >> bocsles-fd JobId 124:      /home is a different filesystem. Will not
>  >> descend from / into /home
>  >> bocsles-fd JobId 124:      /var/lib/ntp/proc is a different filesystem.
>  >> Will not descend from /var into /var/lib/ntp/proc
>  >> 25-May 02:54 bocsles-sd JobId 124: Job write elapsed time = 00:52:53,
>  >> Transfer rate = 22.19 M bytes/second
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: Can't fill File table
>  >> Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId,
>  >> LStat, MD5)SELECT batch.FileIndex, batch.JobId, Path.PathId,
>  >> Filename.FilenameId,batch.LStat, batch.MD5 FROM batch JOIN Path ON
>  >> (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name):
>  >> ERR=disk I/O error
>  >>
>  >> 25-May 03:15 bocsles-dir JobId 124: Error: sql_update.c:178
>  >> sql_update.c:178 update UPDATE Job SET 
> JobStatus='f',EndTime='2009-05-25
>  >>    
>  >
>  >  
>  > 
> 03:15:26',ClientId=1,JobBytes=70287992114,JobFiles=805529,JobErrors=0,VolSessionId=1,VolSessionTime=1243209010,PoolId=3,FileSetId=3,JobTDate=1243214126,RealEndTime='2009-05-25
>  
> 
>  >
>  >  
>  >> 03:15:26',PriorJobId=0 WHERE JobId=124 failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_update.c:178 UPDATE Job SET
>  >> JobStatus='f',EndTime='2009-05-25
>  >>
>  >>    
>  > 
> 03:15:26',ClientId=1,JobBytes=70287992114,JobFiles=805529,JobErrors=0,VolSessionId=1,VolSessionTime=1243209010,PoolId=3,FileSetId=3,JobTDate=1243214126,RealEndTime='2009-05-25
>  
> 
>  >
>  >  
>  >> 03:15:26',PriorJobId=0 WHERE JobId=124
>  >> 25-May 03:15 bocsles-dir JobId 124: Warning: Error updating job record.
>  >> sql_update.c:178 update UPDATE Job SET 
> JobStatus='f',EndTime='2009-05-25
>  >>    
>  >
>  >  
>  > 
> 03:15:26',ClientId=1,JobBytes=70287992114,JobFiles=805529,JobErrors=0,VolSessionId=1,VolSessionTime=1243209010,PoolId=3,FileSetId=3,JobTDate=1243214126,RealEndTime='2009-05-25
>  
> 
>  >
>  >  
>  >> 03:15:26',PriorJobId=0 WHERE JobId=124 failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_get.c:289
>  >> sql_get.c:289 query SELECT
>  >>
>  >>    
>  > 
> VolSessionId,VolSessionTime,PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus,Type,Level,ClientId,Name,PriorJobId,RealEndTime,JobId,FileSetId,SchedTime,RealEndTime
>  
> 
>  >
>  >  
>  >> FROM Job WHERE JobId=124 failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_get.c:289 SELECT
>  >>
>  >>    
>  > 
> VolSessionId,VolSessionTime,PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus,Type,Level,ClientId,Name,PriorJobId,RealEndTime,JobId,FileSetId,SchedTime,RealEndTime
>  
> 
>  >
>  >  
>  >> FROM Job WHERE JobId=124
>  >> 25-May 03:15 bocsles-dir JobId 124: Warning: Error getting Job record
>  >>    
>  > for
>  >  
>  >> Job report: ERR=sql_get.c:289 query SELECT
>  >>
>  >>    
>  > 
> VolSessionId,VolSessionTime,PoolId,StartTime,EndTime,JobFiles,JobBytes,JobTDate,Job,JobStatus,Type,Level,ClientId,Name,PriorJobId,RealEndTime,JobId,FileSetId,SchedTime,RealEndTime
>  
> 
>  >
>  >  
>  >> FROM Job WHERE JobId=124 failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_get.c:672
>  >> sql_get.c:672 query SELECT
>  >> ClientId,Name,Uname,AutoPrune,FileRetention,JobRetention FROM Client
>  >>    
>  > WHERE
>  >  
>  >> Client.Name='bocsles-fd' failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_get.c:672 SELECT
>  >> ClientId,Name,Uname,AutoPrune,FileRetention,JobRetention FROM Client
>  >>    
>  > WHERE
>  >  
>  >> Client.Name='bocsles-fd'
>  >> 25-May 03:15 bocsles-dir JobId 124: Warning: Error getting Client 
> record
>  >>    
>  >
>  >  
>  >> for Job report: ERR=Client record not found in Catalog.
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_get.c:942
>  >> sql_get.c:942 query SELECT
>  >>
>  >>    
>  > 
> MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,VolParts,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime
>  
> 
>  >
>  >  
>  >> FROM Media WHERE VolumeName='LIN-TAG-22' failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_get.c:942 SELECT
>  >>
>  >>    
>  > 
> MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,VolParts,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime
>  
> 
>  >
>  >  
>  >> FROM Media WHERE VolumeName='LIN-TAG-22'
>  >> 25-May 03:15 bocsles-dir JobId 124: Warning: Error getting Media record
>  >> for Volume "LIN-TAG-22": ERR=Media record for Vol=LIN-TAG-22 not found
>  >>    
>  > in
>  >  
>  >> Catalog.
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_get.c:359
>  >> sql_get.c:359 query SELECT VolumeName,MAX(VolIndex) FROM JobMedia,Media
>  >> WHERE JobMedia.JobId=124 AND JobMedia.MediaId=Media.MediaId GROUP BY
>  >> VolumeName ORDER BY 2 ASC failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_get.c:359 SELECT
>  >> VolumeName,MAX(VolIndex) FROM JobMedia,Media WHERE JobMedia.JobId=124
>  >>    
>  > AND
>  >  
>  >> JobMedia.MediaId=Media.MediaId GROUP BY VolumeName ORDER BY 2 ASC
>  >> 25-May 03:15 bocsles-dir JobId 124: Error: Bacula bocsles-dir 2.4.4
>  >> (28Dec08): 25-May-2009 03:15:26
>  >>   Build OS:               powerpc64-unknown-linux-gnu suse 10
>  >>   JobId:                  124
>  >>   Job:                    Nightly-Save-bocsles.2009-05-25_02.01.00.02
>  >>   Backup Level:           Full
>  >>   Client:                 "bocsles-fd"
>  >>   FileSet:                "BOC_Full_Set" 2009-03-17 02:01:00
>  >>   Pool:                   "TagesPool" (From Job resource)
>  >>   Storage:                "LTO4" (From Job resource)
>  >>   Scheduled time:         25-May-2009 02:01:00
>  >>   Start time:             25-May-2009 02:01:03
>  >>   End time:               25-May-2009 03:15:26
>  >>   Elapsed time:           1 hour 14 mins 23 secs
>  >>   Priority:               10
>  >>   FD Files Written:       805,529
>  >>   SD Files Written:       805,529
>  >>   FD Bytes Written:       70,287,992,114 (70.28 GB)
>  >>   SD Bytes Written:       70,412,688,427 (70.41 GB)
>  >>   Rate:                   15749.0 KB/s
>  >>   Software Compression:   None
>  >>   VSS:                    no
>  >>   Storage Encryption:     no
>  >>   Volume name(s):
>  >>   Volume Session Id:      1
>  >>   Volume Session Time:    1243209010
>  >>   Last Volume Bytes:      0 (0 B)
>  >>   Non-fatal FD errors:    1
>  >>   SD Errors:              0
>  >>   FD termination status:  OK
>  >>   SD termination status:  OK
>  >>   Termination:            *** Backup Error ***
>  >>
>  >> 25-May 03:15 bocsles-dir JobId 124: Begin pruning Jobs.
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_create.c:488
>  >> sql_create.c:488 query SELECT ClientId,Uname FROM Client WHERE
>  >> Name='bocsles-fd' failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_create.c:488 SELECT
>  >>    
>  > ClientId,Uname
>  >  
>  >> FROM Client WHERE Name='bocsles-fd'
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_create.c:523
>  >> sql_create.c:523 insert INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0) failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_create.c:523 INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0)
>  >> 25-May 03:15 bocsles-dir JobId 124: Error: sql_create.c:525 Create DB
>  >> Client record INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0) failed. ERR=disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: Begin pruning Files.
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_create.c:488
>  >> sql_create.c:488 query SELECT ClientId,Uname FROM Client WHERE
>  >> Name='bocsles-fd' failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_create.c:488 SELECT
>  >>    
>  > ClientId,Uname
>  >  
>  >> FROM Client WHERE Name='bocsles-fd'
>  >> 25-May 03:15 bocsles-dir JobId 124: Fatal error: sql_create.c:523
>  >> sql_create.c:523 insert INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0) failed:
>  >> disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: sql_create.c:523 INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0)
>  >> 25-May 03:15 bocsles-dir JobId 124: Error: sql_create.c:525 Create DB
>  >> Client record INSERT INTO Client
>  >> (Name,Uname,AutoPrune,FileRetention,JobRetention) VALUES
>  >> ('bocsles-fd','',0,0,0) failed. ERR=disk I/O error
>  >> 25-May 03:15 bocsles-dir JobId 124: End auto prune.
>  >>
>  >>
>  >>
>  >>
>  >> After that I tried to restart bacula and get now the following problem:
>  >>
>  >>
>  >> bocsles:~ # /root/bacula/bin/bacula-dir -t
>  >> bacula-dir: dird.c:878 Could not open Catalog "MyCatalog", database
>  >> "bacula".
>  >> bacula-dir: dird.c:883 sqlite.c:210 Unable to open
>  >> Database=/root/bacula/working/bacula.db. ERR=out of memory
>  >> 26-May 16:54 bacula-dir ERROR TERMINATION
>  >> Please correct configuration file: bacula-dir.conf
>  >> bocsles:~ #
>  >>
>  >> I don't know why it thinks, that there is no memory left. I checked my
>  >> harddrives and there was more then 14GB space left. Also I didn't have
>  >>    
>  > any
>  >  
>  >> problems with my swap or RAM, there should be more then enough memory
>  >>    
>  > left
>  >  
>  >> for the backup job to run.
>  >>
>  >> BTW: The bacula.db has a size of 2 GB, could that be a problem for
>  >>    
>  > SQLite?
>  >  
>  >> I hope you could help me.
>  >>
>  >> Best regards
>  >>
>  >>
>  >> Mit freundlichen Gruessen
>  >>
>  >>
>  >> Wolf-Elmar Rath
>  >>
>  >> SYSTEM COMPETENCE GmbH
>  >> Ohlendorfer Strasse 40
>  >> 21220 Seevetal
>  >>
>  >> Telefon: 04185-77-0
>  >> Telefax: 04185-77-440
>  >> Email: elmar.rath AT systemcompetence DOT de
>  >>
>  >> Sitz der Gesellschaft: Seevetal
>  >> Geschäftsführer: Sven Hannemann, Martin Pranzner, Rigo Rothenbröker
>  >> Handelsregister: HRB 111223, Amtsgericht Lüneburg
>  >>
>  >> ------------------------------------------------------------------------
>  >>
>  >>
>  >>    
>  > 
> ------------------------------------------------------------------------------
>  >  
>  >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  >> is a gathering of tech-side developers & brand creativity 
> professionals.
>  >>    
>  > Meet
>  >  
>  >> the minds behind Google Creative Lab, Visual Complexity, Processing, &
>  >> iPhoneDevCamp asthey present alongside digital heavyweights like
>  >>    
>  > Barbarian
>  >  
>  >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>  >> ------------------------------------------------------------------------
>  >>
>  >> _______________________________________________
>  >> Bacula-users mailing list
>  >> Bacula-users AT lists.sourceforge DOT net
>  >> https://lists.sourceforge.net/lists/listinfo/bacula-users
>  >>
>  >>    
>  >
>  >
>  >
>  >  
>  > ------------------------------------------------------------------------
>  >
>  > 
> ------------------------------------------------------------------------------
>  > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  > is a gathering of tech-side developers & brand creativity 
> professionals. Meet
>  > the minds behind Google Creative Lab, Visual Complexity, Processing, &
>  > iPhoneDevCamp as they present alongside digital heavyweights like 
> Barbarian
>  > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
>  > ------------------------------------------------------------------------
>  >
>  > _______________________________________________
>  > Bacula-users mailing list
>  > Bacula-users AT lists.sourceforge DOT net
>  > https://lists.sourceforge.net/lists/listinfo/bacula-users
>  >  
> 
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT 
> is a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, & 
> iPhoneDevCamp as they present alongside digital heavyweights like Barbarian 
> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users

-- 
Arno Lehmann
IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück
www.its-lehmann.de

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT 
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian 
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users