Hi everyone,
I've got a corrupted catalog on a bad disk and could really use some advice on how to proceed.
I'm very new to bacula (and pretty new to sysadmin after many years programming), and have inherited a bacula system that's been untended for a year and a half or more.
It's running Bacula 3.0.1, which I know is very old.
There's about 30TB of data backed up across about 5 jobs.
**** the issue
I'm seeing the error in bacula logs:
ERR=Duplicate entry '1828889455' for key 1
running dbcheck yields:
"Query failed: SELECT File.FileId,Job.JobId FROM File LEFT OUTER JOIN Job ON (File.JobId=Job.JobId) WHERE Job.JobId IS NULL LIMIT 300000: ERR=Table './bacula/File' is marked as crashed and should be repaired".
But running 'myisamchk -r -q File.MYI' yields:
"myisamchk: error: - check record delete-chain
Can't read delete-link at filepos: 13937713076
myisamchk: error: Quick-recover aborted; Run recovery without switch 'q'
Updating MyISAM file: File.MYI
MyISAM-table 'File.MYI' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag"
Before running a more full recovery, I tried copying the database, but got this error:
"cp: reading `File.MYD': Input/output error"
This seems to be a disk error, but I ran 'xfs_repair' anyway but it didn't fix anything, not surprising.
Catalog backups:
There's a script running on the machine that backs up the catalog nightly to another disk using /usr/lib/bacula/make_catalog_backup and I have several days worth of these. This disc is backed up by bacula too. HOWEVER, looking at catalog.txt log (in the same dir as these backups, it looks like a log from make_catalog_backup ), it shows this error's been happening since Nov 2012, so my recent catalog dumps aren't going to be good.
**** what's next?
After replacing the bad disk:
Can i reconstruct the catalog just from the backup data itself? It's on tapes, fwiw. How do I go about doing this?
If I can get one of the old catalog dumps off one of the backup tapes, can I recontruct from that even though there have been backups since then?
Do I resolve this with Bacula 3.0.1 before upgrading to the latest? Seems to make sense.
Any suggestions or insights would be greatly appreciated!
Thanks,
Michael