Bacula-users

[Bacula-users] Restoring from possibly (probably!) corrupt disk volume

2010-02-24 15:47:08
Subject: [Bacula-users] Restoring from possibly (probably!) corrupt disk volume
From: "Mike Holden" <bacula AT mikeholden DOT org>
To: bacula-users AT lists.sourceforge DOT net
Date: Wed, 24 Feb 2010 20:45:01 -0000
Been struggling with a difficult restore for a few days now, and need some help
please.

The short story is that I am trying to recover files from a disk archive which 
was
subsequently deleted from disk, and then recovered using an undelete process on 
a
reiserfs disk. Most of the files appear to have recovered ok, but I'm pretty 
sure
that the undelete is not complete, and that some files are either missing 
entirely,
or have random or zeroed blocks in them instead of real data.

Nightmare scenario I know, but anything that can be recovered is a bonus here!

I'm now at the stage of using bls and bextract to recover what I can from files,
but seem to be hitting a situation where bls lists more files than bextract is 
able
to recover. bls gets to pretty much the end of the file, then exits with the
following few lines:

-- Begin bls log
bls: butil.c:282 Using device: "/backup/jewel/bacula" for reading.
24-Feb 20:15 bls JobId 0: Ready to read from volume "Monthly-0544" on device
"FileStorage" (/backup/jewel/bacula).
bls JobId 0: -rwxrw----   1 501      501          69930 2009-10-08 00:54:21 
/home/mike/Documents/tumnus/DVD_Collection.txt

... [ big snip ]

bls JobId 0: -rw-r--r--   1 506      506          16992 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-D.jpg
bls JobId 0: -rw-r--r--   1 506      506          16607 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-E.jpg
bls JobId 0: -rw-r--r--   1 506      506          14651 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-F.jpg
bls JobId 0: -rw-r--r--   1 506      506          15159 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-G.jpg
bls JobId 0: -rw-r--r--   1 506      506          16259 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-H.jpg
bls JobId 0: -rw-r--r--   1 506      506          17196 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-I.jpg
bls JobId 0: -rw-r--r--   1 506      506          20404 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-J.jpg
bls JobId 0: -rw-r--r--   1 506      506          34680 2009-10-29 20:31:07 
/home/byang/albums/Cards/Multipack/thumb_C-MP-0127-K.jpg
24-Feb 20:16 bls JobId 0: Error: block.c:275 Volume data error at 0:2147475478!
Wanted ID: "BB02", got "". Buffer discarded.
50614 files found.
--- End bls log

The volume file is as follows:

[root@jewel mike]# ls -l /backup/jewel/bacula/Monthly-0544
-rw-r-----. 1 root root 2147479552 2010-02-07 04:33 
/backup/jewel/bacula/Monthly-0544

The file in question is a single volume from a multi-volume job, split by file 
size
of approx 2Gb each.

When running bextract on the same file, with -vvv -d 3000 options, the progress
gets as far as the following output, but then just sits there, consuming as much
cpu as it can get hold of, but doesn't seem to get any more progress, even when
left for several hours. It always sticks at the same file, which it creates but
leaves as a zero byte file. The last bit of the log looks like this:

--- Start bextract log

... [ big snip ]

bextract: create_file.c:90-0 type=5 newmode=41fd
file=/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.executable.PackageRegistryBackend/
bextract: create_file.c:118-0 Replace=a 97
bextract: create_file.c:362-0 Make dir mode=40775
dir=/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.executable.PackageRegistryBackend/
bextract: guid_to_name.c:138-0 uid=501 item=1f7e1c8
bextract: attr.c:281-0 drwxrwxr-x   2 501      501             48 2008-12-13
11:32:21  *none*
bextract: message.c:1102-0 Enter Jmsg type=13
bextract: message.c:612-0 Enter dispatch_msg type=13 msg=bextract JobId 0:
drwxrwxr-x   2 501      501             48 2008-12-13 11:32:21  *none*
bextract: message.c:799-0 STDOUT for following msg: bextract JobId 0: 
drwxrwxr-x  
2 501      501             48 2008-12-13 11:32:21  *none*
bextract JobId 0: drwxrwxr-x   2 501      501             48 2008-12-13 
11:32:21 
*none*
bextract: record.c:465-0 Block=219743 Ver=2 size=64512
bextract: record.c:473-0 Enter read_record_block: remlen=56984 data_len=195 
rem=0
blkver=2
bextract: record.c:533-0 rd_rec_blk() got FI=303570 SessId=62 Strm=UATTR len=222
remlen=56972 data_len=0
bextract: record.c:592-0 Rtn full rd_rec_blk FI=303570 SessId=62 Strm=UATTR 
len=222
bextract: read_record.c:207-0 read-OK. state= blk=219743 rem=0 
file:block=0:1291207847
bextract: read_record.c:217-0 recno=42 state= blk=219743 SI=62 ST=1264433856 
FI=303570
bextract: read_record.c:286-0 OK callback. recno=42 state= blk=219743 SI=62
ST=1264433856 FI=303570
bextract: attr.c:77-0 Attr: 303570 3
/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: attr.c:83-0 Got Attr: FilInx=303570 type=3
bextract: attr.c:116-0 unpack_attr FI=303570 Type=3
fname=/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
attr=P0K Pe+ IG0 B H1 H1 A DAA BAA Q BLQB0q BJQ51F BJQ51F A A E lname= attrEx= 
ds=0
bextract: match.c:333-0 pat=/
file=/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: match.c:358-0 exc is NULL
bextract: create_file.c:90-0 type=3 newmode=81b4
file=/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: create_file.c:118-0 Replace=a 97
bextract: create_file.c:160-0 unlink
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: create_file.c:189-0 Make path
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend
bextract: create_file.c:208-0
Create=/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: bfile.c:900-0 open file
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: bfile.c:924-0 Open file 4
bextract: guid_to_name.c:138-0 uid=501 item=1f7e1c8
bextract: attr.c:281-0 -rw-rw-r--   1 501      501          12288 2008-12-13
11:32:21 
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: message.c:1102-0 Enter Jmsg type=13
bextract: message.c:612-0 Enter dispatch_msg type=13 msg=bextract JobId 0:
-rw-rw-r--   1 501      501          12288 2008-12-13 11:32:21 
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: message.c:799-0 STDOUT for following msg: bextract JobId 0: 
-rw-rw-r--  
1 501      501          12288 2008-12-13 11:32:21 
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract JobId 0: -rw-rw-r--   1 501      501          12288 2008-12-13 
11:32:21 
/backup/recovery/mike/home/mike/.openoffice.org2.0/user/uno_packages/cache/registry/com.sun.star.comp.deployment.configuration.PackageRegistryBackend/registered_packages.db
bextract: record.c:465-0 Block=219743 Ver=2 size=64512
bextract: record.c:473-0 Enter read_record_block: remlen=56750 data_len=222 
rem=0
blkver=2
bextract: record.c:533-0 rd_rec_blk() got FI=303570 SessId=62 Strm=GZIP len=113
remlen=56738 data_len=0
bextract: record.c:592-0 Rtn full rd_rec_blk FI=303570 SessId=62 Strm=GZIP 
len=113
bextract: read_record.c:207-0 read-OK. state= blk=219743 rem=0 
file:block=0:1291207847
bextract: read_record.c:217-0 recno=43 state= blk=219743 SI=62 ST=1264433856 
FI=303570
bextract: read_record.c:286-0 OK callback. recno=43 state= blk=219743 SI=62
ST=1264433856 FI=303570
                                                                                
     ---
End
bextract
log

My guess is that it is probably disappearing on some kind of null or garbage
pointer chase.

Can someone who understands the internals of what is going on with these 
programs
offer any information on what I can do to extract the rest of the volume?

The file is 2gb in size, but I can make it available for download if that would 
help?

I appreciate that this is probably a lost cause, but the fact that bls lists a 
lot
of files beyond where bextract gets stuck gives me some hope that maybe with a 
few
appropriate bytes poked into the appropriate part of the volume I can recover 
more
than I am currently getting. Of course, Sod's Law dictates that the files I am 
most
interested in are closer to the end of the volume than the beginning!

I should also mention that I am using bacula version 3.0.3 on fedora 12, 
although
the backup was originally created under bacula 2.4.4 on fedora 10.

Any help would be much appreciated!
-- 
Mike Holden





------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users