Hi all,
my BPC 4 Installation (on Debian 8, Pool on ZFS) has been running fine so far.
There are 91 hosts that have been backed up, for a total of:
357 full backups of total size 21698.30GiB (prior to pooling and compression),
1658 incr backups of total size 1605.12GiB (prior to pooling and compression).
But when I did an archive, I noticed quite some errors:
Excerpt from archive.log:
…
Writing tar archive for host n128, backup #39, split to output files /media/truecrypt1/n128.39.tar.gz.*
Unable to open file /data/BackupPC/pool/00/00/00000000000000000000000000000000
Unable to open file /data/BackupPC/pool/34/42/354244353641350000fdab86b305a483
Unable to open file /data/BackupPC/pool/0c/de/0cdf558543a65bcad328000000000000
Unable to open file /data/BackupPC/pool/00/00/00000000000000000000000000000000
Unable to open file /data/BackupPC/pool/46/30/46313339303036314432313432463700
Unable to open file /data/BackupPC/pool/00/00/00000000000000000000000000000000
Done: 166796 files, 54304422997 bytes, 4458 dirs, 0 specials, 6 errors
…
Also, when doing an incremental Backup of that host:
Running BackupPC_refCountUpdate -h n128 on n128
xferPids 26755
BackupPC_refCountUpdate: host n128 got 0 errors
BackupPC_refCountPrint: total errors: 0
xferPids
Finished BackupPC_refCountUpdate (running time: 1 sec)
Running BackupPC_refCountUpdate -h n128 -f -c on n128
xferPids 26759
Host n128 digest.0 00000000000000000000000000000000 count is 18, but should be 19
Host n128 digest.0 a399957ce1fcba591cc58367faabad93 count is 6, but should be 5
BackupPC_refCountUpdate: host n128 got 2 errors
BackupPC_refCountPrint: total errors: 2
xferPids
Finished BackupPC_refCountUpdate (running time: 11 sec)
I then had a look at the general BPC Logfile, and sure enough, the nightly jobs are reporting these files as missing too:
Contents of file /var/log/BackupPC/LOG, modified 2016-01-22 14:00:34
2016-01-21 21:00:00 Running 2 BackupPC_nightly jobs from 0..15 (out of 0..15)
2016-01-21 21:00:00 Running BackupPC_nightly -m -P 9 0 127 (pid=10496)
2016-01-21 21:00:00 Running BackupPC_nightly -P 9 128 255 (pid=10497)
2016-01-21 21:00:00 Next wakeup is 2016-01-21 21:30:00
2016-01-21 21:00:01 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 9 -r 128-255
2016-01-21 21:00:01 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 9 -r 0-127
2016-01-21 21:02:37 admin : Missing pool file 00000000000000000000000000000000 count 35
2016-01-21 21:02:37 admin : Missing pool file 001000001045283f904033e72800848e count 1
2016-01-21 21:02:37 admin : Missing pool file 01a995a601000010bda3f79705a48302 count 1
2016-01-21 21:02:37 admin : Missing pool file 0000c62cf2f281030000100000000000 count 2
2016-01-21 21:02:37 admin : Missing pool file 00f466f81d0000101000000000000000 count 1
2016-01-21 21:02:37 admin : Missing pool file 0010020000c702a9aa18000010190000 count 1
2016-01-21 21:02:37 admin : Missing pool file 0000100010330000f7b0b1aa05a48302 count 1
2016-01-21 21:02:37 admin : Missing pool file 0000d596bc9505a48302000080d007de count 1
2016-01-21 21:02:37 admin : Missing pool file 0000b5ce999705a48302000080fc0583 count 1
2016-01-21 21:02:38 admin1 : Missing pool file 80f8a701000010000010af95a7010000 count 1
2016-01-21 21:04:52 admin1 : Missing pool file 8800053438303139000085c5b4a505a4 count 1
…
2016-01-21 21:21:18 admin1 : BackupPC_refCountPrint: total errors: 25
2016-01-21 21:21:18 admin1 : xferPids
2016-01-21 21:21:18 Finished admin1 (BackupPC_nightly -P 9 128 255)
2016-01-21 21:21:43 admin : BackupPC_refCountPrint: total errors: 39
2016-01-21 21:21:43 admin : xferPids
…
Is this something to be worried about? And what could cause it?
It’s running on server-grade hardware, ECC RAM etc., the ZFS Pool looks completely healthy, so I’d tend to rule out flaky hardware.
Is there a way to find out to which files these hashes belong to? Maybe there is something special about them.
Regards,
Christian
--
Christian Merz
IT Administration
hte GmbH
the high throughput experimentation company
Kurpfalzring 104, 69123 Heidelberg, Germany
T: +49 6221 7497 181
F: +49 6221 7497 137
www.hte-company.com
Registergericht: Amtsgericht Mannheim HRB 717943
Sitz der Gesellschaft: Heidelberg
Geschäftsführer: Dr. Wolfram Stichert