BackupPC-users

Re: [BackupPC-users] server maintenance: reconstruct missing poolCnt; find/delete references to missing pool files

2017-03-07 12:25:52
Subject: Re: [BackupPC-users] server maintenance: reconstruct missing poolCnt; find/delete references to missing pool files
From: Craig Barratt <cbarratt AT users.sourceforge DOT net>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Tue, 7 Mar 2017 09:24:32 -0800
> BackupPC_refCountUpdate: doing fsck on <host> #1188 since there are no poolCnt files
> BackupPC_refCountUpdate: doing fsck on <host> #1190 since there are no poolCnt files
> ...
> BackupPC_refCountUpdate: host <host> got 0 errors (took 5 secs)
The backups in question seem to be fully intact; some are full backups, some are incremental. It's just on a minority of backups (appx. 15 out of 350 backups), and fortunately on small ones where fsck does not take ages, so it does not bother me too much. Nevertheless, can the missing poolCnt data be recomputed? fsck seems to do the counting from scratch; can this be stored?

This is perfectly ok.  BackupPC 4.0.0alpha3 and prior 4.x versions didn't store reference counts per backup.  Only the reference counts for the entire host were maintained (in addition to the pool totals for all hosts).  In 4.0.0, I changed that so reference counts were also stored per-backup (which makes it easier to delete backups and to recompute the per-host ref counts).  So BackupPC_refCountUpdate is simply adding reference counts to backups done by BackupPC 4.0.0alpha3.  It's a one-time thing.

There might be an issue that an incremental done by BackupPC 4.0.0alpha3 with no changes will have an empty backup tree, and BackupPC_refCntUpdate will continually report that there are no poolCnt files for that backup.  That's benign.  In 4.0.0, BackupPC_dump flags that by creating a file "HOST/NNN/refCnt/noPoolCntOk, which makes BackupPC_refCntUpdate quietly ignore that backup.  Perhaps I should have BackupPC_refCntUpdate notice that legacy case and create the noPoolCntOk file...

> BackupPC_refCountUpdate: missing pool file 00000000000000000000000000000000 count 30
> BackupPC_refCountUpdate: missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da count 1
> ...
> BackupPC_refCountUpdate: missing pool file ea1bd7ab2e0000000000000000000000 count 1

This is a bug in rsync-bpc (and BackupPC::XS) that was fixed a couple of weeks ago.  It happened about 2% of the time the attrib file for a large directory was written (attrib file sizes >256k, approx 5k files depending on file name lengths).  If the md5 digest of the last file written to the 256k staging buffer exactly lined up with the end of the buffer, the digest for that file wasn't written correctly (yes, I had "<" instead of "<="... doh!).

Future backups with 4.0.0 (assuming the same file exists on the client) will be updated with the correct digest, but the old backups will still have the wrong one.  The errors will go away when the corresponding backups eventually expire.

Craig

On Tue, Mar 7, 2017 at 6:56 AM, Alexander Kobel <a-kobel AT a-kobel DOT de> wrote:
Dear all,

I have a rather small, private, non-$$$M-mission-critical instance of a BackupPC server running for years (that went from several 3.something through 4.0.0alpha3 to, recently, 4.0.0).
After the 4.0.0 migration, I decided to run again some semi-manual maintenance (read: fsck and refCountUpdate). Yet, after an (admitted, more or less random) sequence of
  BackupPC_fsck
  BackupPC_fsck -f
  BackupPC_fsck -f -s
  BackupPC_refCountUpdate -m
  BackupPC_refCountUpdate -m -F -c
  BackupPC_refCountUpdate -m -F -c -s
  BackupPC_fixupBackupSummary
  BackupPC_nightly 0 255
I'm still stuck with the following messages:

> BackupPC_refCountUpdate: doing fsck on <host> #1188 since there are no poolCnt files
> BackupPC_refCountUpdate: doing fsck on <host> #1190 since there are no poolCnt files
> ...
> BackupPC_refCountUpdate: host <host> got 0 errors (took 5 secs)

The backups in question seem to be fully intact; some are full backups, some are incremental. It's just on a minority of backups (appx. 15 out of 350 backups), and fortunately on small ones where fsck does not take ages, so it does not bother me too much. Nevertheless, can the missing poolCnt data be recomputed? fsck seems to do the counting from scratch; can this be stored?

> BackupPC_fsck: building main count database
> BackupPC_refCountUpdate: missing pool file 00000000000000000000000000000000 count 30
> BackupPC_refCountUpdate: missing pool file 0601e1b90a7f92ce4cffa588ef2cc9da count 1
> ...
> BackupPC_refCountUpdate: missing pool file ea1bd7ab2e0000000000000000000000 count 1
> ...
> BackupPC_refCountUpdate total errors: 70
> BackupPC_fsck: Calling poolCountUpdate

IIUC, this means that there are reference to files with that hash *somewhere* in the backups, but the respective files are missing from the cpool. Since the number is very low, I'm not really worried; in particular, I'm pretty sure that the pool file with hash 0*32 is a spurious one. Nevertheless, I'd like to see which filenames/entries refer to the missing pool files, and if possible delete those altogether to get a "clean" state without nagging messages. Unfortunately, I'm entirely clueless on how to approach that.

FWIW: the server runs on CentOS 7.3, on an ext3 filesystem, with (manually installed) BackupPC 4.0.0, compression activated since the dawn of time.

I'd be grateful for any hints to attack any of the above inconveniences. Thanks!


Best,
Alexander


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/