Holger Parplies wrote at about 18:57:05 +0100 on Friday, November 1, 2013:
> Hi,
>
> I get some diagnostics when reading this with 'use warnings "wrong_numbers"'
> ...
>
> backuppc AT kosowsky DOT org wrote on 2013-11-01 12:18:17 -0400 [Re:
> [BackupPC-users] Disk space used far higher than reported pool?size]:
> > Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013:
> > > >And this would explain why the elements are not being linked properly
> > to
> > > the pool -- though I would have thought the more likely result would be
> > a
> > > duplicate pool entry than an unlinked pool entry...
> > >
> > > >It might be interesting to look for pool chains with the same
> > (uncompressed)
> > > content and with links < HardLinkMax (typically 31999) to see if pool
>
> this one looks correct. 31999. Unless of course you've changed it in
> config.pl
> because your FS requirements differ.
>
> > > entries are being unnecessarily duplicated.
> > >
> > > >Try: (cd /var/lib/BackupPC/cpool; find . -type f -links -3198 -name
> > "*_*" -exec
>
> This one doesn't.
Oops typo... dropped a 9
>
> > > md5sum {} \;) | sort | uniq -d -w32
> > >
> > > > Note this will find if there are any unnecessarily duplicated pool
> > chains
> > > (beyond the base one). Note to keep it fast and simple I am
> > > > skipping the elements without a suffix... with the assumption being
> > that
> > > if there are duplicated elements then there will probably be
> > > > whole chains of them...
> > >
> >
> > I added some more bash-foo so that the following should find *any* and
> > *all*
> > unnecessary pool dups...
> >
> > (cd /var/lib/BackupPC/cpool; find . -name "*_0" | sed "s/_0$//" |
> > (IFS=$'\n'; while read FILE; do find "${FILE}"* -links -3199 -exec md5sum
> > {} \; | sort | uniq -D -w32 ; done))
>
> Nor does this one (the 3199 again).
Typo again... dropped a 9
> You probably wouldn't have chosen the (arbitrary) value
> "3199", though, if you hadn't in fact meant "31999" ;-). And you wouldn't be
> saying "*any* and *all*" if you were meaning "some".
>
> I'd like to point out three things:
> 1.) unnecessary duplication *within* the pool is not the problem we are
> looking for,
This is probably not his *primary* issue since the pool is (only)
~3T. But when he started talking about file read errors, I was
concerned that if the pool file reads were being truncated, then there
likely would be pool duplicates since the byte-by-byte comparisons
would fail for a given partial file md5sum leading to extra chain creation...
> 2.) if it were a problem, then because a duplicate was created way ahead of
> time and repeatedly, not because the overflow happens at 31950 instead of
> 31999,
> 3.) finding "unnecessary duplicates" can have a normal explanation: if at
> some
> point you had more than 31999 copies of one file (content) in your
> backups, BackupPC would have created a pool duplicate. Some of the
> backups
> linking to the first copy would have expired over time, leaving behind a
> link count < 31999. Further rsync backups would tend to link to the
> second
> copy, at least for unchanging existing files (in full backups). In other
> cases, the first copy might be reused, but there's no guarantee the link
> count would be exactly 31999 (though it would probably tend to
> be).
You are absolutely right that there are valid reasons for the link
count overflowing 31999 and then later dropping below as links are
expired.
To tell you the truth, my use of "-links 31999" (corrected) was really
more pedantic -- in reality, I have never seen a case of link counts
getting that high... and if it does, it's probably extremely rare to
have a single non-zero file repeated that many times unless you have a
huge set of clients or a huge set of past full backups... (or some
special situation where users keep large numbers of copies of certain
files).
So, basically, while there may be an exceptional case or two, anything
spewed back by my shell one-liner is worth looking at from the
perspective of potential issues with pool duplication.
> Having so many copies of identical file content in your backups would
> tend
> to happen for small files rather than huge ones, I would expect, and it
> doesn't seem to be very common anyway (in my pools, I find exactly one
> file
> with a link count of 60673 (XFS) and a total of five with more than 10000
> links, the largest having 103 bytes (compressed)).
Exactly -- that's my point.
So other than your one case of 60673 links, any other case of a
duplicate pool chain would be due to an error somewhere...
You may remain correct that adding "-nlinks 31999" or "-nlinks 31500"
or any similar number is not going to limit the search in
reality... and therefore won't have much of a practical difference...
------------------------------------------------------------------------------
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
