Glad to have another person in the party. What's your birthday? ;)
>Are you seriously suggesting that a quote from "Wikipedia" constitutes
>empirical scientific research?
NO. He said that I was misusing the Birthday Paradox, and I merely
pointed to the Wikipedia article that uses it the same way. If you
search on Birthday Paradox on Google, you'll also find a number of other
articles that use the BP in the same way I'm using it, specifically in
regards to hash collisions, as the concept is not new to deduplication.
It has applied to cryptographic uses of hashing for years.
I then went further to explain WHY the BP applies, and I gave a reverse
analogy that I believe completed my argument that the BP applies in this
situation. So..
As to whether or not what I'm doing is empirical scientific research,
It's not. Empirical research requires testing, observation, and
repeatability. For the record, I have done repeated testing of many
hash-based dedupe systems using hundreds of backups and restores without
a single hash occurrence of data corruption, but that doesn't address
the question. IMHO, it's the equivalent of saying a meteor has never
hit my house so meteors must never hit houses. The discussion is about
the statistical probabilities of a meteor hitting your house, and you
have to do that with math, not empirical scientific research.
>I would be the first to admit that "bob944" has made more than a few
>posts that have "pushed my chair back a couple inches", but at least
>they made me THINK!
And you're saying that my half-a-dozen or so blog postings on the
subject, and none of my responses in this thread don't make you think?
I was fine until I quoted Wikipedia, is that it? ;)
>Is pretty gutsy since you have another post within the past few days
>stating you're ready to RETRACT what you already blogged on this, or
>blogged on that.
I am admitting that I am not a math or statistics specialist and that I
misunderstood the odds before. What's wrong with that? That I was
wrong before, or that I'm stating it publicly that I was wrong before?
I was wrong. I was told I was wrong because I didn't apply the birthday
paradox. So I applied the Birthday Paradox in the same way I see
everyone else applying it, and the way that makes sense according to the
problem, and the numbers still come out OK.
>Wouldn't THAT be saying that up until that point, YOU
>WERE SAYING "that no matter what the entire world is saying -- no
matter
>what the numbers are, you're not going to accept..."
No, because I never said those words or anything like them in my
article. I said, "some people say this, but I say that." Then I even
elicited feedback from the audience. The point of that portion of the
article was that some are talking about hash collisions as if they're
going to happen to everybody and happen a lot, and I wanted to add some
actual math to the discussion, rather than just talk about fear
uncertainty and doubt (FUD). I felt there was a little Henny-Penny
business going on.
>If I am asked to restore something for the CEO, and can't, it won't
>matter a hill of beans what all the theory was and what the odds were.
I
>either can, or I can't. I'll be accountable for that result, and why I
>got it. As someone so accurately posted recently: We're in the recovery
>business, not the restore business.
You won't get any argument from me. I think you'll find almost that
exact sentence in the first few paragraphs of any of my books. Having
said that, we all use technologies as part of our backup system that
have a failure rate percentage (like tape). And to the best of my
understanding, the odds of a single hash collision in 95 Exabytes of
data is significantly lower than the odds of having corrupted data on an
LTO tape and not even knowing it, based on the odds they publish. Even
if you make two copies, the copy could be corrupted, and you could have
a failed restore. Yet we're all ok with that, but we're freaking out
about hash collisions, which statistically speaking have a MUCH lower
probability of happening.
>I would thing that almost everyone on this forum does some kind of
pilot
>before rolling something out into production.
I sure as heck hope so, but I don't think it addresses this issue. So
you test it and you don't get any hash collisions. What does that prove?
It proves that a meteor has never hit your house.
What I recommend (especially if you're using a hash-only de-dupe system)
is a constant verification of the system. Use a product like NBU that
can do CRC checks against the bytes it's copying or reading, and either
copy all de-duped data to tape or run a NBU verify on every backup. If
you have a hash collision, your copy or verify will fail, and at least
know when it happens.
>I hope I'm wrong.
About what? That I'm an idiot? ;) I think judging me solely on this
long, protracted, difficult to follow discussion (with over 70 posts) is
probably unfair. Remember also that these posts are often done on my
own time late at night, etc. I never claimed to be perfect.
>I love to learn. I'm actually signed up for one of
>your classes next week. But, if quoting everyone else's
>posts/blogs/Wikipedia entries, etc. without backing up re-posting them
>with empirical evidence or firsthand testing is your program agenda, I
>will skip the engagement...
I don't think you'll find that to be a problem. I'm an in-the-trenches
guy, who has sat in front of many a tape drive, tape library, and backup
GUI in my 14 years in this space. I actually cut my teeth right down
the road from you as the backup guy at MBNA. (I lived in Newark, DE,
and you were my bank.) Don't skip out on the school just because of I
quoted Wikipedia once.
>TW - You "Tilt at Windmills" (Don Quixote), you don't chase them. ;-)
You are right. I stand corrected again. Even Wikipedia backs you up:
http://en.wikipedia.org/wiki/Don_Quixote
(Sorry, just couldn't resist.) ;)
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
