Hi,
Jeffrey J. Kosowsky wrote on 2009-09-01 01:18:28 -0400 [Re: [BackupPC-users]
Problems with hardlink-based backups...]:
> Michael Stowe wrote at about 23:15:15 -0500 on Monday, August 31, 2009:
> >
> > > I don't see the issue here.
> > > - New files are created only when a new file is added to the
> > > pool. Since this happens coincident with the need for a new database
> > > entry, these two operations can be synchronized
> >
> > Unless there's a database problem. Or the executable crashes. Or a
> > programming bug. Or a database bug. Or the database runs out of space.
> > Or the database crashes. Or somebody runs any number of processes on the
> > database that can interfere with inserts. (I can be more specific with
> > any given database.)
>
> And such issues can and do occur with the current situation. Witness
> the instances of people with corrupted attrib files or unlinked pc
> tree files. The point is that the current situation does not have an
> "atomic" connection between the pool files and the attrib files. Nor
> does it have an "atomic" connection even between the pool files and
> the pc tree hard links. Again, I have had to write specific programs
> to identify and try to correct such errors due to crashes or
> filesystem issues.
[have I quoted enough for your taste?]
The part I'm interested in, for the sake of understandability:
> Again, I have had to write specific programs
> to identify and try to correct such errors due to crashes or
> filesystem issues.
Jeffrey, you "had to" write specific programs because you used BackupPC
without satisfying its requirements: a working file system with reliable hard
link capability. You are the only person I know (with me myself as a close
second!) who would have tried to fix the pool rather than starting over with a
known good (i.e. empty) pool after fixing the file system problems - as
everyone including Craig recommended.
A database also has requirements, which need to be satisfied for it to work.
Databases need control over buffering of their disk writes, for instance. If
you don't give that to them, and their metadata writes are reordered and
buffered indefinitely, then a crash is just about guaranteed to leave your
application data in an inconsistent state (i.e. transactions that were
committed being only partially visible, broken relational integrity ...). Of
course, every new user of BackupPC would know that, so that would not happen,
right?
>>From what I have read about other people's problems - not from my own
experience, though! - I would tend to think that BackupPC's attribute file
handling may actually contain bugs (or, at least, a lack of robustness toward
external errors). It's a design decision whether to fix those by debugging
them or by recoding the whole thing. You say: recode. I say: debug. I'm
willing and intending to debug as soon as I find the time. I have already
written code intended for debugging this matter, as you know, I just haven't
had the problem myself yet. If I remember correctly, you have even extended my
code.
> > And this works, until the database is out of sync, and now work needs to
> > be done to recover database orphans before files can be deleted...
>
> Same as would happen when for example BackupPC_link fails to create a
> hard link between the pool and the pc tree.
The point is, that currently this wastes some space, but it does not corrupt
backups. With a database approach, deleting a file that is still referenced
loses the content forever, so you either have a fix ready that can deal with
the situation above, or you lose your ability to expire backups until you have
coded one. BackupPC is not "finished". There are still things to be completed,
yet it is usable as it is. Your approach, it would seem, would need to be
"finished" before anyone can start using it and expect reliability of it.
> > There's a yawning gap between the ability to envision what can go wrong
> > and what can actually go wrong.
My point exactly. I could name you a few things that could go wrong (as Les
has done), but, frankly, why should I? Mathematically speaking, you are trying
to prove a thesis by disproving an infinite number of alternate possibilities.
If I name you 10 of these possibilities and you disprove them, you gain -
nothing. But, for the exercise, start with Les' points.
You don't need to prove anything to me (or anyone else). You can waste your
time on anything you like. And, as I said, I'm glad if you prove me wrong and
come up with a better BackupPC. I just don't see it, so I won't participate
until you *show me* something.
> > Database/file hybrid systems aren't
> > exactly untrodden ground, and the phenonena of potential desynchronization
> > requires substantial code and effort to overcome, and even then it's
> > problematic if the data isn't wholly redundant between the two systems.
>
> Exactly. They are not untrodden ground and the problems are
> solvable.
You are conveniently ignoring the part, "even then it's problematic if the
data isn't wholly redundant between the two systems". You are also ignoring
that we are talking about evolving open source software that needs to be
reliable if anything.
I've recently had a reiserfs crash which left *all* of the FS contents in
lost+found after a rebuild-tree. I could identify the pool (several, actually,
because BackupPC had re-created it several times before I noticed) and the
pc/ directory. From the pc/ directory, I have access to any backup - with
attributes! - if possibly at a low level. Had I lost a database, the pool
files alone would have been useless. Had I found several potential databases,
named only for their inode number, I would have had a great time figuring out
which one is real or current. Sure, you can provide for that within the
database, assuming it is consistent and easily examined read-only.
By separating database and pool onto two disks (optional, I know) you get the
effect that if either disk dies, you lose all backups, because one disk is
useless without the other. Be sure to document that!
Just one more remark about database design. Within databases, you avoid
redundancies, not because they waste space, but because you will eventually end
up with contradictory information. Which source do you trust more? With a
hybrid system, you *need* redundancy. You could, of course, declare the file
system the master copy and use the database only for "caching", but that
doesn't seem to be what you have in mind (though, I believe, we've been at
that point in a previous thread).
> In fact, all the specific issues you mentioned have actually
> occurred with the existing BackupPC implementation.
How that? Desynchronization between file system and database?
Call me naive, but in my eyes, file name and corresponding contents are
crucial, while file attributes are secondary. I know they are vital often
enough, and I wouldn't be satisfied with a backup solution that ignores them,
so I agree with your point that extended attributes are important. For my SOHO
environment, attributes aren't vital, though. For my clients, they usually
aren't either. They've got a large repository with project data to which
everyone has write access. They want "the document XYZ in the state of last
thursday", not the particular permissions. Without file names, it's pretty much
pointless wading through tens of thousands of files, looking for a particular
state of one file. With backup number and file path, it's no problem at all.
So, in my experience and opinion, a backup solution that will preserve the
really crucial information even in the event that the attributes are lost is
simply superior to one that is less robust towards the crucial information, no
matter how much better it performs concerning the secondary information.
> I am willing to admit that there may be more issues to worry about in
> the hybrid database/file system but to think that such issues are
> exclusive to such systems is just plain wrong. And until one analyzes
> the different case and finds cases that are truly intractable, then
> there is no reason to just blanket throw out this approach.
By all means, do whatever you want. For me, it's not worth the time I'd need
to invest, so I won't. The only point I'm making is that I'd regret BackupPC
going that way without thoroughly analyzing the consequences. But I'm
confident it won't, so I guess my participation in this thread is over. I just
hope new aspects of the original topic are made in a new thread.
> But I get the point - you are happy with the current implementation
> and see no need for other approaches... Others have different needs
> and priorities. Hopefully, you can accept that rather than trying to
> "prove" that your way is the only possible way.
Well, I can only speak for myself, but to me you have come across as
critisizing major design decisions of BackupPC as idiotic (though many people
agree that they aren't), rather than trying to advance a discussion about
possible future enhancements. I know you didn't mean it that way, but that's
how it sounds to me and how I expect others will read it. Yes, many things
could be done differently, but it's a tradeoff most of the time, so even if
you think a different solution would be better, it's valid to think otherwise,
and it would serve you better to explain and justify your solutions/suggestions
rather than just attack the present solutions.
Regards,
Holger
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
