Holger Parplies wrote at about 04:22:03 +0200 on Thursday, June 11, 2009:
> Hi,
>
> Les Mikesell wrote on 2009-06-10 15:45:22 -0500 [Re: [BackupPC-users] backup
> the backuppc pool with bacula]:
> > jhaglund wrote:
> > > There are several implied references here to likely problems with rsync
> > > and how they are all deal breakers. [...] I just need to understand
> > > what's
> > > going on.
> >
> > It boils down to how much RAM rsync needs to handle all the directory
> > entries and hardlinks and the amount of time it takes to wade through
> > them.
>
> ... where the important part is the hardlinks (see below), because that
> simply
> can't be optimized, the file list - while probably consuming more memory in
> total - can and has been in 3.0 (probably meaning protocol version 30, i.e.
> rsync 3.x on both sides).
>
Holger, I may be wrong here, but I think that you get the more
efficient memory usage as long as both client & server are version >=3.0
even if protocol version is set to < 30 (which is true for BackupPC
where it defaults back to version 28).
I think protocol 30 has more to do with the changes from md4sums to
md5sums plus the ability to have longer file names (>255 characters I
think) plus other protocol extensions. But I'm not an expert and my
understanding is that the protocols themselves are not well documented
other than looking through the source code.
> > > I'm running rsync 3.0.6 but the server is 2.6.x. I have ~ 1.9 files
> > > found by rsync and it always fails on some level. [...]
> >
> > 3.x on both ends might help. It claims to not need the whole directory
> > in memory at once - but you'll still need to build a table to map all
> > the inodes with more than one link (essentially everything) to
> > re-create the hardlinks so you have to throw a lot of RAM at it anyway.
>
> Please read the above carefully. It's not about "so many hardlinks" (meaning
> many links to one pool file), it's about "so many files that have more than
> one
> link" - whether it's 2 or 32000 is unimportant (except for the size of the
> complete file list, which additional hardlinks will make larger). In "normal"
> situations, you have a file with more than one link every now and then. rsync
> expects to have to handle a few of them. With a BackupPC pool it's
> practically
> every single file, millions of them or more in some cases. And for each and
> every one of them, rsync needs to store (at least) the inode number and the
> full path (probably relative to the transfer root) to one link (probably the
> first one it encounters, not necessarily the shortest one). Count for
> yourself:
>
> cpool/1/2/3/12345678911234567892123456789312
> pc/foo/0/f%2fhome/fuser/ffoo
>
> pc/hostname/123/f%2fexport%2fhome/fwopp/f.gconf/fapps/fgnome-screensaver/f%25gconf.xml
>
> Round up to a multiple of 8, add maybe 4 bytes of malloc overhead, 4 bytes
> for
> a pointer, and factor in that we're simply not used anymore to optimizing
> storage requirements at the byte level.
>
>
> You're probably going to say, "why not simply write that information to
> disk/database?".
>
> Reason 1: That's a lot of temporary space you'll need. If it doesn't fit in
> memory, we're talking about GB, not a few KB.
> Reason 2: Access to this table will be in random order. It's not a nice
> linear
> scan. Chances are, you'll need to read from disk almost every time.
> No cache is going to speed this up much, because no cache will be
> large enough or smart enough to know when which information will be
> needed again. The same applies to a database.
> Reason 3: rsync is a general purpose tool. It can't determine ahead of time
> how many hardlink entries it will need to handle. It could only
> react to running out of memory. Except for BackupPC pools, it would
> probably *never* need disk storage.
>
> > You shouldn't actually crash unless you run out of both ram and swap,
> > but if you push the system into swap you might as well quit anyway.
>
> This is the same as reason 2. You should realize that disk is not "slightly
> slower" than RAM, it's many orders of magnitude slower. It won't take 2 hours
> instead of 1 hour, it will take 10000 hours (or more) instead of 1. That is
> over one year. Swap works well, as long as your working set fits into RAM.
> That is not the case here. [In reality, it might not be quite so dramatic,
> but the point is: you don't know. It simply might take a year. Or 10.
> Supposing your disks last that long ;-]
>
> > > What does one use if not rsync?
> >
> > The main alternative is some form of image-copy of the archive
> > partition. This is only practical if you have physical access to the
> > server or very fast network connections.
>
> "Physical access" probably meaning, that you can transport your copy to and
> from the server. "Never underestimate the bandwidth of a station waggon full
> of tapes hurtling down the highway." (Andrew S. Tanenbaum).
>
> > > Do I really need to shut down backuppc every time I want to attempt a
> > > sync or would syncing to a local disk and rsync'ing from that be
> > > sufficient?
>
> Try something like "time find /var/lib/backuppc -ls > /dev/null" to get a
> feeling for just how long only traversing the BackupPC pool and doing a
> stat()
> on each file really takes. Then remember that "syncing to a local disk" is in
> no way simpler than syncing to a remote disk - the bandwidth for copying is
> simply higher, so that is the only place you get a speedup.
>
> >From a different perspective: either it's going to be fast enough that
> shutting down BackupPC won't hurt, or it's going to be *necessary* to shut
> down BackupPC, because having it modify the file system would hurt.
>
> Just imagine the pc/ directory links on your copy would point to a wrong pool
> file, because a pool chain got renumbered during your "sync".
>
> > > I'd really like to know the specifics of the hardlink and inode problem
> > > talked about in this thread like how to find out how many I have
>
> Inodes in pool file system (including directories and anything outside
> $TopDir
> on the same file system):
> 'df -i $TopDir' (should be fast).
>
> Number of files in pool:
> 'find $TopDir/pool $TopDir/cpool -type f -print | wc -l' (may take a while).
>
> Total number of file type directory entries (i.e. hardlinks):
> 'find $TopDir -type f -print | wc -l' (may take *long*).
>
> Links to pool files from pc/ directories:
> 'find $TopDir/pc -type f -links +1 -print | wc -l' (also slow).
>
> > > and what the threshold is for Trouble
>
> That depends on your system (memory, path names, running processes, kernel
> configuration, ...). But really, what's the point? You seem to know you're
> above the threshold, because copying fails. It's not like there's a sharp
> boundary, below which everything will suddenly work flawlessly. At some
> point,
> it will, obviously, but that's for one pool snapshot and one set of outer
> circumstances.
>
> The problem is really that most pools will grow over time, so even if things
> work now, they probably won't forever. What happens then? It would be nice to
> have an estimate of when "then" will be, but I have no idea how you would get
> that.
>
> > > and how the rest of the community deals with getting pools of 100+GB
> > > offsite in less than a week of transfer time.
> >
> > 100 Gigs might be feasible - it depends more on the file sizes and how
> > many directory entries you have, though. And you might have to make the
> > first copy on-site so subsequently you only have to transfer the changes.
>
> Does anyone actually have experience with rsyncing an existing pool to an
> existing copy (as in: verification of obtaining a correct result)? I'm kind
> of
> sceptical that pool chain renumbering will be handled correctly. At least, it
> seems extremely complicated to get right.
Why wouldn't rsync -H handle this correctly?
It might be slow but it should always faithfully copy $topdir source
to destination (assuming you are doing --delete too). And the
renumbering will change the timestamps which should alert rsync to all
the changes even without the --checksum flag.
Or are you saying it would be difficult to do this manually with a
special purpose algorithm that tries to just track changes to the pool
and pc files?
More generally, I think we really need to find a guinea pig to spend
some time testing the methods that you and I have discussed of
creating a sorted inode database of the pool. Then it would be
instructive to compare execution times vs the straight rsync -H method
and vs. the tar method. For small pools, I imagine rsync -H would be
faster, but at some point the database would presumably be
faster. Presumably the tar method would be slowest of all. The devil
of course is in the details.
Either way, this issue seem to be becoming a true FAQ for this list --
so we should probably agree on some definitive answer (or set of
answers) so that we can put this one to rest.
My personal belief is that while disk images or ZFS may be the "ideal"
answer, there still is a need for an alternative even if slower method
for reliably backing up (and ideally incrementally synching) just
$topdir for those who don't/can't back up the whole partition or who
can't run ZFS. My understanding is that the simple answer of "rsync -H"
seems to not be reliable enough on large pools at least for some
people.
>
> Regards,
> Holger
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users AT lists.sourceforge DOT net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
