Hi,
Les Mikesell wrote on 2009-06-10 15:45:22 -0500 [Re: [BackupPC-users] backup
the backuppc pool with bacula]:
> jhaglund wrote:
> > There are several implied references here to likely problems with rsync
> > and how they are all deal breakers. [...] I just need to understand what's
> > going on.
>
> It boils down to how much RAM rsync needs to handle all the directory
> entries and hardlinks and the amount of time it takes to wade through
> them.
... where the important part is the hardlinks (see below), because that simply
can't be optimized, the file list - while probably consuming more memory in
total - can and has been in 3.0 (probably meaning protocol version 30, i.e.
rsync 3.x on both sides).
> > I'm running rsync 3.0.6 but the server is 2.6.x. I have ~ 1.9 files
> > found by rsync and it always fails on some level. [...]
>
> 3.x on both ends might help. It claims to not need the whole directory
> in memory at once - but you'll still need to build a table to map all
> the inodes with more than one link (essentially everything) to
> re-create the hardlinks so you have to throw a lot of RAM at it anyway.
Please read the above carefully. It's not about "so many hardlinks" (meaning
many links to one pool file), it's about "so many files that have more than one
link" - whether it's 2 or 32000 is unimportant (except for the size of the
complete file list, which additional hardlinks will make larger). In "normal"
situations, you have a file with more than one link every now and then. rsync
expects to have to handle a few of them. With a BackupPC pool it's practically
every single file, millions of them or more in some cases. And for each and
every one of them, rsync needs to store (at least) the inode number and the
full path (probably relative to the transfer root) to one link (probably the
first one it encounters, not necessarily the shortest one). Count for yourself:
cpool/1/2/3/12345678911234567892123456789312
pc/foo/0/f%2fhome/fuser/ffoo
pc/hostname/123/f%2fexport%2fhome/fwopp/f.gconf/fapps/fgnome-screensaver/f%25gconf.xml
Round up to a multiple of 8, add maybe 4 bytes of malloc overhead, 4 bytes for
a pointer, and factor in that we're simply not used anymore to optimizing
storage requirements at the byte level.
You're probably going to say, "why not simply write that information to
disk/database?".
Reason 1: That's a lot of temporary space you'll need. If it doesn't fit in
memory, we're talking about GB, not a few KB.
Reason 2: Access to this table will be in random order. It's not a nice linear
scan. Chances are, you'll need to read from disk almost every time.
No cache is going to speed this up much, because no cache will be
large enough or smart enough to know when which information will be
needed again. The same applies to a database.
Reason 3: rsync is a general purpose tool. It can't determine ahead of time
how many hardlink entries it will need to handle. It could only
react to running out of memory. Except for BackupPC pools, it would
probably *never* need disk storage.
> You shouldn't actually crash unless you run out of both ram and swap,
> but if you push the system into swap you might as well quit anyway.
This is the same as reason 2. You should realize that disk is not "slightly
slower" than RAM, it's many orders of magnitude slower. It won't take 2 hours
instead of 1 hour, it will take 10000 hours (or more) instead of 1. That is
over one year. Swap works well, as long as your working set fits into RAM.
That is not the case here. [In reality, it might not be quite so dramatic,
but the point is: you don't know. It simply might take a year. Or 10.
Supposing your disks last that long ;-]
> > What does one use if not rsync?
>
> The main alternative is some form of image-copy of the archive
> partition. This is only practical if you have physical access to the
> server or very fast network connections.
"Physical access" probably meaning, that you can transport your copy to and
from the server. "Never underestimate the bandwidth of a station waggon full
of tapes hurtling down the highway." (Andrew S. Tanenbaum).
> > Do I really need to shut down backuppc every time I want to attempt a
> > sync or would syncing to a local disk and rsync'ing from that be
> > sufficient?
Try something like "time find /var/lib/backuppc -ls > /dev/null" to get a
feeling for just how long only traversing the BackupPC pool and doing a stat()
on each file really takes. Then remember that "syncing to a local disk" is in
no way simpler than syncing to a remote disk - the bandwidth for copying is
simply higher, so that is the only place you get a speedup.
>>From a different perspective: either it's going to be fast enough that
shutting down BackupPC won't hurt, or it's going to be *necessary* to shut
down BackupPC, because having it modify the file system would hurt.
Just imagine the pc/ directory links on your copy would point to a wrong pool
file, because a pool chain got renumbered during your "sync".
> > I'd really like to know the specifics of the hardlink and inode problem
> > talked about in this thread like how to find out how many I have
Inodes in pool file system (including directories and anything outside $TopDir
on the same file system):
'df -i $TopDir' (should be fast).
Number of files in pool:
'find $TopDir/pool $TopDir/cpool -type f -print | wc -l' (may take a while).
Total number of file type directory entries (i.e. hardlinks):
'find $TopDir -type f -print | wc -l' (may take *long*).
Links to pool files from pc/ directories:
'find $TopDir/pc -type f -links +1 -print | wc -l' (also slow).
> > and what the threshold is for Trouble
That depends on your system (memory, path names, running processes, kernel
configuration, ...). But really, what's the point? You seem to know you're
above the threshold, because copying fails. It's not like there's a sharp
boundary, below which everything will suddenly work flawlessly. At some point,
it will, obviously, but that's for one pool snapshot and one set of outer
circumstances.
The problem is really that most pools will grow over time, so even if things
work now, they probably won't forever. What happens then? It would be nice to
have an estimate of when "then" will be, but I have no idea how you would get
that.
> > and how the rest of the community deals with getting pools of 100+GB
> > offsite in less than a week of transfer time.
>
> 100 Gigs might be feasible - it depends more on the file sizes and how
> many directory entries you have, though. And you might have to make the
> first copy on-site so subsequently you only have to transfer the changes.
Does anyone actually have experience with rsyncing an existing pool to an
existing copy (as in: verification of obtaining a correct result)? I'm kind of
sceptical that pool chain renumbering will be handled correctly. At least, it
seems extremely complicated to get right.
Regards,
Holger
------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
