Robin Lee Powell wrote at about 16:28:43 -0800 on Monday, December 14, 2009:
> On Mon, Dec 14, 2009 at 02:17:01PM -0500, Jeffrey J. Kosowsky wrote:
> > Robin Lee Powell wrote at about 10:12:28 -0800 on Monday, December 14,
> > 2009:
> > > On Mon, Dec 14, 2009 at 07:57:10AM -0600, Les Mikesell wrote:
> > > >
> > > > You can, however, explicitly break the runs at top-level
> > > > directory boundaries and mount points if you have a problem
> > > > with the size.
> > >
> > > That doesn't always work; it certainly doesn't work in my case.
> > > Millions of files scattered unevenly around a single file
> > > system; I don't even know where the concentrations are because
> > > it takes so long to run du/find on this filesystem, and it
> > > degrades performance in a way that makes the client upset.
> > >
> >
> > I wonder how common your use case is where the files are scattered
> > so unevenly, so unpredictably, and in such a dynamically changing
> > manner that you can't make a dent in the complexity by subdividing
> > the share into smaller pieces. If the system is so dynamic and
> > unpredictable, then perhaps the more robust solution is to see
> > whether the data storage can be organized better...
>
> We're a hosting company; these are backups for clients. We can't
> enforce that sort of shift.
>
> Since I've got two clients with exactly the same issue (file trees
> in the millions of files, that take ten hours or more just to run a
> "find" on, let alone du), though, I'm inclined to think that it's
> not *that* uncommon.
>
> I find it more than a little odd that you are telling me "OMG
> HARDLINKS!" on the one hand, and "subdivide the share" on the other,
> since by definition subdivided shares break hard links. What's up
> with that? (not intended confrontationally, just confused)
No confrontation taken ;)
But if you follow the thread, I wasn't the one initially suggesting
subdividing the share nor am I pushing that solution. However, I think
that if you really can't do it all in one session, then subdividing is
a more recommended solution than suggesting a change to the backuppc
source (unless you just do it on your own).
Perhaps your setup is pushing the envelope more than others, but I
haven't seen many people with issues like yours. It would be
interesting to hear whether others are experiencing the same
problem. Also, before you go off rewriting backuppc, it might be
productive to investigate whether there are any correctable
software/hardware issues in your hosting setup that might be
contributing. You postulate several potential causes for your SIGPIPE
errors, but it would be good to identify the actual source(s) and
trigger(s).
ssh and rsync are pretty mature and robust software and they shouldn't
just "flake" out just from being active for 15 hours or from being
pounded hard with lots of files. While my setup is much much smaller
than yours, I keep ssh connections open for months using very
off-the-shelf consumer grade hardware. And if there is something
flakey in your hardware/software, you probably want to know that if
you are a hosting company...
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
