On Thu, Jun 26, 2008 at 02:59:20PM +0200, Christoph Litauer wrote:

> >>>>> If I take a look on the structure of incrementals, I can see lots of
> >>>>> empty directories. It seems as if the whole directory structure of the
> >>>>> backup-source is kind of "duplicated" to the backup disk - although most
> >>>>> of the directories (and the files in them) are unchanged.
> >>>>> This leads to _lots_ of files/directories on the backuppc-disk (about 20
> >>>>> million now). Is it necessary?
> >>>> Yes - the directory structure needs to be complete, even for
> >>>> an incremental.  The storage used should be small.
> >>> Craig,
> >>> can you explain why, please?
> >>> You're right: The storage amount is very small. But one can get _very_
> >>> large directory structures on the backup filesystem. My BackupPC volume
> >>> now uses 147,650,611 inodes in an XFS filesystem. (I think) this leads
> >>> to a very slow directory creation:
> >>>     time for i in `seq 1 10000`; do mkdir $i; done
> >>> runs about 2.5 minutes! This is 66 directories per second, whereas the
> >>> same command on the same server but another (empty) xfs filesystem took
> >>> only 34 seconds (about 5 times faster).
> 
> > Hope that helps.... but at the end of the day, you obviously have a
> > performance issue, and will need to track that down. I haven't seen
> > anyone else with XFS report their statistics, but that might be more
> > helpful, then you will know whether it is a local issue for you, or
> > something common to the XFS filesystem, which you could resolve by
> > changing to a different FS format, or by talking to the XFS developers
> > for assistance in improving the performance.
> 
> Thanks a lot Adam!
> In the meantime I discussed my problem on the xfs mailing list. We are 
> not finished yet, but adding mount option "nobarrier" reduced my 
> performance problems significantly. 

I added that option to my XFS, just in case. My install shouldn't be
using barriers though, because it says:

<5>Filesystem "dm-1": Disabling barriers, not supported by the underlying device

during booting the system...

> I am still in contact to clarify if it's possible to optimize the
> usage of inode allocation groups. We will see ...

I once thought it would be sensible to have larger directory blocks, but
then, a lot of them will turn out empty (except the pool). BackupPC is a
rather extreme use case: lots of files in one place, very little files
but lots of directories in the other place.

Bye,

Tino.

-- 
"What we nourish flourishes." - "Was wir nähren erblüht."

www.craniosacralzentrum.de
www.forteego.de

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/