-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Adam Goryachev wrote:
> Christoph Litauer wrote:
>> Craig Barratt schrieb:
>>> Christoph writes:
>>>
>>>> If I take a look on the structure of incrementals, I can see lots of
>>>> empty directories. It seems as if the whole directory structure of the
>>>> backup-source is kind of "duplicated" to the backup disk - although most
>>>> of the directories (and the files in them) are unchanged.
>>>> This leads to _lots_ of files/directories on the backuppc-disk (about 20
>>>> million now). Is it necessary?
>>> Yes - the directory structure needs to be complete, even for
>>> an incremental. The storage used should be small.
>> Craig,
>
>> can you explain why, please?
>
>> You're right: The storage amount is very small. But one can get _very_
>> large directory structures on the backup filesystem. My BackupPC volume
>> now uses 147,650,611 inodes in an XFS filesystem. (I think) this leads
>> to a very slow directory creation:
>> time for i in `seq 1 10000`; do mkdir $i; done
>> runs about 2.5 minutes! This is 66 directories per second, whereas the
>> same command on the same server but another (empty) xfs filesystem took
>> only 34 seconds (about 5 times faster).
Using reiserfs I get these results:
keep:/var/lib/backuppc/tmp# time for i in `seq 1 10000`; do mkdir $i; done
real 0m22.765s
user 0m7.192s
sys 0m15.497s
/dev/hdc1 on /var/lib/backuppc type reiserfs (rw,noatime,data=writeback)
keep:/var/lib/backuppc/pc# find . -type d|wc -l
5999764
keep:/var/lib/backuppc/pc# find . -type f|wc -l
40436331
On another host I have:
debian:/var/lib/backuppc/pc# find . -type d|wc -l
34817927
I haven't counted the file entries, as it will take too long... but here
is the create directory timing:
debian:/var/lib/backuppc/tmp# time for i in `seq 1 10000`; do mkdir $i; done
real 0m18.008s
user 0m5.092s
sys 0m12.005s
>> Let's take a look at one (out of 10) of the backup clients:
>> A level 0 dump consists of 285,207 directories and 1,140,363 files.
>> A level 5 dump consists of 285,041 directories and 94,449 files.
>> This level 5 dump contains 192069 emtpy directories.
>
>> I am afraid of expanding my backup scheme ... more clients and longer
>> incremental keep periods will lead to lots of millions of empty
>> directories ...
That second machine has daily backups for 4 servers which retain ALL
backups. So far, they have 683 backups each.... I guess that is
equivalent to having more hosts with less backups being kept....
Hope that helps.... but at the end of the day, you obviously have a
performance issue, and will need to track that down. I haven't seen
anyone else with XFS report their statistics, but that might be more
helpful, then you will know whether it is a local issue for you, or
something common to the XFS filesystem, which you could resolve by
changing to a different FS format, or by talking to the XFS developers
for assistance in improving the performance.
Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFIYvNhGyoxogrTyiURAolhAJ9VpnGWcoqTVXZui+5MX0wX8MFRkgCgzYua
Dd1RS+eB01hTLtF3FsJreYU=
=zVSJ
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
