Kris Vassallo wrote:
The box is running redhat 9 with 2.4.20 kernel and ext3 filesystem.
...
sendsize[27747]: time 0.156: spawning /usr/lib/amanda/runtar in pipeline
> [...]
sendsize[27747]: time 11114.835: .....
sendsize[27747]: estimate time for /home level 0: 11114.679
sendsize[27747]: estimate size for /home level 0: 419847640 KB
sendsize[27747]: time 11114.835: waiting for /bin/tar "/home" child
sendsize[27747]: time 11114.835: after /bin/tar "/home" wait
sendsize[27747]: time 11114.882: getting size via gnutar for /home level 6
sendsize[27747]: time 11115.510: spawning /usr/lib/amanda/runtar in pipeline
[...]
sendsize[27747]: time 18815.342: Total bytes written: 69237104640 (64GB,
8.6MB/s)
sendsize[27747]: time 18815.372: .....
sendsize[27747]: estimate time for /home level 6: 7699.861
sendsize[27747]: estimate size for /home level 6: 67614360 KB
sendsize[27747]: time 18815.372: waiting for /bin/tar "/home" child
sendsize[27747]: time 18815.372: after /bin/tar "/home" wait
sendsize[27747]: time 18815.409: done with amname '/home', dirname
'/home', spindle -1
sendsize[27717]: time 18815.493: child 27747 terminated normally
sendsize: time 18815.503: pid 27717 finish time Tue Jul 20 06:13:36 2004
That was 11000 seconds for a level 0 estimate plus 7700 seconds for a
level 6. It could have been worse, when amanda also wanted a level 7
estimate to choose from.
A long time ago, I had also an "long estimates" problem, where a UFS
filesystem on Solaris 2.6 . It was a filesystem with many small files.
GNutar really takes a long time on that.
Some larger filesystems on the same host estimated much quicker, but
those had much less files (but larger ones).
I had the opportunity to migrate the troublesome filesystem to another
host. I just tried ext3 on Linux, which was faster (maybe because the
host also was faster), but not very much. Then I tried Reiserfs.
The time went down incredibly: from over 2 hours on the old machine to
less than 15 minutes on the new machine with Reiserfs.
Just a datapoint...
If you install dump for ext2, then you should also try out that one.
Dump takes only a few seconds or minutes compared to gnutar for such
filesystems.
Splitting the filesystem in smaller DLE's will not gain very much speed
as almost all those little files still need to be estimated. This means
it needs a stat() syscall on each file or directory to determine if the
file is changed, and that's what makes gnutar slow.
The stat-call on a reiserfs is really really very fast. (I never tested
with JFS or XFS.)
A complete different approach is to base the estimates on some
intelligent statistics from the previous runs. Jean-Louis is
experimenting with that idea (or implementing already?).
--
Paul Bijnens, Xplanation Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens AT xplanation DOT com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************
|
