Networker

Re: [Networker] Large directories slow acces using save -I input_ filename

2003-07-24 08:53:57
Subject: Re: [Networker] Large directories slow acces using save -I input_ filename
From: Riaan Louwrens <riaanl AT SOURCECONSULTING.CO DOT ZA>
To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
Date: Thu, 24 Jul 2003 15:00:55 +0200
We have seen under windows that directories containing more than 10,000
files tend to slow the backup process down - on unix it was a tad more -
more than 100,000.

This is a operating system "limitation" and has is the way that the Os walks
the filesystem - nothing to do with networker (I have seen this on Laptop,
NRM, DiskXtender etc etc) ...

We have had a lot of improvement by getting clients to try and limit files
in a directory to less than 10,000 (i.e. create sensible subdirectories).

Regards,

Riaan

-----Original Message-----
From: Lubos Bohm [mailto:lubos.bohm AT OSKARMOBIL DOT CZ]
Sent: Thursday, July 24, 2003 2:48 PM
To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
Subject: [Networker] Large directories slow acces using save -I
input_filename


Hi All,


trying to backup aprox. 200.000 files listed in "input_filename" from large
directory containing aprox. 960.000 files takes IMHO too long. Well, it
takes hours before the tape is even loaded and first bytes are written..

We use:
1/ save -s lgt-blan -q -I /var/opt/Storage/Apr.in -b"BGW"
   on Solaris 8 with VXFS
2/ Networker 6.1.1 build 238 on both sides

I suppose the problem is inside the "save" - it can not search through the
large directory.

Folks, what are your experiences? Could it be a bug solved in new version?
Or should I better take care about pre-processing these files like moving
them in a directory tree (like for example Squid does)? Or some advanced
VxFS tuning?

Thanks a lot, Lubos

--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=

--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=

<Prev in Thread] Current Thread [Next in Thread>