>> We had a situation with one puny C20 with 256MB of memory where they had
>> architected the application to write images files (50K to 200K each)
>> into one single directory.  Unfortunately, they tracked around 1.5
>> million files in a 90 day period.
>
>I believe the extremely poor performance was due to a design problem
>with JFS.   We've battled with this.   We think it's due to an in-core
>i-node table being filled.   We found that when the number of files
>in a single directory exceeds the size of this table (the size of the
>table is computed at boot time based on the amount of physical memory)
>reading i-nodes will peg the CPU and take a looooooong time.

>We opened a crit-sit (critical situation) with IBM because of a new
>application which will have 1.5 million files in one directory.
>Their only solution so far is to either rewrite the application or
>buy a Sun.

This is an old issue which comes up about every two months on the List.
It's not an exotic problem, but merely that traditional file system directories
are primitive data structures which are extremely inefficient and bog down
when you attempt to keep more than about a thousand files in a single
directory.  That's why there are subdirectories.  Try to have your directory
structure similar to an equilateral triangle and you will enjoy much better
performance.  And, no, buying a Sun is not the solution: I know from
experience that the same situation applies there.

Rather than take up the issue with IBM, it would be more appropriate to take
it up with the people who mis-designed the application which tries to keep
such an unreasonable number of files within one directory level - they don't
seem to have the benefit of experience to appreciate the impact that has.
A conventional directory is not a database: it performs poorly if you attempt
to use it as such.

  Richard Sims, Boston University OIT