ADSM-L

Re: ADSM versus Arcserve and Backup Exec

1998-09-20 18:30:06
Subject: Re: ADSM versus Arcserve and Backup Exec
From: Peter Gathercole <peter.gathercole AT VIRGIN DOT NET>
Date: Sun, 20 Sep 1998 23:30:06 +0100
Large directories have ALWAYS been a problem with virtually all Unixes until
recently. The directory is actually a file, and all the normal indirect block
lookup problems affect directories just as much as ordinary files. In
addition, the positioning of the Inode information across the physical disk
can lead to many seek operations to read the referenced Inodes. As I
understand it, JFS spreads the Inodes across the complete file system,
locating some Inodes in each LP of the logical volume. This has some
significant speed  advantages when using many directories with a reasnoable
number of files as the Inode information should be close to the directory
files that reference the Inodes, but may break down when using few directories
with many files. I'm not sure how Sun have got around this (as this type of
distribution of Inode information was inherited from BSD). What they may have
done is to MMap the directory files and Inode information, and allowed the
virtual memory management system to cope with caching the information, rather
than using an in-core table.

What I suspect is the case is that IBM and Sun have tuned their systems using
different criteria. In addition, the security that JFS gives regarding
filesystem corruption does not com for free....

Peter Gathercole
Open Systems Consultant.

Gene Mangum wrote:

> On Tue, 15 Sep 1998, David Hendrix wrote:
> > We had a situation with one puny C20 with 256MB of memory where they had
> > architected the application to write images files (50K to 200K each)
> > into one single directory.  Unfortunately, they tracked around 1.5
> > million files in a 90 day period.
> >
> > V2 client choked and so did the V3 client (heck they couldn't even ls
> > the directory!).  They have rearchitected their system, and with the
>
> I believe the extremely poor performance was due to a design problem
> with JFS.   We've battled with this.   We think it's due to an in-core
> i-node table being filled.   We found that when the number of files
> in a single directory exceeds the size of this table (the size of the
> table is computed at boot time based on the amount of physical memory)
> reading i-nodes will peg the CPU and take a looooooong time.
>
> We opened a crit-sit (critical situation) with IBM because of a new
> application which will have 1.5 million files in one directory.
> Their only solution so far is to either rewrite the application or
> buy a Sun.
>
> Note that plain old "ls" is not so bad, but "ls -l", which reads the
> i-nodes, takes forever at 100% CPU.
>
> --
> Gene Mangum
> University of Michigan Medical Center
<Prev in Thread] Current Thread [Next in Thread>