On Tue, Mar 21, 2000 at 09:27:09AM +0100, Mauro M. TINELLI wrote:
> what about the "FS" structure you are using?

We're using a large file enabled JFS.  The various FS parameters are:

  prompt> lsfs -q /dev/hsm
  ...
  (lv size: 497025024, fs size: 497025024, frag size: 4096, nbpi:
  32768, compress: no, bf: true, ag: 64)

> Is it possible you've
> reached the point to fill up some table (i-node table full in the old FSs
> or fragmentation or whatever else in that structure of FS you're using)

There are plenty of free inodes (only 16% used), but the problem could
indeed be fragmentation (keep reading...).

> What I mean is
> that your FS might allocate 4meg immediately and looks for further space
> which it cannot find.

Indeed; this may be exactly what is happening.  From "Understanding
Large Files" in InfoExplorer:

  Large File Geometry

  In file systems enabled for large files, file data stored before
  the 4MB file offset is allocated in 4096 byte blocks. File data
  stored beyond the 4MB file offset is allocated with large disk
  blocks of 128 KB in size. The large disk blocks are actually 32
  contiguous 4096 byte blocks.
  [...]

  Free Space Fragmentation

  Large disk blocks require 32 contiguous 4KB blocks. If you write to
  large files beyond the 4MB, file offset will fail with ENOSPC if the
  file system does not contain 32 unused contiguous 4KB blocks.

This seems to describe *exactly* what I'm seeing.  There's no problem
writing the initial 4MB of the file, since there are plently of 4KB
blocks in the FS.  However, the FS is apparently fragmented enough
(given the way HSM works, leaving little 4KB stub files all over the
FS in place of the complete files, I wouldn't be surprised at high
fragmentation; or rather, not so much fragmentation of files--after
all, the complete stub fits in one fragment--but files scattered
discontiguously all over the FS) that there's no contiguous 128KB
block.  If this is the case, then on a normal filesystem, my process
would get "filesystem full" (ENOSPC) after writing the initial 4MB
even though the FS is not at 100%; however, since we're using HSM, the
FSM driver intercepts the ENOSPC, tries to free space, and blocks my
process in the mean time.  Hence, my jobs appear to hang after writing
the initial 4MB!

The fix, according to IE, is to run defragfs.  Anyone care to
speculate what happens when you run defragfs on an HSM filesystem?
Should I run it on the mounted FSM, or unmount the FSM first?  The
other option would be to recreate the FS w/o large file support, not
something I'm looking forward to... ;)

The filesystem is back down to only 40% full and functioning.
'defragfs -r' currently reports:

  prompt> defragfs -r /dev/hsm
  statistics before running defragfs:
  number of free fragments 38760250
  number of allocated fragments 23367878
  number of free spaces shorter than a block 1158571
  number of free fragments in short free spaces 13169740

(it hasn't returned the "after defrag" stats yet).

> Do you see anything interesting in the syslog or
> equivalent file?

Nothing in syslog or errlog, or any other log.

John

-------------------------------------------------------------------------
John Valdes                        Department of Astronomy & Astrophysics
John Valdes                        Department of Astronomy & Astrophysics
j-valdes AT uchicago DOT edu                               University of Chicago