> We use the cyrus IMAPD server, which stores mail in a structure grossly

We run cyrus IMAPD server too...

> Number of files per filesystem is really the value I'm trying to tune now.
> Since we have so many, it really appears that this value, and its impact on
> database transfers (for normal incrementals) and updates (as incrementals run)
> is becoming a significant issue.

At one time, we had all 8 million of our files in a single vxfs
filesystem, which I backed up via 12 virtualmountpoints.  We now have 12
vxfs filesystems, which I still backup as 12 dsmc cron jobs.  The main
reason we broke is up was because vxfs had a bug in it that showed it's
ugly head at 8 million files, and over-wrote it's own onide table, causing
us to use adsm to restore all 8 million files!  The best thruput I could
achieve was 40K/hour/adsm stream, running 4 at a time.  It appeared at the
time that the limiting factor was the ADSM Database.

Currently, our 12 filesystems range in size from 765,282 to 846,638 files,
almost all of that being small mail messages.

In July, we will be splitting this across two mailservers (Sun E4000's),
each with 10 filesystems.  Each mail system will be it's own ADSM Server
with a 3570-C02 (2 3570-C1A drives, 20 slots), which will be used for
dd/vxfdump output for disaster recovery.  We will also continue to do the
file level backups via our main ADSM server for use file restores.

> I was hoping someone had noticed something on the order of "We started seeing
> slowdown when we exceeded 200K files on a FS", or some such.

I can backup the 750k files in about 4 hours on the weekends, when the
IMAP server is somewhat quiet.  During the week, they take 12-16 hours.
Each filesystem downloads about 100M at startup time, which can take close
to an hour.

Steve (unVMix Systems Programmer/Dude) Roder
(tkssteve AT ubvm.cc.buffalo DOT edu | tkssteve AT acsu.buffalo DOT edu | 
(716)645-3564 ,
   | http://ubvm.cc.buffalo.edu/~tkssteve)