ADSM-L

Re: Metrics for a "large" filesystem?

2005-04-18 16:56:41
Subject: Re: Metrics for a "large" filesystem?
From: Thomas Denier <Thomas.Denier AT MAIL.TJU DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Mon, 18 Apr 2005 16:56:22 -0400
> So, when do you folks get to figuring that a given filespace is unwieldy
from
> a backup perspective?  I've got several which are ~10M files which I'm
> seriously considering chopping up (virtualmountpoint-wise) just because they
> are a pain to manage.
>
> I watched a dsmc process grow to 960 MB in core downloading the database for
> one of these, and then watching the log scroll ... stutter ... by was really
> painful.
>
> .. so what sounds "big" to you, and in what units do you think of it?

I tend to think of anything over three million files as big, but that
is based on experience with just two cases. We have a client with a bit
over three million files that was chronically troublesome until it went
through a major hardware upgrade. We have a client with over nine
million files that remains chronically troublesome.

Unfortunately, neither of these is a good prospect for the
virtualmountpoint option. In the first case, nearly all of the files
are descendents of a single directory. This directory has several
hundred subdirectories. The application is designed so that each of
these has about the same number of descendents. In the second case,
nearly all of the files are descendents of a single directory with
a few thousand subdirectories. The number of descendents per
subdirectory varies widely, but no one subdirectory accounts for even
one percent of the total file population.

You mention the pain of watching the log scroll. The 'quiet' option,
which eliminates most log output, can result in a significant
performance improvement for clients with large numbers of files.