> On Tue, 9 Dec 2003 11:29:59 -0500
> Thomas Denier <Thomas.Denier AT MAIL.TJU DOT EDU> wrote:
>
> > We have been backing up a Linux client with about five million files on
> > it. The system administrator has told me that he recently updated the
> > include/exclude files to exclude a portion of /var that accounts for
>
> 5 million files in /var??? amazing :-)

The messages for an electronic mail system with several thousand users
reside in a subdirectory of /var. The messages undoubtedly should have
gone into a separate file system, even if the file system's mount point
were in /var.

> > most of the file. I have not so far been able to verify this be
> > inspecting the configuration files myself. Ever since the change, we
> > have been seeing strange and very troublesome behavior. The client
>
> did the admin use exclude or exclude.dir, the latter may save you a lot of
> directory traversal.

I have restored dsm.opt, dsm.sys, and the include/exclude file to a
system I have access to. The change date on the include/exclude file
lines up with the Linux administrator's statements. He used an
exclude.dir. The other two files had not changed in almost six
months. All three files look normal for our site, with the exception
of two anomalies that have no obvious connection to the current
problem. One anomaly is the presence of a nodename statement in dsm.sys.
The node name specified is the same as the host name. The second
anomaly is a schedmode of 'polling' rather than 'prompted'. This is
needed because there is a firewall between the client and the TSM
server.

> > will send about eight gigabytes of data to the server over the course
> > of about twelve hours. At that point data transfer will slow to a
> > crawl and all other sessions and processes will perform poorly until
> > the session is cancelled. We have been through three iterations so
> > far, transferring about 23 gigabytes from client to server (I stopped
> > the last one at about 7 gigabytes). About one gigabyte out of the 23
> > can be accounted for by data written to storage pools. I usually don't
> > get the various session statistics messages. In the one case where I
> > did, they reported 0 objects deleted and only 55 objects expired.
>
> Depending on the filesystem used, Linux can be very slow in listing large
> dirs. The default listing requires in memory sorting of the dir (TSM does
> this as well) and with very large dirs, with lots of files, you may suffer
> from horrific performance on the client just from listing. Most likely, a ls
> in that one dir will either crash or hang forever....

The file system is EXT2. As far as I know, no one directory contains an
outrageous number of files. There is one directory near the top of the
hierarchy with a subdirectory for each of the several thousand users.
Past experience helping the administrator set up restores shows tens
to hundreds of files in individual directories further down the
hierarchy.

> > I would expect a substantial amount of data movement from client to
> > server after excluding large numbers of files as the client told the
> > server to expire the files. However, this would presumably be
> > comparable to the amount of file status information sent from server
> > to client each night before the change. This was about a gigabyte.
>
> I would expect that your server has never had a decent back-up of that one
> dir, that TSM is still listing that dir since the admin didn't use
> exclude.dir and that you have very files to expire.

As far as we can tell, backups of /var had been finishing successfully
on most days (albeit later in the morning than we would have liked).

> > Our entire TSM database is only about 20 gigabytes. The client code
> > is at the 5.1.6.0 level. The Linux kernel is at the 2.4.20-24 level.
> > The TSM server is at the 5.1.7.2 level and runs under OS/390.
>
> 5 milion object would eat about 1.5 GB (give or take half a GB)
> expiration of these files must be noticable on your server.

It accounts for about half of the time we spend on expiration. Most
of the other half is due to another, slightly smaller mail server
run by a different part of our organization.