We've got a similar beast. The problem is the sheer number of files,
which the client must keep a list of, in order to decide which files
need to be backed up. This is the whole idea behind TSM's Progressive
Incremental backup model. We have found that as file counts reach the
many millions, backup (and restore!) performance degrades exponentially.

One fact you must face is that a filespace with 21,000,000 files almost
cannot be restored. We tried it with 18,000,000 and it counted and
sorted its list for about a day before restoring any files - slowly. The
restore would have taken many days when we stopped it to reconsider.
There is no restore equivalent to MEMORYEFFICIENTBACKUP YES or
MEMORYEFFICIENT DISKCACHEMETHOD for restore. The ability to restore is
the whole point of backup, so take restore issues seriously.

The only workable answer we have found, without drastically changing the
thing you are backing up, is Virtual Mount Points. The idea is to keep
the file count in each Virtual Mount Point low enough that the client
can work efficiently.

Roger Deschner      University of Illinois at Chicago     rogerd AT uic DOT edu
               Academic Computing & Communications Center
======I have not lost my mind -- it is backed up on tape somewhere.=====



On Mon, 6 Aug 2012, Arbogast, Warren K wrote:

>There is a Linux fileserver here that serves web content. It has 21 million 
>files in one filesystem named /ip. There are over 4,500 directories at the 
>second level of the filesystem. The server is running the 6.3.0.0 client, and 
>has 2 virtual cpus and 16 GB of RAM.  Resourceutilization is set to 10, and 
>currently there are six client sessions running.  I am looking for ways to 
>accelerate the backup of this server since currently it never ends.
>
>The filesystem is NFS mounted so a journal based backup won't work. Recently, 
>we added four proxy agents, and are splitting up the one big filesystem among 
>them using include/exclude statements. Here is one of the agent's 
>include/exclude files.
>
>exclude /ip/[g-z]*/.../*
>include /ip/[a-f]*/.../*
>
>__Since we added the proxies the proxy backups are copying many thousands of 
>files, as if this were the first backup of the server as a whole. Is that 
>expected behavior?
>
>__Recently, the TSM server database is growing faster than it usually does, 
>and I'm wondering whether there could be any correlation between the ultra 
>long running backup, many thousands of files copied, and the faster pace of 
>the database growth.
>
>__The four proxies haven't made a  big difference in the run time of the 
>backup. Could something else be done to speed it up?
>
>Thank you,
>Keith Arbogast
>Indiana University
>
>
>