We recently ran into this problem on an SGI box with about 5 million files
in a 1TB file system.  Support indicated that there is a
memoryefficientbackup option, but I did not place the call.

This is from the manunal.

Configure Memory-Constrained Workstations to Run Incremental Backups
Incremental backup performance suffers if the workstation has a low amount
of memory available prior to starting the backup. If your workstation is not
memory constrained, specify the memoryefficientbackup No option in your
client options file dsm.opt to provide the best performance.

If your workstation is memory constrained, specify the memoryefficientbackup
Yes option in your options file. Specifying Yes reduces memory consumption
but increases backup time. When you specify Yes, Tivoli Storage Manager
backs up only one directory at a time. If performance remains poor, check
your communication buffer settings and the communication link between your
workstation and the server.

What support said was to backup the directories first using the -dirsonly
option.  Then the files.  Also, look at the memoryefficientbackup option.
This unfortunately is a dsm.opt option on the client so you have to change
it for all directories.  I have no idea how long this is going to extend the
backup.

However, this was a very large server, so a ulimit change may be what is
needed and let it suck up more memory.  We are still working on this
problem.

Like you said there is no documentation on how much memory is necessary to
backup such a large file system.