...
>Server: AIX 4.3.3, TSM 5.1.5
>Client: AIX 5.1, TSM 5.1.5.4
>Virtualnodename client: AIX 5.1 TSM 5.1.5.4
>
>collocation is turned off. compression is off. maxnummp for both nodes is
>set to 6. we are using a 3494 tape library with 12 3590 tape drives. node
>disk storage is on an IBM ESS. the backup/restore LAN is gigabit. system 'A'
>was backed up to tape from 1 mount point. i am attempting to restore to
>system 'B' over 4 mount points. all filesystems are of type jfs2. nothing
>else is running on the TSM Server.
...
>since the restore is going to 4 separate mount points, i started four
>separate restores, broken down by destination mount point. The restores
>appear to begin running fine. Then at some seemingly arbitrary point, a
>session will suddenly slow to a crawl while in the middle of restoring a
>file... it will maybe get less than 1MB every few minutes. Eventually, all
>the sessions get to this point with files growing ever so slowly. The only
>way to "jump start" it is to cancel a session and restart the restore, which
>"usually" runs fine again for awhile until eventually slowing down to a
>crawl again.

I suspect that the multiple command sessions are encountering contention in the
tape storage pool.  Lack of collocation has probably widened the distance
between files being restored, both within a tape volume and across tape volumes:
there may be a substantial number of tapes involved in the restoral, depending
upon how long it took the file system collections to grow, and so one session
may be waiting for another to free the tape it's currently using before it can
seek to file position and retrieve the file's data.

I would use Query SEssion on the server to watch what's really going on, as well
as examine the Activity Log after the fact.

Non-collocation is something that facilitates backups and maximum use of tapes:
it is not something you want to do if restoral time matters.  In your case, I
would superficially think that collocation by filespace would be optimal.

If this were a single file system, you could, at your client-server level,
exploit Multi-Session Restore to good advantage.  With multiple command
sessions, Multi-Session Restore may be hurting in attempting too much
parallelism, depending upon how well the server is dealing with it.

  Richard Sims, BU