Thomas,
I suspect the media waits are down to the three streams wanting the same
tapes
If you are really restoring 9 million separate objects then most likely it
is going to be the client end writing out the data where the majority
elapsed time is spent.
What do the session stats show for network data transfer rate?
I have not done tracing for a while, but when I did I always found
"Perform" tracing on the client to be most useful
The time is usually found to be spent mainly in
Transaction:- A general category to capture all time not accounted for
           in other sections. This category includes file open/close
           time and other miscellaneous processing on the client.
           File open/close processing can make total Transaction time
           a large part of elapsed time with smaller files

File I/O:- Requesting data to be read or written on the client file
         system. Each File I/O usually represents a 32K logical
         request (or the remaining data if less than 32K). File I/O
         may be entered one additional time at the end of the file.
         With compression on some smaller clients a file I/O can
         represent a request for less than 32K. A file I/O request
         may require multiple physical accesses.
         For small files on systems without read ahead, average
         file I/O time for backup is 15 to 40 ms dependent on the
         platform. For large files on system doing read ahead this can
         be significantly reduced. Slow response times from disks will
        contribute to the ammount of time logged here.

Data Verb :- consists of time spent in the network plus time spent on the
host server


.



Thomas Denier <Thomas.Denier AT mail.tju DOT edu>
Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu>
31/03/2005 22:32
Please respond to
"ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu>


To
ADSM-L AT vm.marist DOT edu
cc

Subject
Restore performance problem






We recently restored a large mail server. We restored about nine million
files with a total size of about ninety gigabytes. These were read from
nine 3490 K tapes. The node we were restoring is the only node using the
storage pool involved. We ran three parallel streams. The restore took
just over 24 hours.

The client is Intel Linux with 5.2.3.0 client code. The server is
mainframe
Linux with 5.2.2.0 server code.

'Query session' commands run during the restore showed the sessions in
'Run'
status most of the time. Accounting records reported the sessions in media
wait most of the time. We think most of this time was spent waiting for
movement of tape within a drive, not waiting for tape mounts.

Our analysis has so far turned up only two obvious problems: the
movebatchsize and movesizethreshold options were smaller than IBM
recommends. On the face of it, these options affect server housekeeping
operations rather than restores. Could these options have any sort of
indirect impact on restore performance? For example, one of my co-workers
speculated that the option values might be forcing migration to write
smaller blocks on tape, and that the restore performance might be
degraded by reading a larger number of blocks.

We are thinking of running a test restore with tracing enabled on the
client, the server, or both. Which trace classes are likely to be
informative without adding too much overhead? We are particularly
interested in information on the server side. The IBM documentation for
most of the server trace classes seems to be limited to the names of the
trace classes.



**********************************************************************
The information in this E-Mail is confidential and may be legally
privileged. It may not represent the views of Scottish and Southern
Energy Group.
It is intended solely for the addressees. Access to this E-Mail by
anyone else is unauthorised. If you are not the intended recipient,
any disclosure, copying, distribution or any action taken or omitted
to be taken in reliance on it, is prohibited and may be unlawful.
Any unauthorised recipient should advise the sender immediately of
the error in transmission. Unless specifically stated otherwise, this email (or 
any attachments to it) is not an offer capable of acceptance or acceptance of 
an offer and it does not form part of a binding contractual agreement.


Scottish Hydro-Electric, Southern Electric, SWALEC, S+S and SSE Power 
Distribution are trading names of the Scottish and Southern Energy Group.
**********************************************************************