One of our AIX machines had a similiar (only in such that backup
performance was exceedingly slow) due to an undocumented vmtune parm that
set the read aheads to zero after a patch.  The instrument trace I think
will help tell you if you are spending alot of the time in file I/o (I
think).

Either way, if this is an Aix v5.3 machine you can access the filesystem
tuning options via smitty tuning (for most tuning stuff), it's worth
checking out. Also have you tried local backups of other data not on those
filesystems to see if all backups on the local system are slow, or only
specific filesystems.

I also assume you are not sharing FC adapters with both tape and disk?
Have tried changing the "Maximum Transfer Size" on the adapter (our FC
cards have this), it has helped to bump up our performance with our 3484
library (using Ultrium 2 drives)



Jonathan Kaufman








Andrew Raibeck <storman AT US.IBM DOT COM>
Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
03/31/2005 10:36 AM
Please respond to "ADSM: Dist Stor Manager"



        To:     ADSM-L AT VM.MARIST DOT EDU
        cc:
        Subject:        Re: Archive very, very slow



I doubt very much that tracing did anything to improve the performance. If
anything, it impedes performance. There is most likely something else
going on in the environment, though what I do not know.

The data transfer time of 323 seconds suggests that it took TSM 5 minutes
and 23 seconds to move the data, but the elapsed processing time of 55:13
minutes makes me wonder what was going on during the other 50 minutes.

Unless memory is suspect, I would not recommend turning on the MEMORY
trace flag since that generates a tremendous amount of data.

The SERVICE trace might show some large gaps between timestamps that might
lead to more info regarding the performance problem.

You can also get an instrumentation trace by adding the following option
when you invoke dsmc:

   -testflags=instrument:detail

This will create a file named dsminstr.report that will have
instrumentation info about the operation.

Other questions:

- While the operation is running, what does the administrative QUERY
SESSION command show for the client?

- Are there any error messages in the dsmerror.log file?

- What does your client options file look like?

- What does the client option set defined on the server for this client
look like?

- Are there any anomalous messages in the server activity log between the
time the client operations begins and ends?

Regards,

Andy

Andy Raibeck
IBM Software Group
Tivoli Storage Manager Client Development
Internal Notes e-mail: Andrew Raibeck/Tucson/IBM@IBMUS
Internet e-mail: storman AT us.ibm DOT com

The only dumb question is the one that goes unasked.
The command line is your friend.
"Good enough" is the enemy of excellence.

"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2005-03-30
14:32:43:

> I turned on tracing and ran an archive with the following command:
>
> dsmc archive -traceflag=service,memory
> -tracefile=/tmp/tsm/trace/dbext_trace.out -tracemax=5000 -verbose
> -desc="Old IDX Tape Backups for /restore/dbext - 2005Mar30"
> -filelist=/tmp/tsm/archive_list_dbext_2005Mar30 -archmc=AIX-1YEAR
>
> Here is the summary:
>
> ========================================================================
> ========
>
> IBM Tivoli Storage Manager
> Command Line Backup/Archive Client Interface - Version 5, Release 2,
> Level 3.0
> (c) Copyright by IBM Corporation and other(s) 1990, 2004. All Rights
> Reserved.
>
> Archive function invoked.
>
> Node Name: IDX_RESTORE
> Session established with server TSMNIM01: AIX-RS/6000
>   Server Version 5, Release 2, Level 4.0
>   Data compression forced off by the server
>   Server date/time: 03/30/05   14:16:48  Last access: 03/30/05
> 14:13:11
>
>
> Total number of objects inspected:      865
> Total number of objects archived:         8
> Total number of objects updated:          0
> Total number of objects rebound:          0
> Total number of objects deleted:          0
> Total number of objects expired:          0
> Total number of objects failed:           0
> Total number of bytes transferred:    98.15 GB
> Data transfer time:                  323.03 sec
> Network data transfer rate:        318,611.16 KB/sec
> Aggregate data transfer rate:      31,063.43 KB/sec
> Objects compressed by:                    0%
> Elapsed processing time:           00:55:13
>
> ========================================================================
> ========
>
> Can someone explain how tuning on tracing would give me increased
> transfer rate?
>
> This is not the same archive we have been having issues with, but this
> same one took almost 3 ours earlier this week.
>
> Gary Galloway
> ___________________________________________________________
> ___________________________________________________________
> Health Care Information Systems
> University of Iowa Hospitals & Clinics
> gallowayg AT healthcare.uiowa DOT edu
> Phone:  319.356.7036
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Ben Bullock
> Sent: Wednesday, March 30, 2005 2:01 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] Archive very, very slow
>
>    We also recently migrated from SCSI attached 3590E drives to SAN
> attached 3592 drives (both in a 3494 library).
>
>    Unfortunately, we only saw an increase in the performance with
> the new tape drives.
>
>    As Richard mentioned, it may be an issue with the mounting of
> the data on the host, but I will throw in a few things to perhaps check
> on the Tape drive configuration.
>
>    - Are you using the latest and greatest ATAPE and ATLDD drivers?
>     Mine are a little old, but you should at least be at these
> levels:
>    Atape.driver               9.0.7.0
>    atldd.driver               5.5.1.0
>
>    - Are the drives defined to TSM as a 3592 drives and not
> "Generic" or something?
>    Here is a little look at one of mine:
> tsm: TSMX>q drive f=d
>
>                                 Library Name: BOITAPELIB1
>                                   Drive Name: 3592DRV0
>                                  Device Type: 3592
>                                      On-Line: Yes
>                                 Read Formats: 3592C,3592
>                                Write Formats: 3592C,3592
>                                      Element:
>                                  Drive State: LOADED
>                                 Allocated to:
>                                          WWN:
>                                Serial Number: 000007XXXXXX5
>               Last Update by (administrator): BBULLOCK
>                        Last Update Date/Time: 02/10/05   12:17:54
> Cleaning Frequency (Gigabytes/ASNEEDED/NONE):
>
>    - On the SAN ports, are you sure that all the drives are linked
> up as 2GB devices?
>    - If you have multiple FC cards, have you enabled multipathing
> to take advantage of it?
>     (a nice feature to have, but probably not a contributer to your
> issues).
>
> Ben
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Miller, Ila
> Sent: Wednesday, March 30, 2005 12:15 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Archive very, very slow
>
> We recently migrated from a locally attached 3590 E11 tape drive to two
> 3592 tape drives in a 3494 tape library.  We installed TSM 5.2.4.0 on a
> p5 570 lpar running AIX 5.3.  Our data is on an IBM ESS 800.  Every day
> we quiese the application and flash copy it then mount the flash copy
> volume groups to another system and backup.  We were just doing the
> native backup to the 3590 E11 locally attached drives.  Then we mounted
> the flash copy to the TSM server and did an archive.  The four volume
> groups comprise about 400G of data 95% being in CACHE.DAT files, one
> which is about 200G in size.
>
> The problem is the archive over fibre is taking 12-14 hours to complete.
> We were backing up to the 3590 local attached tape drives using native
> commands in about 8 hours.  The Aggregate data transfer rate:
> 10,793.34 KB/sec seems network speed and we are using fibre attached
> tape drives and ESS storage.  The data is mounted to the TSM server so
> the TSM server is also the client and the data should not be going over
> the network at all.
>
> We have looked at ESS 800 performance of LUNs, software compression,
> various settings including shmport and commMethod sharedmem and have not
> been able to come up with anything to make the backup run faster.  We
> have a pmr open at IBM but it has not produced any useful information
> that would resolve the problem.
>
> Has anyone else ever seen anything like this?
>
>
> Ila Z. Miller
> ___________________________________________________________
> ___________________________________________________________
> Health Care Information Systems
> University of Iowa Hospitals & Clinics
> ila-miller AT uiowa DOT edu
> Phone:  319.356.0067
> FAX: 319.356.3521
>
> Notice: This e-mail (including attachments) is covered by the Electronic
> Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may
> be legally privileged.  If you are not the intended recipient, you are
> hereby notified that any retention, dissemination, distribution, or
> copying of this communication is strictly prohibited.  Please reply to
> the sender that you have received the message in error, then delete it.
> Thank you.