ADSM-L

Re: Slow backup performance

2005-10-18 09:45:03
Subject: Re: Slow backup performance
From: Troy Frank <Troy.Frank AT UWMF.WISC DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 18 Oct 2005 08:43:57 -0500
It looks like all your tsa/tcp modules are a little out of date.  If possible, 
I'd get nw65sp4a installed on it.  Alternately, you could just run the tsa5up18 
& tcp659j updates.  Your tsa.cfg file looks pretty standard, and I've never 
really had trouble with it unless "Cache Memory Threshold" is set over 20/25%.  
 
The Delayed Acknowledgement parameter can be set a couple different ways.  
Either a "set delayed acknowledgement = off/on" at the commandline, or through 
the remote manager portal for the server (click "set parameters", then 
"communications").  If you do it from the commandline, the setting won't stick 
after reboot unless you add it to the autoexec.ncf.
 
The other thing I noticed is that you've got resourceutilization set to 6 in 
dsm.opt.  Usually there's not much reason to set it any more than #tape 
drives+1.  I've experienced problems here trying to set it to anything more 
than 4. ymmv, as always.
 
If the problem is disk-system performance, you will want to verify that you're 
running the latest drivers for your raid/scsi adapter.  Also be sure to update 
the controller's firmware to the latest version.  Here's some netware settings 
to look at in Remote Manager under "set parameters"/"disk"....
 
Auto scan for devices ON
Auto load of CDM modules ON
Snapshot Allocate Block Count 5000
Sequential Elevator Depth 6000 (the default is way to low at 8)
Enable IO Handicap Attribute OFF
Mirrored Devices Are Out Of Sync Message Frequency 28
Remirror Block Size 1
Concurrent Remirror Requests 32
Ignore Partition Ownership OFF
Ignore Disk Geometry OFF
Multi-path Support OFF
Enable Hardware Write Back ON (Depending on your controller, you might want to 
experiment with this OFF)
Enable Disk Read After Write Verify OFF (make sure this is off)
Auto LFVMount ON
 
Troy Frank
Network Services
University of Wisconsin Medical Foundation
608.829.5384

>>> cameron_ambrose AT ENTERPRISESERVICES.COM DOT AU 10/18/2005 1:57:22 AM >>>

Hi Troy,

      Thanks for responding. Checked nic/switch both set to 100FD.
Currently the server is at Netware 6.5SP2.

            Not sure what tsaup?? we're runing but the following are the
revision numbers for the nlm's mentioned

            TSAFS.NLM         6.50.09
            SMDR.NLM          6.54.01
            TCP.NLM           6.57.03
            TCPIP.NLM         6.57.06

      The tsa.cfg file is thus, pretty close to the defaults used.

            Read Buffer Size:
            65536
            Read Threads Per Job:
            4
            Read Thread Allocation:
            100
            Read Ahead Throttle:
            2  
            Cache Memory Threshold:
            10
            Disable Cluster:
            no

      We've currently set  ResourceUtilization to 6

      Couldn't find where the "TCP Delayed Acknowlegement" setting is
applied? Is it a Novell OS option or a TSM server/client option?


Thanks in advance Cameron




             Troy Frank
             <Troy.Frank@UWMF.
             WISC.EDU>                                                  To
             Sent by: "ADSM:           ADSM-L AT VM.MARIST DOT EDU
             Dist Stor                                                  cc
             Manager"
             <[email protected]                                     Subject
             .EDU>                     Re: Slow backup performance


             12/10/2005 11:42
             PM


             Please respond to
             "ADSM: Dist Stor
                 Manager"
             <[email protected]
                   .EDU>






It would also be helpful to know what revision levels your server is at.
Do you have any 6.5 service packs applied?  Any of the TSA or TCP updates?
What settings are you using when you load tsafs (look in
sys:\etc\sms\tsa.cfg), and are you using "ResourceUtilization" or any other
performance setttings in your dsm.opt?  You might also want to check your
server's tcp settings to make sure that TCP Delayed Ackknowlegement is off.
As has been mentioned already, it's also very possible that there is
something wrong in the nic/switch setup.  There is no "always right" answer
as to how they should be configged.  Some nic/switch combos work best with
both hardcoded, some with both on auto, and some with one hardcoded/one on
auto.  I would also make sure you're running the newest version of the nic
driver.



Troy Frank
Network Services
University of Wisconsin Medical Foundation
608.829.5384

>>> cameron_ambrose AT ENTERPRISESERVICES.COM DOT AU 10/11/2005 6:30:04 AM >>>

Guys,

      Thanks for the info, I'll double check the network settings. Though I
have changed the COMMtimeout value from 180secs to 1800 secs as suggested
by Richard Simms, which has stopped the session restart error messages and
infact the backup seems to be error free.

       As for the speed, I'm  still leaning towards a tsm netware client/
Netware OS issue. I say this because last night I performed a test backup
on a 10gig volume (50% full)  that hadn't had any changes and it still took
10 hours to scan only 80000 odd files. The Client itself hovered around
87-89% utilisation. When this server was netware 6.0 it used to flatline on
100% when performing a backup with the same slow performance. We thought
once we'd upgraded to 6.5 with the additional memory required with 6.5,
that performance would improve. Unfortunately it hasn't yet. Has anyone
experienced this before?

Regards Cameron



             Richard van
             Denzel
             <RvanDenzel@SLTN.                                          To
             NL>                       ADSM-L AT VM.MARIST DOT EDU
             Sent by: "ADSM:                                            cc
             Dist Stor
             Manager"                                              Subject
             <[email protected]         Re: Slow backup performance
             .EDU>


             07/10/2005 07:25
             PM


             Please respond to
             "ADSM: Dist Stor
                 Manager"
             <[email protected]
                   .EDU>






Oops, I don't know if there is an iperf for Netware.

Do other clients have the same problems from the network the NW server
is on? Also you did not specify what kind of network is between server
and client(s) (100Mb/1000Mb)?

Met vriendelijke groet, With kind regards,
Richard van Denzel.


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Leigh Reed
Sent: vrijdag 7 oktober 2005 10:37
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Slow backup performance

If you have already checked this and it is very obvious to you, please
don't be offended, but it has been a very common occurrence throughout
the life of network backups.

Have you checked that your NIC is hard coded to 100MB Full Duplex and
your switch is also the same. You could also check if your switch is
showing any CRC errors. Lastly, try an FTP from client server to TSM
server to establish if the network is performing outside of TSM.

Again, if I'm teaching you to suck hen produce, apologies.

Leigh

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Cameron Ambrose
Sent: 07 October 2005 07:42
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L]

Hello,

      We're currently experiencing backup failures/extreme slowness on a
Netware 6.5 SP2  server

            TSM Client 5.3.0
            TSM Server 5.2.0
            TSA's up to date as recommended by Client Doco

      Client Logs
            10/05/2005 14:43:19 ANS1809W Session is lost; initializing
session reopen procedure.
            10/05/2005 14:43:19 ANS1809W Session is lost; initializing
session reopen procedure.
            10/05/2005 14:44:15 sessRecvVerb: Error -50 from call to
'readRtn'.
            10/05/2005 14:44:15 ANS1999E Incremental processing of
'SYS:'
stopped.

            10/05/2005 14:48:24 ANS1999E Incremental processing of
'INF:'
stopped.

            10/05/2005 14:48:25 ANS1017E Session rejected: TCP/IP
connection failure


      Server Logs

            10/05/05   14:39:53      ANR0481W Session 2177 for node
SERVERNAME (NetWare)
                              terminated - client did not respond within
180 seconds.
            10/05/05   14:41:10      ANR0406I Session 2178 started for
node
SERVERNAME
                              (NetWare) (Tcp/Ip xxx.xxx.xxx.xxx(19578)).




      Currently we have had a backup running for 24 hours and all that
has been transfered is 4.19 gig and 255,000 files scanned. Has anyone
else experienced similar issue's

      Any help on this would be appreciated
      Regards Cameron



Confidentiality Notice follows:

The information in this message (and the documents attached to it, if any)
is confidential and may be legally privileged. It is intended solely for
the addressee. Access to this message by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution
or any action taken, or omitted to be taken in reliance on it is
prohibited and may be unlawful. If you have received this message in
error, please delete all electronic copies of this message (and the
documents attached to it, if any), destroy any hard copies you may have
created and notify me immediately by replying to this email. Thank you.



Confidentiality Notice follows:

The information in this message (and the documents attached to it, if any)
is confidential and may be legally privileged. It is intended solely for
the addressee. Access to this message by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution
or any action taken, or omitted to be taken in reliance on it is
prohibited and may be unlawful. If you have received this message in
error, please delete all electronic copies of this message (and the
documents attached to it, if any), destroy any hard copies you may have
created and notify me immediately by replying to this email. Thank you.

<Prev in Thread] Current Thread [Next in Thread>