Restore extremely slow via Tivoli Storage Manager for SAN

[hungng89]

Hi Everyone

I have faced the issue when trying restore any kind of file ( single/multi/big/small ) via TSM for SAN . The transfer rate merely 150kb/sec . But backup was fast ( 4 GB / minute)

From analysis we found that the comm_wait is so high .
There is no error on dsm error log or dsmsta log .

I have updated to TSM Server 6.1.4.0 on Windows Server 2003 - 32 bit . I already Tried to reinstall the Storage Agent .
Re-create Storage Pool .

I wonder from your experiences what may be the root issue on this case , does any one give me some advices/ tips .
dsmserver.opt

COMMmethod HTTP
HTTPPort 1580
COMMmethod TCPIP
TCPPort 1500

TCPWindowsize 128
TCPNODELAY Yes
TCPADMINPort 1500

ADMINONClientPort Yes
COMMmethod NAMEDPIPE
NAMEdpipename \\.\pipe\Server1
NPBUFFERSIZE 8
SECUREPipes No
ADSMGROUPname adsmserver
NPAUDITSuccess No
NPAUDITFailure No

MSGINTerval 1
MAXSessions 25

COMMTimeout 60
IDLETimeout 15


DATEformat 1

TIMEformat 1

NUMberformat 1

MESsageformat 1
LANGuage AMENG

EXPInterval 24
EXPQUiet Yes

VOLUMEHistory "volhist.out"

DEVCONFig "devcnfg.out"
RESTOREINTerval 1440
DISABLESCHEDS No

EVENTSERVER Yes
REQSYS Yes
ENABLE3590 Yes
3494SHARED Yes
ASSISTVCRRECovery Yes
QUERYAuth NONE
ADREGISTER No
ADUNREGISTER No
MIRRORLOGDirectory    E:\MIRRORLOG
ARCHFAILOVERLOGDirectory    E:\ARCHIVEMIRRORLOG
ACTIVELOGDirectory    D:\LOG
ARCHLOGDirectory    D:\ARCHIVELOG

ACTIVELOGSIZE 8192

TXNGROUPMAX 4096
MOVESIZETHRESH 2048

MOVEBATCHSIZE 1000

#haipt 29 June 2010
RESOURCETIMEOUT 5
IDLETIMEOUT 5 
DISPLAYLFINFO=YES

sandiscovery on

dsm.sys

                SErvername  tsm
                  COMMMethod         TCPip
                  TCPPort            1500
                  TCPServeraddress   10.50.10.78

                  NODENAME vthmtxerp1
                  PASSWORDACCESS generate
 enablelanfree yes
                LANFREECommmethod TCPIP 
                
LANFREETCPServeraddress  127.0.0.1
LANFREETCPPort 1502
                  
                  MANAGEDSERVICES webclient schedule
                  HTTPPort           1581

                  ERRORLOGNAME "/usr/tivoli/tsm/client/ba/bin/dsmerror.log"
                  ERRORLOGRETENTION 7

                  SCHEDLOGNAME "/usr/tivoli/tsm/client/ba/bin/dsmsched.log"
                  SCHEDLOGRETENTION 7

                  QUERYSCHEDPERIOD 2
                  SCHEDMODE POLLING

        *         INCLEXCL "/usr/tivoli/tsm/client/ba/bin/include_exclude.txt"

        TXNBYTELIMIT 2097152
                *  RESOURceutilization 5
                tcpwindowsize 128 
                  *DIRM MGMT
LARGECOMmbuffer NO

Thanks you

 

[bphealy]

Generally the delay I have seen with restores is the time to load the tape, is there an available drive, small files over a SAN is very slow etc. Have you tried the restore without LAN-Free ?

Generally restores should be as fast if not faster than backups

 

[chad_small]

Is the server able to get a LAN-Free connection for the restore or does it fail and drop to the LAN connection?
My experience has shown it to be a lack of collocating your data onsite. Is the data collocated?

 

[TonyB]

Another possibility is a routing problem...

You may find that your backups are coming in on one network interface (perhaps an etherchannel), with restores flowing out of another interface with lower bandwidth. With this type of problem you don't notice it when performing backups (only metadata type traffic flows out of the TSM server) but when you try to move bulk data from the TSM server its slooooow.

edit: doh, or it could be a lan-free backup/restore...ignore me, going senile.

 

[hungng89]

HI Chad_small

I think that what you have just mention about wouldn't happen when i try to restore a single file right ? .
When i try to restore a single file that i have just backed it up it still take a long time to restore ( transfer rate 150kb/s)

I using
<storageagentname> : q sess
and see that the data was sent via Storage Agent .

HI Bbhealy

The restore using LAN is fast and smoothly . So i think it's may not because of the Tape LIB

HI TonyB

We are working with the Network Team to verify and isolate traffic of TSM servers . Will get info back from them soon.

Thanks for your all great advices

 

[hogmaster]

Just to get more information:
put a start (*) in front of enablelanfree yes in dsm.sys and run a backup of a faile noting the transfer rate
once more for the backup, this backup will be over the lan but using the lanfree config.

 

[hungng89]

Hi everyone

We figured out that the problem is with the TCP/IP connection between client node and tsm storage agent ( 1 storage agent in 1 node )
I have reconfigured it to use sharedmem to by pass TCP/IP error and it now work great .
dsm.sys

enablelanfree yes
lanfreecommmethod sharedmem

dsmsta.opt

commmethod sharedmem
commmethod tcpip
tcpport 1502

Thanks for your all great advices

 

[hogmaster]

Well done!
Thanks for sharing the solution.

 

[oguri]

hi thanks for solution,
can you please explain the concept under this

 

[moon-buddy]

hi thanks for solution,
can you please explain the concept under this

The path did not use the network but was through memory (internal).

 

[oguri]

thanq very much,but in my case the storage agent is on same system , i am able to ping with local system then why client not communicting with storage agent?
And why the backup is going in lan free path?
Please explain