LAN Free Throughput

[tapley]

Can anyone tell me the expected throughput of LAN-Free?

Here is my setup;

TSM (5.4.1.x) server run on AIX
Storage agent Windows 32bit. (Try 64bit with no improvement) 2 HBAs disk 2 HBAs tape. This is also the BAClient.
4 Gb SAN
4 LTO 4 Drives
Backend storage NSeries 7800 (112) 300GB FC Drives
Data approx 3TB (this is a FlexClone of Exchange 2007 data) data files are 250GB some log files of 1MB.

The throughput I am getting is 90MB per second writing to 4 LTO4 drives.

Dsm.opt file Windows Client;

enablelanfree yes
lanfreecommmethod sharedmem
lanfreeshmport 1502
resourceutilization 8
TXNBytelimit 2097152
DISKBuffsize (tried many settings currently not using according to the Tuning Guide.)

Dsmsta.opt file Windows

DEVCONFIG divconfig.txt
TCPPORT 1502
LANFREECommmethod SHAREdmem
LANFREEShmport 1502
RESOURCETIMEOUT 120

 

[hogmaster]

90Mb /s sounds very good, is it the total perforamnce or per LTO4 drive?

/regards

 

[tapley]

That was the total for all (4) LTO4 drives.

 

[dhaue]

90 MB/sec total does not sound good to me.

If you have 2 x 2GB HBA's and you had your drives balanced (2 using each HBA), your HBA capacity would be approximately 320 MB/sec, and your gross LTO-4 capability would be 960 MB/sec... meaning the HBA to the tape fabric would be the chokepoint (320 MB/sec across the 4 drives meaning would probably do as good with 2 drives)... unless the data access from the data fabric is slowing things down (busy array or disk contention) or the OS itself is too busy to satisfy.

I've never actually used LTO-4, but have seen near spec numbers for LTO-3 if you can get to the data fast enough and have fast enough CPU's.

 

[tapley]

I have been working with a netapp engineer, and he thinks that the data was fragmented on the netapp. This would slow down the total throughput of the netapp.

The data was created with the Jet Stress tool. We have now deleted that data, and have put the servers in production. It will take a while for the data to grow to the size of the test data, and hopefully the netapp can keep up with the fragmentation.

 

[johansen]

I want to come back to this topic as we are facing a terrible througput within our lanfree backups.

setup:
- 5 tsm servers (lpars) on a p560 running AIX6.1
- storage agent windows 2008 64bit / 1x 4GB dual port emulex HBA for tapes
- 4GB SAN
- 3584 tape library with 30x LTO-3 tapes
- each storage agent has about 10 tapes for usage (some may be blocked by other processes but that was never the problem)

client setup / dsm.opt baclient:

COMMMethod TCPip
TCPWindowsize 63
TCPBuffSize 32
ENABLELANFREE YES
LANFREECOMMMETHOD TCPIP
LANFREETCPPORT 1500
LANFREETCPSERVERADDRESS ******
TXNBYTELIMIT 2097152


client setup / dsm.opt tdpexchange:

COMMMethod TCPip
TCPWindowsize 63
TCPBuffSize 32
RESOURCEUTILIZATION 5
ENABLELANFREE YES
LANFREECOMMMETHOD TCPIP
LANFREETCPPORT 1500
LANFREETCPSERVERADDRESS *****
TXNBYTELIMIT 2097152

client setup / dsmsta.opt tdpexchange:

DEVCONFIG "D:\Program Files\Tivoli\Tsm\storageagent\devconfig.txt"
SERVERNAME TSM
TCPPORT 1500
COMMMETHOD TCPIP
TxnGroupMax 4096
MoveBatchSize 1000
SANDISCOVERY ON
IDLETimeout 400
COMMTimeout 900



- exchange 2007 DBs: around 100GB
- exchange 2007 LOGs: around 1GB

- We are running VSS backups from the passive CCR node
- 4 SGs in parallel at maxium per node (maximum 4 cluser nodes backing up the same time)

The problem is that the troughput is at around 15-20MB/sec and the backup is taking about 12-18 hrs per day!!!
We have the same problem when running legacy backup from the active CCR node...

Also we tested a DD880 (datadomain) with 800GB tapes (VTL) but the troughput was about 25MB/sec for a single backup which is also not the best...


does anyone has any suggestions what the problem might be?
thanks!!

 

[bphealy]

Hi
LAN free backups will only provide a substantial gain over GB network where you are backing up 500GB plus chunks of data from what I have seen. Your log files could be the issue as 1mb files will take ages to backup LAN free. Have you tried without LANfree to validate the issue is there ? I would also try a flat file backup of a 500Gb file LANfree to see if the TDP is the probelm. I have noticed the v6 TDP for exchange seems to have a lower throughput that v5.

 

[johansen]

hi,
well, I read many articles about lanfree backup and people are complaining about a aggregate transfer rate of 500MB. I guess that a troughput of 15MB is not really that much so something seems to be not ok in our environment.
I will try to copy a huge file directly to a SAN attached disc from a server with san connectivity and let you know about my findings.

 

[johansen]

well, have made some analyses and we could see that the storage backend (where we have hosted the exchange dbs & logs) seems to be the bottleneck.

storage for exchange was designed for best IO what means a degration of the bandwidth.

setup:
- lanfree with baclient (6.2.2) to tape (LTO3)
first try:
copy /B testfile NUL (40MB/sec)
2nd try:
copy /B testfile NUL (120MB/sec)

=> that means that the file is still in cache on the symmetrix. meaning that the symmetrix cannot provide enough bandwidth when reading from disks as they are still having a lot IO (also during the night).

what we could do is to create a BCV from the echange dbs and logs and backup these via another server, but the problem is that we need the storage capacity on the storage system (about 10TB)...

maybe /offloaded backup will help? anyone has another idea?

 

[matt.kaleta]

Greetings from Poland,

I'm now by implementing Lanfree in our company and spotted this thread. So here's my first experience.

Setup:
- TSM v 6.2 running on PS700 Power7 Blade, AIX6.1
- 2 x DS4700 + 2 x DS4400 + 1xDS3400 storage backend
- IBM SVC Storage Virtualizer (1 cluster, 4 nodes)
- TS3310 Tape library, 5xLTO4 SCSI + 5x LTO4 FC drives
- x4650 Exchange server running Exchange 2003, 2x Qlogic 4Gbit HBAs. StorageAgent running: v6.1, TDP: v.5.5

All is wired thru 4/8Gbit SAN (2 Fabrics, 5 switches each)
Amount of data - about 200 GB (3 Exchange SGs)

Throughput - about 130 Megabytes / sec writing to single LTO4 drive ...

and, regarding smaller files. Here's log from backup of DB2's transaction logs using legacy v6.2 client with maxnump=3:

Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010716.LOG  ** Unsuccessful **
ANS1114I Waiting for mount of offline media.
Retry # 1  Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010716.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010717.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010718.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010719.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010720.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010721.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010722.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010723.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010724.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010725.LOG [Sent]
Normal File-->        26,685,440 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010726.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010727.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010728.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010729.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010730.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010731.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010732.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010733.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010734.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010735.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010736.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010737.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010738.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010739.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010740.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010741.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010742.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010743.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010744.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010745.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010746.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010747.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010748.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010749.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010750.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010751.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010752.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010753.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010754.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010755.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010756.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010757.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010758.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010759.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010760.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010761.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010762.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010763.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010764.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010765.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010766.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010767.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010768.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010769.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010770.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010771.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010772.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010773.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010774.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010775.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010776.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010777.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010778.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010779.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010780.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010781.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010782.LOG [Sent]
Normal File-->        32,776,192 /ap/fh/fhdba04/fh0435ac/sqlog/NODE0000/S0010783.LOG [Sent]
Archive processing of '/ap/fh/fhdba04/journ/log/tsmlog.closed' finished without failure.


Total number of objects inspected:       69
Total number of objects archived:        69
Total number of objects updated:          0
Total number of objects rebound:          0
Total number of objects deleted:          0
Total number of objects expired:          0
Total number of objects failed:           0
Total number of bytes inspected:      2.10 GB
Total number of bytes transferred:    2.10 GB
LanFree data bytes:                    2.07 GB
Data transfer time:                    1.71 sec
Network data transfer rate:        1,283,002.23 KB/sec
Aggregate data transfer rate:      15,009.35 KB/sec
Objects compressed by:                    0%
Total data reduction ratio:            0.00%
Elapsed processing time:           00:02:26

 

[sipocz]

lanfreecommmethod - I would use sharedmem.

We first got the backups to work over the LAN. With 1 Gb/s NICs, we got around 100 MB/s. Once that was working, we changed the solution to use LAN-free. Note that LAN-free has better performance that LAN only for large files.
http://publib.boulder.ibm.com/infoc...=/com.ibm.itsmm.doc/b_perf_tuning_guide38.htm