If I had to guess, I'd say you were runing out of BBCredits on the ISL
links between
the two datacenters due to the distance. BBCredits are basically buffers
on the link.
It takes time for a packet to travel the 30km. If you don't have enought
credits
to allow enought in-transit packets you loose thoughput. This is similar
to
the tcp window processing to keep a WAN link full.
Below is my writeup I did for our docs.
NOTE: For Brocade switches, the Extended Fabric feature which allows
allocating more credits to a link is a extra charge feature, although I
believe it's standard on DCX director switches.
Brocade Extended Fabric Feature
1) What is it?
Fibre Channel switches use Buffer-to-Buffer-Credits (BBC's) to control flow
control of packets. In different places you will see this refered to as
Buffer Credit, Frame Buffers, BB_Credit, and even some other names.
2) What does it do?
In a fibre channel switch, a reply packet is required for every sent packet
(I think). BBC's represent memory buffers. The number of them assigned to
a port dictates how many packets a switch can send that without receiving a
reply. Once this number of packets have been sent, the switch will NOT
send any more packets until it receives a reply. This prevents a
transmitting switch from overrunning the buffers of a receiving switch.
3) Why does it need changed for long distance ISL links?
Think of a packet as requiring a finite amount of time to travel the fibre
cable. A fibre channel frame/packet is approximately 2k in size. At 1Gbps
a frame is about 4km long. At 2Gbps the same frame is aboutt 2km long. A
10km ISL link running at 2Gbps would be able to hold 5 frames in flight.
A 50km ISL link at 2Gbps would be able to hold 25 frames. To NOT have
enough BBC's available to fill the cable results in decreased throughput.
Unfortunely, to be able to adjust the BBC's of a port you must PURCHASE an
extra feature for Brocade swithes: Extended Fabric. It comes as a license
which you enter to enable this feature.
4) How do you implement it?
Extended Fabric works on a PER PORT basis. For any switch port for which
you want to enable Extended Fabric, you enter the cmd:
portcfglongdistance <port> <distance_level> <init_flag> <distance>
<port> = switch port #
<distance_level> = a MODE for the port that sets the distance of the
ISL link
<init_flag> = port initialization setting. Should be "1" for FE.
<distance> = for MODE "LD", this is the max distance to configure
for.
Command used to configure Extended Fabric on Fab5:
portcfglongdistance 13 LD 1 50
13 = switch port 13 (or whichever one you are working on)
LD = Mode "LD" is AUTO - it COMPUTES the distance of the ISL link.
1 = just use it!
50 = For AUTO mode (LD mode), this sets the MAX distance it will use.
For the GO-to-ISOC links on the new DWDM, the LD mode set the distance to
17km.
To un-set a port from Extended Fabric (ie: turn it off), issue the cmd:
portcfglongdistance 13 L0
13 = switch port
L0 = default distance (no extended fabric)
5) Requirements
For Extended Fabric to work, you must . . . .
- Ports MUST be running at the same speed.
- Ports MUST be set to the same MODE, init flag, and max distance.
In other words, the ports MUST be setup and operating IDENTICALLY.
6) IMPORTANT CONSIDERATION
The extra credits that Extended Fabric allows you to allocate to a port ARE
NOT FREE. The are STOLEN from another port. IF a certain distance on a
Extended Fabric setup for a port requires enought credits that some other
ports is starved, the swith can/will DISABLE a the starved port. This
behavior is different between 3900 and 4100 class switches, but this is a
situation you MUST understand.
7) More Info
Brocase Fabric OS Reference Manual (command reference manual)
see cmds: portcfglongdistance
portcfgshow
portshow
Brocade Fabric OS Procedures Guide
see section: Extended Fabrics Procedures
See attached documents
8) Example
fab5_103:admin> switchshow
switchName: fab5_103
switchType: 32.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 103
switchId: fffc67
switchWwn: 10:00:00:05:1e:35:bc:8e
zoning: ON (fabric5)
switchBeacon: OFF
Area Port Media Speed State
==============================
0 0 id N4 Online E-Port 10:00:00:05:1e:35:c0:7e
"fab5_102" (downstream)(Trunk master)
1 1 id N4 Online E-Port (Trunk port, master is Port 0 )
2 2 id N2 Online F-Port 10:00:00:00:c9:4e:50:03
3 3 id N2 Online F-Port 10:00:00:00:c9:4e:4d:ec
4 4 id N4 Online F-Port 50:05:07:63:0f:4d:8b:01
5 5 id N4 Online F-Port 50:05:07:63:0f:4d:8b:03
6 6 id N4 Online F-Port 50:05:07:63:0f:4d:8b:05
7 7 id N4 Online F-Port 50:05:07:63:0f:4d:8b:07
8 8 id N2 Online F-Port 10:00:00:00:c9:4e:4f:d1
9 9 id N4 Online F-Port 50:05:07:63:0f:4d:8b:0c
10 10 id N4 No_Light
11 11 id N4 No_Light
12 12 id N4 Online F-Port 50:05:07:63:0f:4d:8b:09
13 13 id N4 No_Light
14 14 id N4 No_Light
15 15 -- N4 No_Module
16 16 id N1 Online L-Port 1 public
17 17 id N1 Online L-Port 1 public
18 18 id N1 Online L-Port 1 public
19 19 id N1 Online L-Port 1 public
20 20 id N1 Online L-Port 1 public
21 21 id N1 Online L-Port 1 public
22 22 id N1 Online L-Port 1 public
23 23 id N1 Online L-Port 1 public
24 24 id N2 Online LD E-Port (Trunk port, master is Port 28 )
25 25 id N4 No_Light
26 26 id N4 No_Light
27 27 id N2 Online F-Port 10:00:00:00:c9:3d:a1:3a
28 28 id N2 Online LD E-Port 10:00:00:60:69:90:04:8f
"fab5_101" (downstream)(Trunk master)
29 29 id N4 No_Light
30 30 id N4 No_Light
31 31 id N2 Online F-Port 10:00:00:00:c9:3d:a1:49
fab5_103:admin> portshow 28
portName:
portHealth: No License
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x490b PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE
LOGIN LED
portType: 10.0
POD Port: License available to enable the port
portState: 1 Online
portPhys: 6 In_Sync
portScn: 16 E_Port Trunk master port
port generation number: 126
portId: 671c00
portIfId: 43020018
portWwn: 20:1c:00:05:1e:35:bc:8e
portWwn of device(s) connected:
20:19:00:60:69:90:04:8f
Distance: auto (desired = 50 Km, actual = 17 Km)
portSpeed: N2Gbps
Interrupts: 0 Link_failure: 5 Frjt: 0
Unknown: 0 Loss_of_sync: 1922 Fbsy: 0
Lli: 928406 Loss_of_sig: 3822
Proc_rqrd: 445239 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 2
Overrun: 0 Lr_in: 32
Suspended: 0 Lr_out: 5
Parity_err: 0 Ols_in: 4
2_parity_err: 0 Ols_out: 16
CMI_bus_err: 0
fab5_103:admin> portcfgshow 28
Area Number: 28
Speed Level: AUTO
Trunk Port ON
Long Distance LD
VC Link Init ON
Desired Distance 50 Km
Locked L_Port OFF
Locked G_Port OFF
Disabled E_Port OFF
ISL R_RDY Mode OFF
RSCN Suppressed OFF
Persistent Disable OFF
NPIV capability ON
"Meuleman, Ruud"
<ruud.meuleman@CO
RUSGROUP.COM> To
Sent by: "ADSM: ADSM-L AT VM.MARIST DOT EDU
Dist Stor cc
Manager"
<[email protected] Subject
.EDU> Tapedrive 3592 E06 speed of
160MB/sec
02/10/2010 10:43
AM
Please respond to
"ADSM: Dist Stor
Manager"
<[email protected]
.EDU>
Hi,
In our TSM environment we use two locations that are about 30 km away
from each other. On both sides we use TSM servers and 3592 tapedrives.
We use SAN and DWDM between the two locations. Writing data from a TSM
server to a tapelibrary that are on the same location have the speed
according the technical specifications. A TSM server writing to the
other location has not half of the speed.
For example:
On one side we have 3592 E06 tapedrives, that can write with a speed of
160MB/sec, when the TSM server is on the same location. There is 1 SAN
switch between the server and tapelibrary.
A TSM server on the other location, that is also connected with that
library, get a speed of 60MB/sec. There is a SAN switch and a DWDM
switch on one location and also a DWDM switch and a SAN switch on the
other location between the server and tapelibrary.
The speed of the HBA's in the TSM servers is 2Gb/sec, the speed of the
ports on the SAN switches are 2 Gb/sec and 4 Gb/sec and the speed of the
DWDM swiches is 2,5 Gb/sec. 2Gb/sec is about 320MB/sec. So the
bottleneck is should be 320MB/sec. When there is no other traffic on
this network, the speed should be more than 60MB/sec. The speed is of
the tests is determined of data of the summary table of the TSM
datebase.
Can some explain why the speed is very slow? (only 60MB/sec instead of
160MB/sec). Does anyone know how one can determine the speed of DWDM
switches?
Thanks,
Ruud Meuleman
**********************************************************************
This transmission is confidential and must not be used or disclosed by
anyone other than the intended recipient.
Neither Tata Steel Europe Limited nor any of its subsidiaries can accept
any responsibility for any use or misuse of the transmission by anyone.
For address and company registration details of certain entities within the
Corus group of companies, please visit
http://www.corusgroup.com/entities
**********************************************************************
-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.
|
