Hi Richard,
The Extended Fabric feature is installed on our Brocade switches. This
is necessary for using the DWDM.
The ports of the Brocade switches are autonegotiate, mayby those have to
be fixed 2Gb/sec. Mayby someone knows for sure?
Now we have also a XIV storage, that have a speed of 140 MB/sec between
the two locations. Maybe in the application TSM or maybe in the OS I can
make modifications to improve the speed in the SAN? I know there are
parameters of packed sizes for TCP traffic.
Regards,
Ruud Meuleman
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Richard Rhodes
Sent: Wednesday, February 10, 2010 5:19 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Tapedrive 3592 E06 speed of 160MB/sec
If I had to guess, I'd say you were runing out of BBCredits on the ISL
links between the two datacenters due to the distance. BBCredits are
basically buffers on the link.
It takes time for a packet to travel the 30km. If you don't have
enought credits to allow enought in-transit packets you loose thoughput.
This is similar to the tcp window processing to keep a WAN link full.
Below is my writeup I did for our docs.
NOTE: For Brocade switches, the Extended Fabric feature which allows
allocating more credits to a link is a extra charge feature, although I
believe it's standard on DCX director switches.
Brocade Extended Fabric Feature
1) What is it?
Fibre Channel switches use Buffer-to-Buffer-Credits (BBC's) to control
flow control of packets. In different places you will see this refered
to as Buffer Credit, Frame Buffers, BB_Credit, and even some other
names.
2) What does it do?
In a fibre channel switch, a reply packet is required for every sent
packet (I think). BBC's represent memory buffers. The number of them
assigned to a port dictates how many packets a switch can send that
without receiving a reply. Once this number of packets have been sent,
the switch will NOT send any more packets until it receives a reply.
This prevents a transmitting switch from overrunning the buffers of a
receiving switch.
3) Why does it need changed for long distance ISL links?
Think of a packet as requiring a finite amount of time to travel the
fibre cable. A fibre channel frame/packet is approximately 2k in size.
At 1Gbps a frame is about 4km long. At 2Gbps the same frame is aboutt
2km long. A 10km ISL link running at 2Gbps would be able to hold 5
frames in flight.
A 50km ISL link at 2Gbps would be able to hold 25 frames. To NOT have
enough BBC's available to fill the cable results in decreased
throughput.
Unfortunely, to be able to adjust the BBC's of a port you must PURCHASE
an extra feature for Brocade swithes: Extended Fabric. It comes as a
license which you enter to enable this feature.
4) How do you implement it?
Extended Fabric works on a PER PORT basis. For any switch port for
which you want to enable Extended Fabric, you enter the cmd:
portcfglongdistance <port> <distance_level> <init_flag> <distance>
<port> = switch port #
<distance_level> = a MODE for the port that sets the distance of
the ISL link
<init_flag> = port initialization setting. Should be "1" for FE.
<distance> = for MODE "LD", this is the max distance to configure
for.
Command used to configure Extended Fabric on Fab5:
portcfglongdistance 13 LD 1 50
13 = switch port 13 (or whichever one you are working on)
LD = Mode "LD" is AUTO - it COMPUTES the distance of the ISL link.
1 = just use it!
50 = For AUTO mode (LD mode), this sets the MAX distance it will
use.
For the GO-to-ISOC links on the new DWDM, the LD mode set the distance
to 17km.
To un-set a port from Extended Fabric (ie: turn it off), issue the cmd:
portcfglongdistance 13 L0
13 = switch port
L0 = default distance (no extended fabric)
5) Requirements
For Extended Fabric to work, you must . . . .
- Ports MUST be running at the same speed.
- Ports MUST be set to the same MODE, init flag, and max distance.
In other words, the ports MUST be setup and operating IDENTICALLY.
6) IMPORTANT CONSIDERATION
The extra credits that Extended Fabric allows you to allocate to a port
ARE NOT FREE. The are STOLEN from another port. IF a certain distance
on a Extended Fabric setup for a port requires enought credits that some
other ports is starved, the swith can/will DISABLE a the starved port.
This behavior is different between 3900 and 4100 class switches, but
this is a situation you MUST understand.
7) More Info
Brocase Fabric OS Reference Manual (command reference manual)
see cmds: portcfglongdistance
portcfgshow
portshow
Brocade Fabric OS Procedures Guide
see section: Extended Fabrics Procedures
See attached documents
8) Example
fab5_103:admin> switchshow
switchName: fab5_103
switchType: 32.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 103
switchId: fffc67
switchWwn: 10:00:00:05:1e:35:bc:8e
zoning: ON (fabric5)
switchBeacon: OFF
Area Port Media Speed State
==============================
0 0 id N4 Online E-Port 10:00:00:05:1e:35:c0:7e
"fab5_102" (downstream)(Trunk master)
1 1 id N4 Online E-Port (Trunk port, master is Port 0 )
2 2 id N2 Online F-Port 10:00:00:00:c9:4e:50:03
3 3 id N2 Online F-Port 10:00:00:00:c9:4e:4d:ec
4 4 id N4 Online F-Port 50:05:07:63:0f:4d:8b:01
5 5 id N4 Online F-Port 50:05:07:63:0f:4d:8b:03
6 6 id N4 Online F-Port 50:05:07:63:0f:4d:8b:05
7 7 id N4 Online F-Port 50:05:07:63:0f:4d:8b:07
8 8 id N2 Online F-Port 10:00:00:00:c9:4e:4f:d1
9 9 id N4 Online F-Port 50:05:07:63:0f:4d:8b:0c
10 10 id N4 No_Light
11 11 id N4 No_Light
12 12 id N4 Online F-Port 50:05:07:63:0f:4d:8b:09
13 13 id N4 No_Light
14 14 id N4 No_Light
15 15 -- N4 No_Module
16 16 id N1 Online L-Port 1 public
17 17 id N1 Online L-Port 1 public
18 18 id N1 Online L-Port 1 public
19 19 id N1 Online L-Port 1 public
20 20 id N1 Online L-Port 1 public
21 21 id N1 Online L-Port 1 public
22 22 id N1 Online L-Port 1 public
23 23 id N1 Online L-Port 1 public
24 24 id N2 Online LD E-Port (Trunk port, master is Port 28
)
25 25 id N4 No_Light
26 26 id N4 No_Light
27 27 id N2 Online F-Port 10:00:00:00:c9:3d:a1:3a
28 28 id N2 Online LD E-Port 10:00:00:60:69:90:04:8f
"fab5_101" (downstream)(Trunk master)
29 29 id N4 No_Light
30 30 id N4 No_Light
31 31 id N2 Online F-Port 10:00:00:00:c9:3d:a1:49
fab5_103:admin> portshow 28
portName:
portHealth: No License
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x490b PRESENT ACTIVE E_PORT G_PORT U_PORT
LOGICAL_ONLINE
LOGIN LED
portType: 10.0
POD Port: License available to enable the port
portState: 1 Online
portPhys: 6 In_Sync
portScn: 16 E_Port Trunk master port
port generation number: 126
portId: 671c00
portIfId: 43020018
portWwn: 20:1c:00:05:1e:35:bc:8e
portWwn of device(s) connected:
20:19:00:60:69:90:04:8f
Distance: auto (desired = 50 Km, actual = 17 Km)
portSpeed: N2Gbps
Interrupts: 0 Link_failure: 5 Frjt: 0
Unknown: 0 Loss_of_sync: 1922 Fbsy: 0
Lli: 928406 Loss_of_sig: 3822
Proc_rqrd: 445239 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 2
Overrun: 0 Lr_in: 32
Suspended: 0 Lr_out: 5
Parity_err: 0 Ols_in: 4
2_parity_err: 0 Ols_out: 16
CMI_bus_err: 0
fab5_103:admin> portcfgshow 28
Area Number: 28
Speed Level: AUTO
Trunk Port ON
Long Distance LD
VC Link Init ON
Desired Distance 50 Km
Locked L_Port OFF
Locked G_Port OFF
Disabled E_Port OFF
ISL R_RDY Mode OFF
RSCN Suppressed OFF
Persistent Disable OFF
NPIV capability ON
"Meuleman, Ruud"
<ruud.meuleman@CO
RUSGROUP.COM>
To
Sent by: "ADSM: ADSM-L AT VM.MARIST DOT EDU
Dist Stor
cc
Manager"
<[email protected]
Subject
.EDU> Tapedrive 3592 E06 speed of
160MB/sec
02/10/2010 10:43
AM
Please respond to
"ADSM: Dist Stor
Manager"
<[email protected]
.EDU>
Hi,
In our TSM environment we use two locations that are about 30 km away
from each other. On both sides we use TSM servers and 3592 tapedrives.
We use SAN and DWDM between the two locations. Writing data from a TSM
server to a tapelibrary that are on the same location have the speed
according the technical specifications. A TSM server writing to the
other location has not half of the speed.
For example:
On one side we have 3592 E06 tapedrives, that can write with a speed of
160MB/sec, when the TSM server is on the same location. There is 1 SAN
switch between the server and tapelibrary.
A TSM server on the other location, that is also connected with that
library, get a speed of 60MB/sec. There is a SAN switch and a DWDM
switch on one location and also a DWDM switch and a SAN switch on the
other location between the server and tapelibrary.
The speed of the HBA's in the TSM servers is 2Gb/sec, the speed of the
ports on the SAN switches are 2 Gb/sec and 4 Gb/sec and the speed of the
DWDM swiches is 2,5 Gb/sec. 2Gb/sec is about 320MB/sec. So the
bottleneck is should be 320MB/sec. When there is no other traffic on
this network, the speed should be more than 60MB/sec. The speed is of
the tests is determined of data of the summary table of the TSM
datebase.
Can some explain why the speed is very slow? (only 60MB/sec instead of
160MB/sec). Does anyone know how one can determine the speed of DWDM
switches?
Thanks,
Ruud Meuleman
**********************************************************************
This transmission is confidential and must not be used or disclosed by
anyone other than the intended recipient.
Neither Tata Steel Europe Limited nor any of its subsidiaries can accept
any responsibility for any use or misuse of the transmission by anyone.
For address and company registration details of certain entities within
the Corus group of companies, please visit
http://www.corusgroup.com/entities
**********************************************************************
-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If the
reader of this message is not the intended recipient or an agent
responsible for delivering it to the intended recipient, you are hereby
notified that you have received this document in error and that any
review, dissemination, distribution, or copying of this message is
strictly prohibited. If you have received this communication in error,
please notify us immediately, and delete the original message.
**********************************************************************
This transmission is confidential and must not be used or disclosed by anyone
other than the intended recipient.
Neither Tata Steel Europe Limited nor any of its subsidiaries can accept any
responsibility for any use or misuse of the transmission by anyone.
For address and company registration details of certain entities within the
Corus group of companies, please visit
http://www.corusgroup.com/entities
**********************************************************************
|
