#What port speed should we upgrade to for customer data interfaces?

1 messages · Page 1 of 1 (latest)

short cosmos
#

The customer LIF's are all made up by a LACP group on each node in the 8-nodes cluster. Currently LACP is bound by : 2 x A700(4x10G), 2 x 700s(2x40G), 2 x A400(4x10G), and 2 x FAS8200(2x10G). The cluster switches are 2 x cisco 3132Q-V. With the exception of 4x10G for cluster ports on two FAS8200's, all the oters are 2x40G. 2 FAS8200's already have enough 10/25 G cards installed.

We can upgrade those ports to either 25/40/100G based on network switches. Currently, there are no obvious performance issues on networking throughputs, but, we wanted to upgrade them anyway.

What is your recommedation on what speed we should upgrade to?

karmic lagoon
#

I'd start with the 2x10G and 4x10G first, because those are the slowest and might profit the most from being upgraded (e.g. to 2x40G). That being said, I doubt you can ever close to saturate the current LAGs on those systems, as even 2x10G would allow 2 gigabytes per second to pass through, and I highly doubt the 8200 can saturate that.

So update at your discretion, depending on how many ports are available on the switches (or are feasible to buy/install) and don't worry too much about the perceived performance benefits

short cosmos
#

Sorry, I made a typo. 2 x FAS8200 IS 4x 10G, not 2x10G.
The switch has enough ports for either 25G, 40G, or 100G. Some are arguing we sometime hit the bottleneck. But, no matter of if we close to saturate or not, the management wanted to upgrade anyway

carmine pier
#

Um, point of information: if you actually have FAS8200 controllers they only support:
Onboard 10G Ethernet and/or up to 16G FC (last “big” platform to support cna)
X1144 -> 2x40g or 4x10g breakout
X1117 -> 4x10g and/or up to 4x16g fc
X1120 -> 2x10g Base T
X1143 -> 2x10g or 2x16g fc
There are more fc and 1g adapters. There is no 25g

short cosmos
#

@carmine pier You are right. I had a typo.
A400 has 2 X dual 10G/25G cards(working as 10G for now), and 1 x dual 40G/100g(unused)
Should I upgrade them all to 40G or 25G?

carmine pier
#

Look at clients. Remember, a single client is no faster than a single link.
Depending on switch port density, I would opt for 4x25g do possibly better distribution. You can always upgrade/add/remove ports later on

primal socket
#

10GE -> 25GE is most often the cheapest path. No changes in form factor and most newer cisco switches support both speeds with just a swap of sfp

karmic lagoon
primal socket
carmine pier
short cosmos
#

So, If I need to upgrade all ports in LACP for customer data to4 x 25G across 8 nodes-cluster, as you all recommended, I then need to do the following based on current configurations:
2 x A700 HA --> Add 4 x 25G card on each node.
2 x A700s HA --> already have 2 x 40G , leave as is.
2 x A400 HA --> since there are already 2 x dual 10/25G cards on each node, just need to enable 4x25G instead of 4x10G.
2 x FAS8200 HA --> ??? According to @carmine pier , I cannot use 25G, so, the only option is to use 2x40G.

Correct?

carmine pier
# short cosmos So, If I need to upgrade all ports in LACP for customer data to4 x 25G across 8 ...

Additionally, on the fas8200/a300/a700s platforms, the 2x40g has a maximum throughput of 40g combined (both ports total, due to ASIC design of card). If you want greater throughput use two cards and only one port on each. If that is not possible just understand the point above. You could maintain 4x10g and still get the 40g combined possible throughout. With 4x10g you might achieve better client distribution. Might.

carmine pier
short cosmos
#

From "switch density" perspective, or to save numbers of switch ports being used, shouldn't we use 2 x 40G (seperated cards) better than 4x25G?

additionaly, you have discussed about "a single client is no faster than a single link", and "better client distribution"..
Pariticularly, if a client communicates with 4x10G LACP group on the filer, Is the the connection going to aross over all 4 links or just over 1 out 4 links?

Appreciate it if you can please explain those to me.

primal socket
#

well, 100/40GE ports and 10/25GE ports aren't the same form-factor on switches, so it depends a bit on what you have for leaf switches. I imagine the 100/40GE switches are a bit more expensive as well, but you can split up (break-out) 100/40GE ports.
But basically, you should just track your growth. The bump to 25GE is cheap and gets you double the bandwidth per port (where possible, of course) for the cost of a pair of sfp's per port (one controller, one switch)

#

LACP can use a few different algorithms to balance traffic over "bundled" interfaces, but just google LACP and you'll find that easily enough

short cosmos
#

I couldn't find the answer to this specific question, and really interested in it if somebody can please help:

If a client communicates with a LIF built on 4x10G LACP group on the filer, Is the the connection going to across over all 4 links or just go over 1 out 4 links?

neat plume
#

The switch has load balancing that will distribute the incoming traffic to the IFGRP, likely to a single port, but it could be different depending on the type of load balancing used. From ONTAP, each IFGRP has a Distribution Function that sets the load balancing type for outbound traffic. For a single client connection, you are usually only using one of the physical ports.

short cosmos
#

If I understand you correctly, a client no matter how many NFS mounts it has, all these incoming NFS traffics will go over one physical link within IFGRP, only if there are a number of clients which is the case in the real world, then the other 3 links will be used?

karmic lagoon
#

yes

#

and if you happen to have your clients' IP addresses selected badly (say, they are spaced 4 apart, like 10.0.0.1, 10.0.0.5, 10.0.0.9 etc) then they will all end up on the same link 😉

primal socket
#

you could theoretically have traffic coming in on one link and out on another but connections stick to one link (theoretically in each direction) placement

carmine pier
#

On the Netapp. The recommended distribution method is port. However the default in the GUI is IP.

final citrus
#

What speed are your clients?

#

If clients are 10GbE or even gigabit, would it make sense to upgrade? Are you running into network limiations now?

#

Unless you have an A/C700/800/900 you won't hit way more than 10GbE across multiple ports.

#

If you have differing speeds make sure the switch isn't shallow buffered otherwise you will drop packets and the speed improvement won't do anything.

short cosmos
karmic lagoon
#

There can only be one X1151A in the A400. The RoCE card (X1148A) can be installed in slot 4

#

what do you find unclear in HWU? because the X1148 is listed twice?

carmine pier
#

Look at hwu. There can only be one x1151 card on the a400. It’s a dual purpose card for cluster network and some efficiency offload.

The a400 only supports the x1148 for 40/100g Ethernet. Without looking, the rules are something like this:

If you need storage ports, you can use the x1148 in slot 5 and the onboard ports. Up to a total of 4 storage ports. If I have that config, I usually make e0c and e5b the pair for the first shelf and then e5a and e0d for networking. This requires flipping the port personality between storage/network as necessary.

Cards in slots 1 and/or 4 and/or 5 can be any cards.

If I recall without looking, I believe the slot 2 on this platform can only accommodate 10g base cards. The x1151 gets hot and can assist in damaging another card that gets hot ( like a sas card or a 100g card)

short cosmos
#

No, slot 3 on A400 can only use 2 x 1151A, that has already been used for Cluster Swtitches here/

I am planning on adding 4 x 40G for customer interfaces. Just wanted to confirm I would then have to use 2X1148A(2ports card) on slot 4? Will I be getting 2x40G or 4x40G in this case because 2 ports card can only used for 40G throughput?

carmine pier
#

Sorry. Like I said without looking 😁. I just updated the post to be correct.

#

The x1148 is a 2 port card. For each card you install in each node you’ll get 2 40/100g ports. This platform the card runs full speed (not like the 8200 where you get a total of 40g between both ports!)

short cosmos
#

Thanks for the message.

What about X91440A cards, 40G for A700, will they run full speed 4x40G as well?

karmic lagoon
#

X91440 is a 2-port card, not 4-port. but yes, it will run full speed on the A700

short cosmos
#

On A700s, for 2 ports 40G card, I can only get 40G, is this due to this particular platform?

carmine pier
#

The A700s suffers the same fate as the fas 8200/a300. It’s the same card. The card operates total at 40g. The only platforms that get full bandwidth with the correct card: a700/a900/fas8300/fas8700/fas9000/fas9500/x800/x400/x250

short cosmos
#

Acoording to the networking team here, 40Gbase-SR requires 8 strands of fiber and MPO connectors, upgrading all nodes to 40G would have a cost impact. 25G interfaces are only 2 strands of fiber, which is his recommendation.

I don't quite understand about strands or MPO. What would you say about his comment?

The reason we wanted to go for 40G across the board is because A700 and FAS8200 don't support 25G, and A700s is already at 40G. The other pair is A400.

karmic lagoon
#

the number of fibers should not matter unless you need to cross rooms or sites through a patch panel... usually you just connect the MPO cable to the NetApp and the other side of it to the switch, whereever that is (it's usually in the same room or even rack)

short cosmos
#

What exact extra work he needs to do in terms 40G/8 strands versus 25G/2 strands? As for the cost as his second point, will the prices be a lot different between 2x40G and 4x25G? We do have patch panel between NetApp’s cluster and networking switches although they are in the same datacenter room

carmine pier
#

I’ve heard though the grapevine there is a feature policy variation request (fPVR) that allows the use of the BiDi transceiver in the Netapp. That is an LC connector (2 strands). You would need a supported BiDI transceiver on the switch side also

#

You most certainly would need to work with your local sales team to get that approved for a supported configuration.

#

Otherwise, look at it this way:
How far from the switch is the Netapp? How hard would it be to run a handful of new MPO fibers from the switch to the Netapp?

#

Sounds like the network team being lazy

short cosmos
#

I was told that the cost on the card and transceiver for 100G on A700 is only half for 40G. Either one should be good for us in terms of performance. I see no reasons not go for 100G, what would you say?

vestal canopy
#

If they are willing to pay the cost, I would go for 100G just simply because it prevents another future purchase/rework if you decide to move that route at a later time, or transfer the kit to new gear if possible down the road.
but that's just me and my experience with purchasing issues

short cosmos
#

If we go with 100G on A700, we will then have mixed 25G/40G/100G for data links across 8 nodes cluster...

vestal canopy
#

that shouldn't be a problem unless you are linking them directly to each other and the ports don't match the lowest speed for some reason