#Network Connections on C800?

1 messages · Page 1 of 1 (latest)

jaunty mulch
#

There are two different type of network adapters/cables as listed below in our quotes for C800. After checked with HWU, it is still not clear to me which one is for Host Connections and which one is for Cluster Switch connections. Can somebody please help me out?

NetApp Bare Cage - network adapter - PCIe - 100 Gigabit QSFP28 x 2
2xX1148A-N-C
NetApp Bare Cage - network adapter - PCIe - 40Gb Ethernet 100Gb Ethernet
2xX1146A-N-C

**NetApp 100GBase-CU direct attach cable - 1.6 ft **
X66211A-05-N-C
**NetApp network cable - 5 m **
X66200-5-N-C

Also, there are 4 x 10G onboard connections, does that mean we are not going to use those for host connections?

tough anvil
#

Typically i've seen the x1146 is used for host access. It will ship default with a x1146 in slot one.

#

cards in slots 2,3,4,5 are for client access.

#

There's no onboard host connects on the x800 (there are 2x100G ports for clusternet and HA

leaden siren
split crescent
#

Here it is. The cold hard truth
The x1146 MUST be used if you want to use optical transceivers.
You may use PASSIVE twinax in either card
The x1148 is the Mellanox card which runs hot. The system itself is hot. You cannot put optics in the x1148 as it gets too hot

Second, if you plan to use RDMA/NFS you must use the x1148/twinax. No choice on this platform

#

Also if you scroll down that list you should see x1146 listed for networking by itself

jaunty mulch
#

Is x1146 is used for cluster switch connections, and x1148 for customer connections?

There are 2 items for x1146, and each item has 2 under qty. The same list for 1148 as well. So, I think 2 cards for two different types on each node in HA

leaden siren
#

the usage is in the "category" column. e.g. for Cluster/HA traffic the card is only supported in slot 1, while it's supported for client access in other slot(s)

#

as TMAC said, you can use the X1148A only with DAC cables, not with optics, so your requirements in that area usually already determine which card you need

jaunty mulch
#

Since X1146 can be used for either customer or switch connections depending on which slot you use, then x1146A in slot 1 should be for the cluster switch. Is my understanding correct as listed below ?

card "X1148A" & cable "X66211A" for customer connections ? X66211A: 100GBase-CU direct attach cable
card "X1146A in slot 1" & cable "X66200" for cluster switches ? X66200: network cable

X66100 (NetApp 40GB 3m DAC Cable SFP+), Don't know which is this cable used for?

leaden siren
#

what do you mean "customer or switch connections"? those are not mutually exclusive (in most installations, all network connections go over a switch)

#

the x1146a card in slot 1 is always for the backend network, you cannot change that (neither the card type nor the slot nor what it's used for)

jaunty mulch
#

switch connection is referring to cluster switch connection; customer connection meant the client connection

leaden siren
#

the x66100 cable is probably for client (front end) networking. the 100g cards down-negotiate to 40g if a 40g cable is connected

#

so if your front-end switch only has 40g ports you need a 40g cable not a 100g cable (as the cables do not support multiple speeds)

#

usually you have to be careful with NetApp branded DAC cables as some switches might not accept them (in many Cisco switches for example you need to enable the "unsupported transceiver" switch)... usually it's safer to use optics for front-end connections

jaunty mulch
#

Sorry for being slow. X1146 in slot 1 should be for the cluster switch and X1148 for client connections, correct?

leaden siren
#

yes

split crescent
#

If you check the “cable” options, that should be in line with what I said.

split crescent
split crescent
grim garden
# jaunty mulch Sorry for being slow. X1146 in slot 1 should be for the cluster switch and X1148...

The X-1146 are 100g Chelsio cards and are used for HA and Metro Cluster connections, with priority slot of 1. They can also be used for networking with a slot priority of 2, 3, and 5. The X-1148 are 100g Mellanox ConnectX-5 cards and are used for NVMe with a slot priority of 5 and 3, and for network access connectivity with slot priority of 5, 3, and 2. You can find out about supported cards, usages, and slot priority using the NetApp Hardware Universe site at https://hwu.netapp.com. Select Product > Platform > Show Results, then find supported adapters.

leaden siren
split crescent
#

Agree to agree 🤓

jaunty mulch
#

Please make sure one thing with me, on C800, any one type of 1146 (in slot 2,3,5) or 1148 can be used for either cluster switches or client connections. Correct? I know 1146 can go for either 100G or 40G, and 1148 can only go for 100G

leaden siren
#

Only the card in slot 1 and the onboard ports can be used for "cluster switches"

#

and that card has to be an X1146A

#

For client connections, both cards work (generally). But even that depends, so it really comes down to what you want to do with your system:

  • As was mentioned before, if you have to use optics (SFPs) you cannot use the X1148A as that only supports passive twinax cables due to thermal issues
  • If you want to use NFS over RDMA you need the Mellanox card (X1148A), the X1146A won't work
#

and both cards support 40g and 100g connections

image.png
#

It is not that difficult really. to summarize:

  • every node needs an X1146A card in slot 1. This is for cluster and HA, it must be an X1146A, it must be in Slot 1 and it cannot be used for anything else. It's best if you don't think about this card and ignore it, since you cannot change it/do anything with it anyways.
  • Other than that card, you are free to mix and match X1146A and X1148 cards in slots 2, 3 and 5 for client networking
  • The X1148A card can only use passive DAC cables up to 5 meters in length as SFPs are not supported in that card
  • The X1146A card cannot do NFS over RDMA but it can use optical transceivers or DAC cables.
  • both cards can do 40g and 100g depending on the SFP or cable you plug in (cables and SFPs usually do not support both speeds, you need the right one for eithert 40gig or 100gig)
  • breakout of ports (converting the 40/100g port to 4x10/25g) is not supported on either card