sturdy cedar
For client connections, we are planning on using 1146 for A800 and 1148 for C800, all 100GbE(Don’t know why using two different types of cards). But, People here is asking if those client connections are all 100Gbase-SR4?
Also, can we use MPO cables? since we have some in Data Center.
I am not familiar with cables/modules.
We use NFS but not over RDMA.
uncut acorn
The x1148 in the x800 (c800/a800) platform DOES NOT support optical connections
The x1148 is actually a Mellanox card. Those cards always run hot. There is too much heat to use optical transceivers or AOC cables. Must use only passive twinax.
If you must use optics you need to contact NetApp and get replacement x1146 cards.
The other difference between the cards
The x1148 allows for the use of RDMA/NFS. The X1146 does not support it. So if you need the RDMA/nfs you must use the x1148 but you are limited to passive twinax
The optics on the Netapp side (x65405) use MPO cables. Your cables should be “type B” for proper function. Most cables are.
clever meadow
yeah, I think we explained the difference between the cards before already here
The supported cables and connections can be found in HWU if you click on the "cables" link in the list of supported adapter cards (see screenshot). It also tells you that you need "Type B" cables
uncut acorn
FYI
Type b are most common
Type “A” are meant for patch panel to patch panel (straight through, no flips if I recall)
sturdy cedar
On C800, there could be any benefits to use one port from x1148 in slot 2 and one port from x1146 in slot 5 to form LACP for client connections? as supposed to be both ports from same NIC card?
For instance, that might truely achieve 200G throughtput?
clever meadow
25 gb/second is tough to achieve even for a C800. There are benchmarks that reach up to 23 gb/s but that is with synthetic benchmarks and FCP (SAN), not ethernet. And I'm not even sure how many nodes they used for that benchmark, it might well be a 12-node cluster that delivered that. So I doubt that you would see any difference in practice.
But at least from a redundancy PoV it would make sense because if the PCI card broke, you could still serve traffic with the other card without having to failover the LIF to the other node
sturdy cedar
I am not worried about a failure on the card, because LIF's could be failed to the other node as you said temoprarily...
clever meadow
oh, but if you have a use-case that requires 200gbit throughput, you definitely should be worried about LIF failover to the other node as that will move 12 gigabytes/s traffic over to it as well.
OTOH, if you don't need 200 gbit/s then it doesn't matter if you build your LACP over one card or two cards
sturdy cedar
So, the question is, in terms of total of througput I am getting, is there any difference between 2 ports from a card, or from 2 different cards?
clever meadow
I would say "no" but then again I have never seen an environment that came even close to saturating two 100g links, so you might take my word with a grain of salt 🤷♂️
without knowing your environment in detail, I would bet that the throughput of a 2x100g lacp link on a single card will never be your bottleneck 🙂
uncut acorn
Don’t ever do that @sturdy cedar ! You should never mix chipsets in a port channel. It may lead to undesirable results.
The x1146 is the Chelsio card and the x1148 is the Mellanox card. Plus remember you must use twinax in the x1148 or severely risk overheating the node
clever meadow
Will it make you feel better to distribute the load across two cards? Then by all means go for it. The cost of two additional network cards is negligible in the cost of the whole system
clever meadow
Don’t ever do that <@1016009213153128559> ! You should never mix chipsets in a p...
wait I think we were talking about the same card model? If not, then yea, don't mix card models in a channel, ever
uncut acorn
Do a bit a research about it. It’s not recommended
He said a port from x1146 and a port from x1148
clever meadow
He said a port from x1146 and a port from x1148
yeah, apparently I missed that small but important detail 🙂
uncut acorn
Just because you can doesn’t mean you should.
clever meadow
We had a few cases where customers did that, and in every single one of those cases they had strange networking problems that suddenly disappeared when the port channels were removed or made across the same chip type. So yeah, never mix card types
inland garnet
Got it. Thank you!
As for 2 ports for cluster interconnections, one is used on x1146 in slot1, and the other is on e0a. Why in this case 2 ports can be located on two different places?
clever meadow
Because it's not a Link Aggregate