Having little luck finding anyone "official" answering my questions so I will give it a try here
Existing customer with A300 based FabricMC (FC) we would like to replace this with a C800MC IP
On the existing setup we have two dark-fibers running 16G FC between the two fabrics. The DCs are 20KM appart.
On the new C800 we struggle to find the right setup especially the ISL. As we can understand Nvidia and Cisco are the two supported switches for the backend.
Because we have 6 dark fibers now, we are seriously investigating a MUX. There are apparently two ways to MUX. Passive whre the SFP-Optics are tuned to a specific freq. which allows you to pass it through a passice MUX.. the limit here is about 10Gbs but we can run multible pairs via the MUX. The SFP optics can be coded as an Nvidia or Cisco SFP module, so the switches woun't complain about it... But how does this affect the general support of the setup? Will NetApp invalidate everything if they ever found out we were using "non-genuine" SFP modules?
Another way would be to use an active MUX which allows us to use multimode SFP modules in the switches. The MUX then handles the connection between the two sites. Here we can use genuine multimode SFP modules, but the MUX could introduce latency etc. so we would like to knowif there are any MUXs that are supported by NetApp? (we are looking at the vendor PacketLight).
Yes another question if the bandwidth needed... the current setup has an ISL of two 16Gbps links which have been working OK as long as we do not have too much cross traffic.
If you configure your MC in Fusion, you have the option to select 100Gbs, 40Gbs or 10Gbs speeds on the ISLs... but will 4 x 10Gpbs ISLs be just as good as 1 x 40Gpbs ISL?
The issue is that it is limited as to which speeds you can reach with a single SFP optics at 20KM, 100 and 40Gps is out of the question. Only an active MUX can give you 100+ Gps because it is better to use the whole frequency range.
Help is appreciated
#MetroCluster ISL questions...
quasi gorge
manic hedge
The official stance has always been "whatever the switch vendor supports is supported by NetApp" ... We have customers who are using Cisco optics with specific wavelengths (CWDM) and mux them using a passive MUX (basically a prism), for example
shadow inlet
Regarding throughput check the MetroCluster ISL Throughput Calculator in FieldPortal, so can find out the theoretical maximum write-throughput limit for your nodes. For C800 it should be 2520MiB/s. With 2x10Gb ISLs per switch you would "only" have 940MiB/s.
Also check this: https://docs.netapp.com/us-en/ontap-metrocluster/install-ip/concept-requirements-isls.html#supported-number-of-isls-and-breakout-cables
"NetApp recommends that you configure a small number of high bandwidth ISLs, rather than a high number of low bandwidth ISLs. For example, configuring one 40Gbps ISL instead of four 10Gbps ISLs is preferred. When using multiple ISLs, statistical load-balancing can impact the maximum throughput. Uneven balancing can reduce throughput to that of a single ISL."
inland tangle
Supported and required minimum bandwidth (and i think it’s also checked during the metrocluster configure process) is 4.5Gbps between the sites. If you are not happy with the SFPs that are listed in hwu you can always use the vendor supported SFPs. For Cisco it’s tmgmatrix.cisco.com
Depending on the planned workloads you should provide a solid backend speed - as mentioned above it’s technically better to go with 1x40G instead of 4x10G. For the Highend Controllers category you should plan at least a minimum of 1x40G between the switch pairs if you wanna have some fun with heavy workloads
quasi gorge
Yeah 4.5Gbps isn't much as you know, but of cause also just a minimum. If you use say 4 x 10G,, I guess that say one workload (a VM) will not be able to write with a speed above 10G, or are the controllers able to split the load out over the 10G connections? The problem with speeds above 10G is the 20KM distance, and so we will be forced to use an active MUX setup, which is a fair bit more costly 😉
inland tangle
For dark fiber you can maybe try QSFP-40G-ER4 - that 40G SFP is for a max. distance of 40km. (but maybe also expensive). The "problem" with multiple ISLs with smaller bandtwith is, that the effective bandwidth doesn't get much better if you add 4,5 or more ISLs in the LCAP channel. The recommendation is therefore to have fewer ISLs with a larger bandwidth.
quasi gorge
Yeah I am sure the QSFP-40G-ER4 will get the job done, but we would like to multiplex the connection, because right now we have 8 dark fiber connections, and we could save quite a bit if we could MUX them into two lines... So I think we need to investigate if Cisco has any tuneable optics that we can use...
quasi gorge
I just found this module: QSFP-100G-ER4L-S which seems to be able to provide 100Gbps at a range of 24-40KM (we are within 20KM). Sadly NetApp cannot provide any of the long range optics 😦 Only the switches... which seems strange, because with Brocade it's no problem...
inland tangle
yes, thats often a problem - hopefully the Partner is a Cisco Partner too or a Distri can help out to order the vendor specific SFPs.
quasi gorge
Well Cisco pricing is just horrific... we are talking a 10-fold of the prices at 3rd party vendors like flexoptix or fs.com... I seriously doubt the customer are ready to pay the cisco penalty as it is just not worth it.. so my guess is that we will just take our chances with a 3rd party vendor and take the potential arguement if a support issue will ever arise 😉
inland tangle
yeah, keep fingers crossed that you will never need TAC (Cisco Support) 😉
quasi gorge
well I won't because the switches are provided by NetApp... so NetApp will handle the support towards Cisco... or at least that how it works with Brocade... ;.)
inland tangle
unfortunately not - that has changed with MCIP and Cisco/Nvidia/Broadcom Switches - the customer always gets/buys Switch Vendor Support. NetApp will only tell you that you have to rise a ticket with the specific vendor.
quasi gorge
Hmm ok.. that's a change... is this also valid for things like Flexpod then?
inland tangle
good question - i'm not sure but that would be my expection since this is offered with cooperative support. If you have Cisco Servers too, you should check if that the "flexpod solution" checkmark is set. (check with SAM or Account Team)
manic hedge
yeah, keep fingers crossed that you will never need TAC (Cisco Support) 😉
even if... The optics vendor is not the first thing Cisco TAC looks for. Only if they reasonably believe that the optics are the cause