#IS FC conections used in Flexpod and what is MDS/Fabric Connection for?

1 messages · Page 1 of 1 (latest)

pliant bobcat
#

We use FlexPod for NAS connections only(NFS,CIFS and iSCSI). We don't use FC SAN. I am on the storage side, and not familar with Fabric Connection or UCS.

Do we have to use FC connections in FlexPod, which also means that we have to use MDS cisco switch since it is used for FC connections? When I read FlexPod architecture documents, I am confused with all these terms.

I can understand what is FI used for in general, but, I still don't understand concretly about What really Fabric Connection is used for. Can you please illustrate to explain what is going to happen if without it?

shadow junco
#

Fc is not mandatory for FlexPod use.

If you choose to use fc, you have the option of using a pair of MDS switches. Alternatively, if you are using fabric interconnects, then it is fully supported to put the FIs in to switching mode and connect the Netapp to the FIs with fc connections. Be aware, certain FIs have requirements to implement fc. Like the 6554…you must purchase the $7,000+ optic which supports breakout from the qsfp28 port to 4x32g fc and then you also need the optical breakout cables

#

Everything in FlexPod is “flexible”. You can choose to use or not use fc and/or iscsi and/or NFS and/or cifs and/or s3 and/or nvme over tcp and/or nvme over fc.

plain cloud
#

I assume you are using NetApp, Cisco, VMware... If you don't want FC, you would use iSCSI for blade/rack boot and NFS for datastore.

#

Look under Physical Topology sections. The list different Topologies.

#

This is a little dated on the software versions, but the concepts are the same for the newest.

#

Here is a newer version with IP-based storage

plain cloud
#

I reread your original question. Basically you need to be able to boot a server. You can do this with disks in a server (not a good strategy, especially for a blades. There are other issues such as rolling back, backing up, or tying an identity to a piece of hardware which nullifies the point of UCS), boot via iSCSI with a network adapter that has firmware that can inject a bootable storage device in the hardware table, boot via FibreChannel (FC) with a host bus adapter that has firmware that can inject a bootable storage device in to the hardware table, and lastly NVMe over fabric, and that fabric can be frame based, IP based, or FC, each requiring a network adapter of host bus adapter that can inject a bootable device. Cisco UCS interface cards, called Virtual Interface Card (VIC) support network interfaces and FibreChannel over Ethernet (FCoE). Newer generations support NVMe. The roll of the Fabric Interconnects in relation to FibreChannel, which are really Nexus switch hardware running UCS the management plane, is to convert FC from its native fabric to FCoE that the blades and servers can consume. The roll of the MDS is to zone, switch, aggregate, control, and monitor the FC topology. Basically it is a switch for FibreChannel.

#

iSCSI and NVMe/TCP are considered IP-based storage, of course CIFS and NFS are too but you can't boot hardware with them. NVMe/FC and FC are FibreChannel based and you would need a FibreChannel infrastructure to support it. Technically, as @shadow junco illuded, the UCS Fabric Interconnects can run in end-host mode or switch mode. Best practice, and default, is end-host mode, as you should have Nexus switches performing the switching functions for ethernet, and MDS switches performing the switching functions for FibreChannel, in front of the Fabric interconnects. If you don't, and you plug the FC storage directly in to the FI's, you would need to run the Fabric Interconnects in switch mode so that the zoning function can be performed. In summary, unless you have preexisting FC infrastructure, just ignore FC and go IP-Based for storage.

pliant bobcat
granite river
#

UCS without FIs is just standard rackmount servers so yes, you can connect those to your Nexus switches

shadow junco
#

Not entirely true @granite river
You can direct attach servers to switches and manage using Intersight. FIs are mandatory for blade servers and optional if only using pizza boxes

granite river
#

yeah but then Intersight is just a fancy web management GUI for the standalone servers 😉
At the end of the day the servers are still standard rackmount servers that you can directly connect to a switch like every other server

plain cloud
#

There may be a couple edge cases... more details would help. Do you have a UCS chassis? If yes, and you don't have FI's, then you may have a UCS-mini, where the IOM modules are the FI's. If no chassis (blade servers), and you only have rack servers, the rack servers CIMC can operate in standalone mode or managed mode. If you don't have FI's, your rack servers are in standalone mode.

pliant bobcat
#

We have Chasis, 2 FI's and 2 NX9k, and 8 nodes NetApp cluster, as shown in the picture, I can understand the Chasis is used for ESXi servers, NX9K is for networking. But, I don't understand why do we have to use FI's here, and what they are for?

granite river
#

FIs are used for managing the blades (service profile provisioning, hardware config, etc.) and to consolidate all network traffic onto one or 2 actual cables. A single blade can have up to dozens of "virtual" network cards (NICs) and Fibre Channel HBAs, and all that traffic is merged and then to/from the blades by the FI

shadow junco
#

By connecting everything to the FIs you can manage everything in one location. The CIMC on the rack servers are directed through the VIC to the FIs for server management.

There are three modes of operation and from what I’m hearing there may soon only be one

  1. Stand Alone. No UCS management. No Intersight
  2. UCSM mode. This is UCS management mode. This is minimally required for the older Cisco 5108 chassis. This uses FIs where each FI has an IP and there is a virtual IP that lands on the current primary for management.
  3. IMM. This is Intersight Managed Mode. This is what you must be using as it is mandatory when using the Cisco 9108 chassis

IMM is more flexible. If you are not using blade servers then you don’t need FIs. However you can still manage the rack servers from a central point.

Blade servers are totally dumb. They can’t do anything on their own. They must be attached to the FIs so they may be programmed as needed.

If you are using iscsi but then these blades are considered stateless. If anything happens, remove the service profile, replace the blade and reapply the same service profile and away you go

#

I’m hearing that newer m7 blade servers may only be manageable with intersight. I’m not 100% positive on this. Just something I’ve heard

pliant bobcat
#

I understand the picture better now.

We recently upgraded 2 FI's firmware, and it resulted in a few iSCSI LUN's offline. We are mainly NFS, CIFS shop, and also a hndful iSCSI for Window file server or cnetral cluster.

I understand that MPIO on Windows should take care of a path failure in a back-end mainternance like this. However, I have had multiple NetApp upgrades/maintenaces, it never caused such issue.

My question is, why you think the FI's could cause the issue? Since FI stick with UCS, did this came from UCS which take care of VM's computing power/memory, from NetApp cluster, or could be both?

shadow junco
#

When deploying iSCSI using FIs you are supposed to use two non routed , dedicated iSCSI vlans. Vlan a on FI-a and vlan-b on FI-b.

All clients should then connect using both vlans making sure mpio is in place

If the FIs are not setup correctly then when one of them is excavated your iSCSI connections will fail and without mpio and multiple connections it will also fail.

When you setup the FlexPod did you follow any of the guides?

I’ve had a few customers just wing it with iSCSI and got frustrated instead of reading and doing the process correctly

#

I’ve setup many many iSCSI-boot FlexPods.

granite river
# pliant bobcat I understand the picture better now. We recently upgraded 2 FI's firmware, and ...

there's a lot that can go wrong there. NIC pinning, UCS policies, etc. If set up correctly it should not cause an outage, but if for some reason or another both of your NICs end up on one FI, you will have a single point of failure. My suggestion is to go through the CVD and make sure all your service profiles are configured as per the best practices. Note that if you change stuff in the service profiles, it sometimes requires a host reboot so make sure the boot policy is set to "user acknowledge" so that you get a chance to click "cancel" if you don't want your ESX to suddenly reboot 🙂

pliant bobcat
#

As you may recognize I don’t fully understand FlexPod.
The iSCSI LUN’s we had the issue with are connected on Window VM’s, not on FI’s. They are not bootable only for data. I am not sure if the VLAN is routable or not, I need to investiage and get back to you.
Did this information help you to troubleshoot at all?

What is CVD?

granite river
#

of course iscsi LUNs are not connected from the FI, but the connections go through the FI. Normally you have two connections (paths), one through each FI, but if you misconfigured it, both connections go through a single FI, creating a SPoF.

CVDs are Cisco Validated Designs which are documents written by Cisco and NetApp on how to "best practice" stup a FlexPod. You can find them here or here

pliant bobcat
#

I have one more follow-up. If you can please shed any light on it.

Let's assume something went wrong with FI's, the end users lost NetApp storage connections, shouldn't they also lost the access to VM's as well? because users access UCS or ESXi hosts through Nexus switches and FI's. So, not only they couldn't access the storage, but also lost the datastore and entire VM, right?