storm chasm
I've got an existing vSphere 7U3 deployment and want to flip on NVMe with my C250.
Is it possible to run NFS/SMB/NVMeTCP over the same 10Gb adapters? I know the SVM config in ONTAP allows for all three on the same, so long as there's a namespace, but I just wanted to make sure I didn't need additional discrete adapters for NVMe over TCP.
round kraken
NVMe/TCP NVMe encapsulated in a TCP datagram. NVMe/TCP has the potential to be the most popular NVMe over Ethernet variant—and could ultimately become the logical replacement for iSCSI.
Like iSCSI, NVMe/TCP uses standard NICs and Ethernet switches, which makes it an attractive option for an environment that would like to introduce NVMe-oF on Ethernet without having to use specialty building blocks like RDMA NICs (RNICs) and data center bridging (DCB) switches required to support RDMA over Converged Ethernet (RoCE).
storm chasm
Thanks Mike! I’m gonna try and create one SuperSVM and collapse everything into one. I’ve already hit a snag where something I mount over SMB I also need in NFS.
Then just manage everything in volumes
midnight vine
a small side note: I benchmarked NVMe-TCP vs iSCSI performance in our lab last week, and just by changing the protocol (same routed 10G network, same 2-CPU VM, same A300 controller, same SSD aggregate) I got 1.5x - 2x the IOPS (52k -> 86k) with half the latency (2.4ms -> 1.4ms). So yeah, NVMe-TCP will be a game-changer
lunar belfry
interesting!
arctic thorn
We do a lot of remote backup typically via SnapMirror from our customers to our datacenter. Now some of our customers are starting using NVMe over TCP. Snapmirroring it is not a problem, but restores might be... we can of cause mount the name spaces in our own datacenter, but with iSCSI the customer can mount the datastore over the backupline directly into their vSphere environment which is nice... (of cause the line is a big limit here). But have anyone tried this with NVMe over TCP? And is it even routable like iSCSI?
past furnace
It's on the TCP/IP stack, so naturally yes, it's routable.
But just like iSCSI: Keep in mind currently there is only endpoint authentication but no data-in-transit encryption. A man-in-the-middle could sniff the packets since they are in cleartext. So make sure you tunnel this in IPsec for example.
For NVMe/TCP encryption is actually planned but not yet really implemented in the real world. You would need NVMe 2.0 which requires TLS 1.3 support. ONTAP does not yet support NVMe 2.0 (nor any other storage vendor I'm aware of).
ancient rampart
We did a POC of NVMe/TCP over a routed network and worked pretty fine. Over the same hardware and network we had a throughput increase of over 100% and a reduced latency of 50%. Thanks @past furnace for the info about encryption. Is there a timeline for ONTAP about that?
We also missing the NVMe/TCP support for MCC and even in 9.13 this is not mentioned
midnight vine
if you tunnel NVMe/TCP over IPSec, your latency will skyrocket, since IPSec in ONTAP is not supported for NVMe, so you'd need an external IPSec box, and those usually don't work at these low latencies that you want for NVMe/TCP
past furnace
"with iSCSI the customer can mount the datastore over the backupline directly into their vSphere environment" sounds to me like it's over WAN, or at least a bigger distance, so latency will already be higher than usual.
I think the question was if a customer starts using NVMe/TCP if that scenario would still be possible irrespectively of the performance impact.
past furnace
We did a POC of NVMe/TCP over a routed network and worked pretty fine. Over the ...
Can't speak for NetApp but it will take time.
The 2.0 spec is only about two years since ratification and TLS support for NVMe/TCP is not even in the Linux kernel last I checked (https://lwn.net/ml/netdev/20230824143925.9098-1-hare@suse.de/).
libnvme & nvme-cli only got initial support end of March this year:
https://github.com/linux-nvme/libnvme/releases/tag/v1.4
https://github.com/linux-nvme/nvme-cli/releases/tag/v2.4
arctic thorn
if you tunnel NVMe/TCP over IPSec, your latency will skyrocket, since IPSec in O...
this should only be used for restores, and we are connected via MPLS with no VPN or FW in between. I think we can live with the added latency as this will only be used to map a datastore to an ESXi host and copy out the VM needed...
arctic thorn
It's on the TCP/IP stack, so naturally yes, it's routable.
A long time ago I learned about the OSI Layers, and since it's NVMe over TCP and not NVMe over TCP/IP, I though the routing part may have been cut off for efficiency 🙂
midnight vine
A long time ago I learned about the OSI Layers, and since it's NVMe over TCP and...
TCP is above IP in the OSI layer, so everything that is "over TCP" is automatically also "over IP" 😉