Utilize LIF's in Vserver and to avoid migrating volumes from one Vserver to Another?

[sinful dove]

Let's say we have two vservers:
vserver "nfs-layer2" is based on layer2 not routable network and with jumbo frame, for vmware datastores or high performance NFS volumes. vserver "nfs-layer3" is based on layer3 network for all NFS volumes.

We wanted to move some NFS volumes in "nfs-layer3" to "nfs-layer2". This can be achieved by using "volume rehost" which will take quite sometime. Is there a way somehow we can utilize vserver networking, for instance, to create layer 2 LIF's in "nfs-layer3", and VM's(Linux servers) can mount the volumes via layer2 LIF's? We will of cause create layer2 NIC on Linux first. The goal here is not to migrate volumes. Is this something we can somehow achieve?

[stuck fern]

which will take quite sometime
What do you mean? volume rehost is almost instantaneous...

but sure, you can just create another LIF (there is no such thing as a "layer 2 LIF" ot "layer 3 LIF", it's all just TCP/IP) but if you need to "migrate volumes" you would still need to copy data over afterwards? that, or I didn't quite understand what you want to do

[sinful dove]

Your points on that volume rehost wouldn't take long time are taken. Thanks for that.

For my question, let me try again. It is about ONTAP networking. The SVM nfs-layer2 has 10.192.30.x not routable network. SVM nfs-layer3 is layer3, routable network. I can create a LIF (10.192.30.153, for instance) in SVM "nfs-layer3), and then remount the volume via 10.192.30.153. Thus the network traffic to the volume will go through 10.192.30.x network, and achieved my purpose without using volume rehost.

But, here is the part I don't understand: I thought we cannot do that, because we cannot route the traffic out from SVM "nfs-layer3", or route the traffic into .30 network, because .30 is Layer2 not routable network. Make sense to you?

[narrow oxide]

SVMs don't have networks - they have LIFs, which are part of networks. You can put a LIF from any network/vlan you want into any SVM in the cluster

if I had to guess, I'd say you're looking at the problem the wrong way.. Do you want to move access for a volume from a L3 VLAN to an L2 VLAN? You can do that one of two ways - a) put a new IP LIF on the L2 VLAN onto the SVM serving it already, or b) rehost it into a SVM which already has a LIF on the L2 VLAN. In both cases, you will have to change the IP address used to access the volume, which involves a short downtime. With option a) you can simply remount onto new IP and still serve off the old one until all clients are moved over, but you will end up with an SVM being on multiple networks and it sounds like you don't want that, so option b) is what to do, and live with the downtime for rehost and client cutover

[stuck fern]

The only difference between what you call "layer-2 network" and "layer-3 network" is the existence of a default gateway (default route). So no, I still don't understand your question. You can have a gateway configured and still access the LIF without using that gateway if you are in the same broadcast domain (what you call "layer-2 network"). It's basic TCP/IP routing, essentially. Maybe talk to your network admin, I'm sure he can help you set this up (i.e. give you the correct configuration for you to apply and so on)

[sinful dove]

@narrow oxide Understood and answered my questions. But, please bear with me, I still have one more question.
Please find the outputs below as illustrated.
Those 10.192.30.x in SVM “nfs-layer3” were just new created for serving the volumes that were used to be accessed through SVM “nfs-layer2”, if I choose option a) as Alex listed above. By the way, the design has been all layer2 LIF’s should fall into “nfs-layer2”, and layer3 LIF’s into “nfs-layer3”. Since we added .30 LIF’s, this design has been broken. Now, the “nfs-layer3” has "layer2" LIF’s as well.

As you can also see the routing table with “nfs-layer3”, those are routing entries available for all LIF’s within “nfs-layer3”. Since .30 network is not routable, how come their LIFs in “nfs-layer3” can be routed out? This is the part I still don’t understand and if you can help me out
`>net int show -vserver nfs-layer2
nfs-layer2-g-02 up/up 10.192.30.12/24 cls-10 a0a-308 true
nfs-layer2-g-05 up/up 10.192.30.115/24 cls-05 a0a-308 true
nfs-layer2-g-06 up/up 10.192.30.116/24 clss-06 a0a-308 true
...

net int show -vserver nfs-layer3
nfs-layer3-05 up/up 10.192.27.15/24 cls-05 a0a-318 true
nfs-layer3-06 up/up 10.192.27.16/24 cls-06 a0a-318 true
...
nfs-layer3-g-05 up/up 10.192.30.153/24 cls-05 a0a-308 true
nfs-layer3-g-06 up/up 10.192.30.154/24 cls-06 a0a-308 true
nfs-layer3-g-07 up/up 10.192.30.155/24 els-07 a0a-308 true
route show -vserver nfs-layer3
nfs-layer3
0.0.0.0/0 10.192.18.1 30
0.0.0.0/0 10.192.26.1 20
0.0.0.0/0 10.192.27.1 10`

[narrow oxide]

For access to non routed .30 IPs in the l3 svm, the only hosts that can access it are on the same VLAN

It’s not routed out, it’s listed as a directly attached network that it has a route to. Those are different things

[stuck fern]

I don't understand why you have three default gateways in your routing table, one of which is not even reachable. I think you should meet with your network admin to double-check these settings, I get the feeling the person who set up that SVM didn't fully understand what they were doing.

what does "routed out" even mean? Usually that term refers to an OSPF announcement or something similar, but ONTAP does nothing like that (at least not in your config)

[sinful dove]

What routing table should look like? .18, .26, and .27 are all networks included in the SVM. Traffics from the SVM will try out the entries in the order. That not routable network is not included in the routing table.

I heard this was not the way how ONTAP should work, and we didn't full understand how they were doing. But, didn't hear what the right configuration should be, and how ONTAP routing table should work.

[silk gazelle]

This seems all a bit unnecessarily complicated unless you have multiple vmware clusters, each on their own network and even then, it's not a good idea unless you explicitly need to share a datastore between clusters.
Your labelling of "layers" is also confusing since everything is still using "layer 3" in the networking world: IP-based traffic.
In addition, if you want to do something like multiple gateways, the remote side has to have a pretty similar idea of the same route going back to the svm or you will get asymetric traffic which things like firewalls will drop.

[stuck fern]

yeah, I've seen people create multiple default routes "because the SVM is in multiple networks"... I mean it's called "default" for a reason. I feel like it's the same here. You should only ever need one (except for first-hop redundancy scenarios maybe)

[sinful dove]

I use "layers" to label just to distinguash the difference between these two SVM for my questions. They are not labeled like this in real world. The SVM "nfs-layer2" has 2 networks, one is dedicated for multiple vmware clusters, and the other for regular NFS volumes.

It sounds like there would be no problems to have multiples gateway for different networks. The questions is if it is worth of creating a default gateway for all networks, and how can we do that?

[stuck fern]

It sounds like there would be no problems to have multiples gateway for different networks.
Absolutely. That's what host and network routes are for. The problem is when you have multiple default gateways. Those must then be all available and equal, otherwise you get asymmectric routing and similar potential problems. My suggestion would be to have only a single default gateway that knows the routes to all networks. Or, if you really need it, have a select few network routes to different gateways that you absolutely require

[sinful dove]

One more follow-up:

If LIF on NetApp, and NIC on the client are all on the same routed network, although it is routable, but, since they are all in the same network, data traffic would not go beyond the gateway. In this case, essentially, the routed network will be no different than non-routed network. Is my understanding correct and why?

[narrow oxide]

Yes your understanding is correct. That’s because of how networking works in essentially every OS - if it has a locally attached route, it won’t use the router. By defining the IP and subnet mask, you’re defining the locally attached nature of that network