#Upgrade 4x10GbE TO 2x100GbE on 2 x AFF-A700 nodes

1 messages · Page 1 of 1 (latest)

humble tapir
#

Currently, 4 x 10GbE cards are located on slot1 and slot10. The new 100GbE is X91148A card, and 2 such cards on each node.
Based on my checking, it should be inserted on slot3, and 7, both will be configured in the same LACP group. We are on 9.11.1P11 now.

Does this plan sound alright to you?

green hound
#

HWU says they should go in slots 9 and 2

humble tapir
#

You are correct, it should be 9, 1, 10, 2.

Is the preference in the order as listed? So, the first two slots should be 9 and 1. We can use slot9 for the first 100G.

But, which slot should we use for 2nd 100G, should we take the 10G out from slot 1, and put 100G in the same slot 1, or we could skip slot 1, and slot10, and use slot 2?

green hound
#

no need to swap out the old cards, slots 9 and 2 are fine

humble tapir
#

Thanks!

The other question is about load balacing method in LACP group for these two new added 100G cards, which could be port-based, IP-based, MAC-based, or round robin methods. The best practice is to use port-based.

My question is, to set up which method on ONTAP side to use, is there any corresonding setting need to be done as well on the switch side?

split wadi
#

The load balancing is from the perspective where you set it. I always do port on the Netapp to give the best chance of better distribution.

On the switches you typically run a command to setup the distribution. On a nexus switch:

port-channel load-balance src-dst l4port

humble tapir
#

@split wadi
Are you saying we need to run "port-channel load-balance src-dst l4port" on the switch side, in addtion to run "network port ifgrp create -node cluster-1-01 -ifgrp a0a -distr-func port -mode multimode" on the netapp side?

green hound
#

The load-balancing on the switch is Independent to that on the end device. Usually you want both to use the same policy.

humble tapir
#

Last question with regards to NIC's upgrade from 4x10G to 2x100G.

If we create a new ifgrp group based on 2x100G, what about those vlan's based on the old ifgrp (4x10G)? for instance, those a0a-311, a0a-315 etc.. Do we have to recreate all those vlan's or we can contiue to use vlan's without doing anything additional?

covert creek
#

Create new ifgrps like a1a for example. Add 1x 100Gb port from each card to the ifgrp. Check if the LACP is full. Add your VLANs to the ifgrp. Add the new VLAN ports to their broadcast-domain (like a1a-311, a1a-315, etc). Check if layer2 reachability is working. Maybe even create test-LIFs in the various subnets to make sure everything is working correctly.

Then simply migrate your existing LIFs to the new ports.

Afterwards remove the old ports from the broadcast-domain, ifgrps, etc.

humble tapir
#

Can I add the 2 new 100Gb ports into the existing ifgrp a0a with 4x10Gb ports, then remove 4x10Gb ports from a0a? Thus, we can utilize all existing vlans and no need to migrate LIF’s?

split wadi
split wadi
#

You may wish to verify before bringing lifs back. You can use the
Network port reachability scan -node nodex -port a0a

Network port reachability scan -node nodex -port a0a-123 (put your vlan here)

Then check in a couple minutes with
Network port reachability show

humble tapir
#

Thanks! This procedure sounds better.

Now, for making it more complex, not only we need to upgrade NIC cards on 2xA700 nodes, meanwhile we also need to upgrade 2 Nexus 9k switches to 2 newer model of Nexus 9k switches due to EOL. So, 4x10Gb ports are connecting to 2 old switches, and 2 x100Gb ports will be connected to 2 newer Nexus 9k’s.

My question is, can the 2x100Gb ports be added to the same ifgrp a0a as the 4x10G ports and remove them later as we follow your steps above?

split wadi
#

Personally if you are going to do that, I would likely do something similar.

Swap out one switch and cables. This will cut all ifgrp active ports in half. One node at a time. Migrate data lifs from node. Add both 100g to the ifgrp. And remove the 10g ports. It should still show as partial, 1 up and 1 down. Test if possible. Bring lifs back. Repeat on next node.

When adding in second switch, if it is setup properly, it should just plug and go and there ifgrps should go back to full

humble tapir
#

I am sorry, I didn't quite follow steps you just said.

My question eseentially is if the new 100G ports from new switches can be added into or be part of members of the existing ifgrp a0a group and with those members (4x10G) connecting to old switches? Members(ports) in the ifgrp would connected to 2 different model of 4 switches, if that is going to work?
If yes, I can then remove all 4x10G ports from ifgrp, then only leave 2x100G, that will complete the goal.

split wadi
#

Not suggested to be serving data when mixing speeds. There are timings to take into account and all members of the group should be using the same chipset/speed. That’s why I suggest migrating data lifs away, adding in 100g, removing 10g before returning data lifs

Are you going to have all four switches up at the same time? (Usually I see people swap them out/replace one for one).

If you are able to have all four up then by all means do the first thing I said! Migrate data lifs away, as in the 100g ports, remove the 10g ports, verify the ifgrp is back to full. Test if possible then return data lifs

humble tapir
#

All steps will be performed in a maintenance window.

If you are able to have all four up then by all means do the first thing I said! Migrate data lifs away, as in the 100g ports, remove the 10g ports, verify the ifgrp is back to full. Test if possible then return data lifs

So, which means after data lif's are migrated away, the 100G ports can be the members of the same ifgrp group with the other 4x10G ports, then remove 4x10G ports from ifgrp and migrate lif's back. The goal is that I can continue to use the same a0a and all vlan's without creating a new ifgrp or vlan's. This was the part I would like to confirm with you?

split wadi
#

Exactly!

#

Not a fan of having to recreate new ifgrps and vlans, making sure they all merge correctly then cleaning everything up. Easier to do this

green hound
#

somne switches don't let you add links of different speeds to the same LACP group though. And personally I'm not a big fan of doing that (although in this case it's probably fine). So I would go the route of creating a new LACP ifgroup, cerating the VLANs and moving the LIFs over. But I assume both methods work totally fine and are just up to your personal preference

cerulean elk
#

(just did this for a bunch of 100Gb ports)

humble tapir
cerulean elk
#

I didn't, maybe I have a bit to share.

#

Hmm, not really for this case, I tend to treat this stuff as ephemeral, and I'm a weirdo who thinks in shell script. But here's a sort of example I used when I needed to add a bunch of intercluster LIFs (for SnapMirror). It iterates through a list of IP addresses we chose, and then generates ontap cli commands:

do 
  echo "network interface create -vserver uana1 -service-policy default-intercluster -home-port a0a-395 -netmask 255.255.255.0 -home-node ${LIF%%-intcl*} -lif $LIF -address $IP"
done < ~/Documents/icls.txt```
#

where icls.txt looks like:
lif-name1 10.0.1.1
lif-name2 10.0.1.2
...

#

Here's another one where we changed out physical interfaces on nodes 05 through 12 in a cluster and I had them shut down for some reason, and brought them up:

do
  echo "network port modify -port e2a -up-admin true -node nodename-$I"
  echo "network port modify -port e4b -up-admin true -node nodename-$I"
done```
#

This isn't particularly elegant but it's direct and lets you see exactly what's going to happen.

#

So basically I think about the order of operations you need to do in ONTAP, and then iterate through whatever parameters apply, working down from "larger" to "smaller" (nodes, ports, vlans, lifs).

#

The approach of moving all your LIFs to one node and "rewiring" everything on the other is a good one too - assuming you're not also changing your cluster interconnect at the same time. ;)

humble tapir
humble tapir
# cerulean elk Hmm, not really for this case, I tend to treat this stuff as ephemeral, and I'm ...

Thanks for sharing. So, in case we cannot combind 2x100G and 4x10G ports, or cannot combine two different model of switches. Then I need to do:

  1. create a new ifgrp LACP group, a1a on each node
  2. add new added 2x100 ports into the a1a
  3. Since I have about 12 vlan's, and each vlan on one node has about 10 LIF's for different vservers
    then I need to create 120 LIF's on each node, and 240 LIF's for 2-nodes HA.

Do I need to worry about failover group, since I guess these new ports should be automatically added into broadcase domain?

cerulean elk
#

and this I did recently on my homelab cluster. :)

humble tapir
#

Right....So, if I don't need to create all new LIF's then, I need to keep all vlan's and ifgrp the same name, which means that I shouldn't create a new ifgrp (a1a) or new vlans (a1a-313)... because LIF's are created on vlan's.

I should delete a0a, and a0a-313 first before I can create a new ifgrp and vlan's with the same name... and use the same a0a to create ifgrp on top of new 2x100G lacp, right?

cerulean elk
#

Nah don't need to delete everything unless you really want them called "a0a-" specifically. I'd just create a new ifgrp like "a1a" with the new 100Gb NICS. Then you'll make new VLANs off that (a1a-313) and move the LIFs to those VLANs.

#

(you can't rename or modify ifgrps other than add-port and remove-port)

cinder escarp
#

You didn't ask this, but the only thing switching to 100GbE is to make sure your clients support it or aren't transmitting high amounts of throughput.

cerulean elk
#

I'll also add that creating ifgrps and vlans on the new ifgrps are non-disruptive, and if you create a test vserver with test LIFs to migrate around, you can get a lot of this done and tested before a maintenance window. (subject to your change control rules or what have you, of course.)

split wadi
#

In the past when I do what I said (migrate data lifs away, add high-speed ports which breaks the ifgrp going to multiple switches and then removing the low speed link which then re-establishes the ifgrp) I haven’t had problems. Just make sure no data in play while it happens. If you have iscsi lifs, admin them down

green hound
#

yeah pretty sure with NX-OS you cannot mix different speeds in one port-channel

humble tapir
split wadi
#

If you are manipulating the ifgrp where iscsi is, you should knock the lif down (can’t migrate with aff/fas and iscsi) to reduce any issues. If you have multipath enabled it will be fine. When the ifgrp is updated, bring the iscsi lif back online

humble tapir
#

So, if we can fail over accesses to the other node, and we can failback when we can make sure the new ifgrp is working. We then don’t have to bring down iscsi LUN’s , right?

green hound
#

you don't bring down the LUNs. You bring down the LIFs. You should have more than 1 LIF for iSCSI and working client-side MPIO/multipathing so that taking down a LIF will not have any impact. You cannot move an iSCSI LIF to a different port while it is up, you have to take it down

humble tapir
#

@cerulean elk @here,

With regards our concern about if two different type of sets of switches or 100G and 10G can be comebined into the same ifgrp a0a group,
our network team stated that they cannot do that. They have to create a separated ifgrp for new 100G ports.

So, the qeustion is, despite that cannot be done on the network side, can it be done on the NetApp cluster and why?

I didn't find any documents saying that that cannot be done. The point is that if we CAN combind 100G port into the existing ifgrp a0a, then we don't have to create all new ifgrp, new vlan's and all confustion thereafter.

cerulean elk
# humble tapir <@530841904187703297> @here, With regards our concern about if two different ty...

It appears that you can mix speeds in ONTAP - at least as far as I could test.
I have some unused ports on a couple nodes. One set is a 40/100Gb, the other is 10/25Gb. I created a multimode_lacp ifgrp using one of each and it didn't complain. However, none of these ports actually are plugged in (so no reported speed-oper) so this may or may not be a valid test. I can't put any traffic through it to test.

#

But if your switches don't support it, it's a moot point.

cinder escarp
#

Why would you want to anyway?

#

Just have your whole network be congruent.

cerulean elk
#

I think they want to in order to make the transition easier. Add the 100Gb ports to the ifgrp, then drop the 10Gb ports

cinder escarp
#

Just create a new ifgrp then migrate the LIF to the new ifgrp.

cerulean elk
#

That's what was suggested. (And make the VLANs on that new ifgrp)

cinder escarp
#

Yeah...

#

That's the easiest.

#

You'll kill performance doing some weird mixed speed ifgrp thing.

cerulean elk
#

For my test I was wondering if I could hook it up just to see it, but we're an Arista shop and port-channel speed mixed is hardware model dependent. And my network engineers would show up with pitchforks.

#

Yeah performance will be weird though as a brief interim state I guess that might not matter.

cinder escarp
#

It might even break networking entirely.

#

It's just too much risk.

humble tapir
#

So, people in power prefers this method. That’s why I need some documentation or some reasons to show the risk or why it wouldn’t work. Besides, there some advantages about this approach, but this is not the point here

cerulean elk
#

The other issue here @humble tapir is I think you said you have different switches for 10Gb and for 100Gb. Creating an LACP ifgrp across them may not be possible at all.

#

(maybe Arista MLAG or if they're all "stacked" in some way, this I'm not sure)

#

but ONTAP won't bring up the ifgrp, or at least won't (shouldn't!) bring up the new ports if the LACP control packets don't match up

#

And I'd agree with Paul's comment on too much risk, esp in production. This will be unpredictable at best.

#

I thought of a disruptive way of doing this, but I have no way to test it.

humble tapir
#

So, all these will be done in a maintenance window and when all LIF’s are failed over to the other node. During the time, we will remove 10G, leave 100G only in a0a , then failback LIF’s. So if it works, then I don’t see risk.

cerulean elk
#

Ok, my disruptive thought is basically that. Start with the 100G ports disabled, add to ifgrp. Disable 10G ports, enable 100G ports, cross fingers.

#

If you move the LIFs... yeah I suppose it'd be mostly non-disruptive

#

I, and I'll wager a lot of others, reach for the non-disruptive, online, doesn't-need-an-outage-window approaches first these days when they're available. I'd rather do a bit more work during the daytime and do my changes with everyone paying attention than try to be clever at 2am.

green hound
cerulean elk
#

And I qualified that I didn't have a way of testing this. The OP can test it (or not) :)

green hound
#

I mean you can try and find the most complicated or brittle way to do it... 😉
or you can just create a second LACP group, check that it's working, create the VLANs and modify all LIFs to use that. It's like 5 lines of ONTAP commands (plus the VLAN creation), and you can be sure that it works because you have time to check the new ifgrps with a test LIF first

cerulean elk
green hound
#

I mean this is literally one of the easiest maintenance tasks that you can do on an ONTAP cluster. There's really no need to overthink it 🙂

cerulean elk
#

The testing aspect I think is what OP needs to stress to their management. Doing it this way (with LACP hackery) means you're testing during your cutover and if it doesn't work... well, which of the N things isn't working? Hurry up, clock's ticking...

split wadi
#

Personally have done this:
Current working 10g port channel on switch group 1. Create a second 100g port channel on switch group 2.

Pretty sure ONTAP will let you add all the ports. Once you add 100g ports to the new channel group, you will see temporary errors/issues. Once you remove the 10g ports, the channel group will figure itself back out and be fine

humble tapir
green hound
#

Just curious, but why is avoiding creating a few VLANs in ONTAP so high on your priority list?

#

it's just a copy/paste of a couple commands into the CLI. done in 5 seconds. Moving all LIFs over to the new ifgrp is one command per VLAN (not even per LIF), again easily copy/pasteable

#

something like net int modify { -home-node X -home-port a0a-123 } -home-port a1a-123

cinder escarp
cinder escarp
green hound
# cinder escarp Manglement...

I don't see how management is in any way involved in technical decision-making...? I mean what manager goes "hey, I want you to configure VLAN 142 on port 72 with a 9000MTU and then set up HSRP for the 10.0.0.1/24 network on that VLAN"?
Usually it goes like "here's the plan. These are the potential risks and how we prepare for them". Or maybe "Here are two plans, one quick and risky, the other a bit slower but safer. Which do we do?"
If management doesn't listen to (and trust in) their engineers about what's the technically best way to do something, it is high time to look for a different management IMHO

split wadi
#

finally at a keyboard.

cinder escarp
#

AKA...manglement.

split wadi
#

I have done it both ways. Lots of customers for silly reasons do not want to rename the ifgrp. So I use both methods

  1. Create new ifgrp (dist:port, multi-lacp), create vlans. Make sure vlans get into correct broadcast-domains. Modify all ports to use new ifgrp ("net in modify {-home-port a0a-123} -home-port a0b-123"). Verify connections. delete VLANs (vlan delete -vlan-name a0a-123. Repeat with remaining vlans. Delete ifgrp a0a (must delete all vlans on a0a to delete ifgrp)
  2. append to current ifgrp. Create NEW port-channel on new switches. Migrate data-lifs away from node. Add new ports to current ifgrp (ifgrp will now be in partial state, failing to the new switches as expected) and then remove old ports from ifgrp. After a mintue or so, the ONTAP node and the new switches send out updated BPDUs and re-establish the port-channel on the new faster ports. Test as appropriate. Migrate data-lifs back
  3. If you are no longer using the old ports: "net port modify -node xx -port e0e|e0f|e0g|e0h -up false"
humble tapir
# split wadi I have done it both ways. Lots of customers for silly reasons do not want to ren...

Those are ways that I am looking for!
Thanks!

If you can please continue to stay with me. Let's just focus on your method 1 for now: (then later move on for method 2 after I made sure I fully understand method 1)

Now for method 1, what we should run first is "net int migrate-all -node node-01" before start your steps as listed in method 1 above. Then after complete these steps in method 1, run "net int revert *" to migrate NAS LIF's back. Is my understanding Correct? Please confirm.

split wadi
#

Method 1 does not require any LIF migration until such time the new ifgrp is ready.

  • Create New port-channel (mode active) on new switches
  • Connect cables to ONTAP Node
    -- depending on how long you wait, you may need to delete the "Default-X" broadcast-domains
    -- i have seen scenarios where the port joins another broadcast-domain. If so, you would need to manually delete the port from whatever broadcast-domain it joined
  • create new ifgrp (ifgrp create -dist port -mode multimode_lacp -ifgrp a0b -node node-01)
  • if needed, set a0b to mtu of 9000 (net port modify -node node-01 -port a0b -mtu 9000)
    -- repeat for each node
  • add the appropriate ports to the ifgrp
  • create new vlans as needed (vlan create -vlan-name a0b-123 ; vlan create -vlan-name a0b-456; etc)
  • WAIT
  • verify the vlans on a0b automatically merge into current broadcast-domains. If they do NOT, fix the network first
  • after verifying (broadcast-domain show) you can modify LIFs
  • IF ALL nodes have the same ifgrp (a0b) and same vlans, you can do:
    -- net int modify {-home-port a0a-123 } -home-port a0b-123
    --> repeat for each vlan
    --> if your lifs are currently set to "-auto-revert true" then after the net-int-modify, they should automatically move
    --> if not, they you should be able to: net int revert *
split wadi
#

method 2 requires doing each node, one at a time, migrating data LIFs away and/or shutting down iSCSI LIFs (making sure multipathing is working first)

  • Create new port-channel (mode active) on new switches
  • Connect cables to ONTAP Node
    -- depending on how long you wait, you may need to delete the "Default-X" broadcast-domains
    -- i have seen scenarios where the port joins another broadcast-domain. If so, you would need to manually delete the port from whatever broadcast-domain it joined
  • if necessary, modify the ports for jumbo: net port modify -node node-01 -port e2a|e2b -mtu 9000
    -- set the MTU for the 100G ports as needed first if a0a is currently using MTU 9000
  • Migrate all data lifs away from node
    -- net int migrate-all -node node-01
    -- if you have any iscsi LIFs: set adv ; net int modify -node node-01 -vserver iscsi -lif iscsi_01 -status-admin down
  • Add the 100G ports into the current a0a ifgrp
    -- ifgrp add-port -node node-01 -ifgrp a0a -port e2a
    -- ifgrp add-port -node node-01 -ifgrp a0a -port e2b
  • Remove the 10G ports from the current a0a ifgrp
    -- ifgrp remove-port -node node-01 -ofgrp a0a -port e0e
    -- ifgrp remove-port -node node-01 -ofgrp a0a -port e0f
    -- ifgrp remove-port -node node-01 -ofgrp a0a -port e0g
    -- ifgrp remove-port -node node-01 -ofgrp a0a -port e0h
  • Wait bout 30-60 sesconds and check the ifgrp status
    -- ifgrp show
    --> should now be full if configured correctly
  • Send LIFS home/ enable iscsi LIFs
    -- net int revert *
    -- net int modify -node node-01 -vserver iscsi -lif iscsi_01 -status-admin up

Then repeat for each node

humble tapir
#

WOW!! You've just completed all I need!

I have only one question left to iscsi LIF's in method 2. Why would I need to take status-admin down, what if I don't? I know LIF's migration won't migrate iscsi LIFs. What if I don't do anything during the whole process?

split wadi
#

the reaon is this: Method 2 is really mucking with the "internals" of the ifgrp. Taking a working ifgrp (with 10g ports on switch-pair 1) and then we are temporarily "breaking" it (when we add the 100G ports on switch-pair 2) and then making it work again (after the 10G ports are removed and the netApp and the new switches re-negotiate the BPDUs for LACP). I dont want iscsi traffic to get corrupted or go nuts while the ifgrp repairs itself.

#

But, I get to keep the ifgrp name (a0a)

#

And if you choose not to do anything (your words), I will leave an "I told you so" on the table 😁

humble tapir
#

Your last senternce made me laugh! 😁 But, you answered my question right to the point.

So, finally, I can understand both methods well( except broadcast-domains, which I am not going to ask you about it, I need to do some home work on my own).

Now, I can have my own preference on which one should be better. I would go with method 2, mainly, it is because it can keep the same naming convention with all other nodes, which is important, in addition to that there is no need to create all vlan's

split wadi
#

Yes. Go look into broadcast domains. It us important to understand how they work and what they are for

humble tapir
#

Yes, I will. Thank you very much for your all sharing and patience. I really appreciate it!

green hound
#

I feel that this migration is something better handled with the help of a competent NetApp partner (or even NetApp PS resources)

cerulean elk
split wadi
#

Sure. But if you ever plug the cable in and there is an issue you will likely forget about that setting. By knocking the port down, it shows black in the GUI. A sort of friendly reminder to enable the port.

cerulean elk
#

Well now that I know I've done that I'll probably remember and the "Ignore Health Status" field shows up by default in net port show to clue me in.

#

Reading back in this thread (which has been useful even not as the OP) I'm wondering if this is why other people at my job have a weird distaste for iSCSI. Misunderstanding around iSCSI LIFs not-migrating.

cinder escarp
#

"I dont want iscsi traffic to get corrupted or go nuts while the ifgrp repairs itself."

This is a good reason in itself...

green hound
#

iSCSI is getting a bad reputation for being brittle or easily corruptable in light of network issues. While that might have been true in the early days, the iSCSI stacks of the common operating systems (Linux and Windows at least) have become very stable, so that this fear of "disturbing iSCSI" is generally not required anymore

cerulean elk
#

I've had little trouble with it. My experience has been primarily VMware and some Windows, with just a little Linux. If I'm doing planned maintenance I treat iscsi and FC the same way - gracefully shut down interfaces/ports during low-traffic times. This network upgrade scenario, I'd do the same.

#

Either should handle network disruptions fine, but I'd rather have some control over things than not.

green hound
#

yeah, that's basically the norm today. I mean back in 7mode times, it was different. the iSCSI stacks were new and unstable, and the error recovery levels were basically stubs and always resulted in "meh, let's do a level 0 recovery 🤷‍♂️ " from the initiator

cerulean elk
#

My one iSCSI screwup was largely self-inflicted by my employer and the ridiculously bad networking I had to work around. I feel a little bad for my successor who had to untangle that janky mess, but it did work okay despite itself.

#

Oh, although I did just think of an iSCSI "gotcha" that you could run into doing a Netapp NIC upgrade like this. At least on VMware ESXi you have (had?) a maximum number of iSCSI paths per LUN, and a total maximum of 1024 SCSI device paths. If you add more iSCSI LIFs for your new ports/VLANs, it's possible you could exceed one of those limits until you remove the old LIFs. IME with FC on arrays with lots of frontend ports, ESXi can get into very weird states.

green hound
#

yeah but you don't add new LIFs. You just move the existing ones

cerulean elk
#

Thought you can't move iSCSI LIFs?

green hound
#

you can. That has been a main point in the discussion here 😉 you just have to take them down before moving them

cerulean elk
#

Ok, I missed that small detail about them being offlined.
Was this always the case in ONTAP 9? I just did this a few weeks ago with my homelab Netapp, which is stuck on 9.old and ended up adding new LIFs - but then I didn't want to take anything down.

green hound
#

I think it has been always like that. Not sure exactly, but at least since 9.8, probably even longer

#

if you create new LIFs, you have to assign new IP addresses, add them to the initiator of all hosts, rescan, and do the same after the old ones are deleted. Offlining them, then moving them and bringing them up again is much less hassle and you don't have to touch the initiator side at all

#

it's when you use portsets that it gets annoying 🙂

cerulean elk
#

For sure that'd be a lot easier.

#

Home Netapp has too few ports, portsets would be pointless, so I didn't trip over that. I did use them on old-old-job's FC environment to reduce that path count. :)

green hound
#

we try to avoid them with our customers. In almost all cases you can achieve the same (reduced path count) by using SLM and/or zoning

#

but in some cases they are being used, which massively increases the time that it takes to do a hardware refresh, node upgrade or similar 😉

split wadi
#

The main point is being overlooked. By adding in ports to an ifgrp from a second distinct port channel may cause issues period. Which is why I move/migrate lifs away from the ifgrp. Iscsi lifs don’t migrate in normal ONTAP so I disable them. That is what you have multipathing for