#Anyway to avoid the downtime due to network switches upgrade?

1 messages · Page 1 of 1 (latest)

devout axle
#

Our network team needs to upgrade two core (customer data) switches, and they require multiple downtimes. As for why it cannot be transaparent, they have exauhsted all thoughts with the vendor.

So, they ask us the storage team to work with them and figure out a workaround to avoid the downtime. Is there any method we can achieve that?

For instance, we can fail over one node to the other in the same pair, then let them work on the connections for one new switch. Before that, we need to migrate connections from one old switch to the other. Then same thing again to do the other way around. Repeating the same steps across all pairs in the cluster. This is a large of VMware, NFS, CIFS, and some iSCSI envronment.
Not sure of if there are any issues with this approach?

If some experts here can please shed some light on this task?

ionic lantern
#

Typically for iscsi there should be to separate physical switches, is that not the case??

devout axle
#

Well, I shouldn’t have included the iscsi, because only a couple of hosts using it. It’s not our concern

ionic lantern
#

Got ya. They should be doing only 1 core at a time. But You can move NAS lifts and the cluster mgmt lif around From one node to another. I would recommend disabling Any auto revert during the process though.

torpid scarab
#

you don't really need to do a failover of the nodes, you can simply migrate all of the "lifs" from one node to the other assuming you have at least standard failover policies. Should work the same as during upgrades.

#

net int migrate-all -node $from_node

analog osprey
#

and, just to note, during the lif migration you can/will have a short blip in traffic. depending on what the client is doing it might not be an issue, but for some of our specific software it causes them to be restarted. Yea, short coming of the software, but it's still something to consider

wheat palm
#

What kind of switches? I’m routinely assisting in system configs where there are redundant switches and the use of vPC/MLAGs are widely used. You can update one switch at a time and have zero outages

devout axle
analog osprey
#

just a second or 2 usually. but anything on some of our apps causes them to die because they were just written poorly and have no reconnect/retry functions

devout axle
ionic lantern
#

Either during togb or When the NAS lif moves. Unless there is CAS in play.

analog osprey
#

aside from what's been offered, the last resort would be during a power cycle and putting the node into maintenance mode, and going through the disk maintenance options from there. but, even still, that might not work.

devout axle
sturdy perch
#

Takeover/Giveback...

devout axle
#

One more follow-up, if you still stay with me:
when i manually run net int migrate-all -node abc, all LIF's on node "abc" will be migrated to a node in a different HA within the cluster. Compariing to Takeover/Giveback scenario, all LIF's will be failover to the other node in the same HA. Correct?

I am just thinking if there are any differences and what could result in differently in these two scenario?

gritty river
#

Takeover/giveback will move LIFs wherever the failover policy says it can go. It may go to a random node if not manicured well.

#

Manually migrating LIFs like you suggested would force it to go where you want. Honestly if the switches are gonna be down one at a time but the services must be up, a LIF migration is probably better than takeover/giveback.

sturdy perch
#

and TBH it does not really matter if it gets moved to a different node on another HA pair or the HA partner. Performance-wise you will see no impact (unless your cluster network is saturated, but with that being 40/100 gig in recent systems I doubt you even can saturate that in any meaningful way... even 2x25g on the low end systems is plenty)

#

and fun fact: with block storage (iSCSI/NVMe-TCP) we have seen instances (reproducably) where access from a different node than the one hosting the LUN/namespace is actually significantly faster than accessing through the "correct" node.

devout axle
#

With all your help, "net int migrate-all -node" should be all I need to do with regards taking care of NAS protocol.

Now, if I can come back to iSCSI, as @ionic lantern reminded me first time.
Here is my understanding: I don't need to do anything with those LUN's during the network maintenance, because lun's are mapped to the same initiator through multiple iscsi LIF's on different nodes, since we are doing the network maintenace one node at a time, so, the multi-path software on the clients should take care of the redundancy.
For instance:

iscsi session show -vserver iscsi-vserver -initiator-name iqn.1991-05.com.microsoft:hostfqdn.com
Tpgroup Initiator Initiator
Vserver Name TSIH Name ISID Alias

vserver-iscsi node-07_iscsi_lif_1 74 iqn.1991-05.com.microsoft:hostfqdn 40:00:01:37:00:06 -
vserver-iscsi node-08_iscsi_lif_1 12 iqn.1991-05.com.microsoft:hostfqdn 40:00:01:37:00:08 -
vserver-iscsi node-5_iscsi_lif_1 113 iqn.1991-05.com.microsoft:hostfqdn 40:00:01:37:00:01 -
vserver-iscsi node-6_iscsi_lif_1 41 iqn.1991-05.com.microsoft:hostfqdn 40:00:01:37:00:04 -

Is my understadning correct?

sturdy perch
#

yes, at least if everything was configured correctly. It could still be that you have 4 sessions that are all going through one particular switch, but that's something only you can rule out

wheat palm
#

I usually do something like this
Set diag
iscsi initiator show -fields tpgroup, igroup,vserver -sort-by vserver, igroup,tpgroup

Unless I used the wrong field above it will show all connected iscsi connections, sorted by vserver, then by igroup and then by interface name.

You can then look at the output and verify if hosts are multipathed manually. You can also use the GUI and look at SAN and see if any hosts are partially connected

sturdy perch
#

true, but it might still be that somewhere along the path they go through the same single network link or switch... 😉

torpid scarab
#

aren't all of the connections still going to be IP connections and when the lif's are migrated, the connections will simple continue to where the "lif" lands, no matter what the protocol? Seems to be the case after 9.11.1

wheat palm
#

Iscsi lifs don’t might migrate. I’ve heard there may be version that changes, but in ONTAP you must use multi path io and hosts must have connections to more than one controller

torpid scarab
#

the docs for 9.11.1 say they do HA failover, but perhaps only for ASA... don't know from experience. It's been a long time since I used block on NetApp

#

frankly, the only reason why one wouldn't perhaps want such behaviour is the hiccups, but path failover is a hiccupy affair no matter how it's done.

wheat palm
#

It’s in the url. It’s only on asa

sturdy perch
devout axle
wheat palm
#

If the nexus 9k switches are setup with virtual port channels (vPC) then the lif migration is useless! Look at your Netapp. Are you using interface groups:

ifgrp show

If you have igroups, verify where they connect:

Network device-discovery show -port xx|yy (indicate the ports of the ifgrp)

If nothing shows, enable CDP/lldp

system node run -node * options cdpd.enable on

system node run -node * options lldp.enable on

Wait three minutes and try the discovery again.

If you have a vPC, you will be redundantly connected to both switches. You reboot one switch, ONTAP sees the link go down and moves traffic to the surviving link. It will send it back when complete

torpid scarab
#

why would the lif migration be "useless"? The vpc configuration isn't going to be across nodes, just switches. One can still disconnect all of the ports from one node and connect them to new switches while the remaining ports on the partner node, even if only one leaf switch is left, will still work, even if it's just half of them.

devout axle
# wheat palm If the nexus 9k switches are setup with virtual port channels (vPC) then the lif...

I did all you've asked for. The following is the result:
"ifgrp show" and "network device-discovery show -port" showed that all customer data, NAS and iSCSI connections are part of LACP group. Every two ports in the group are belong to two different N9K switches, so, the connections supported by the group won't be down and only LIF's on one node could get disconected at any single time during the maintenance.

" system node run -node * options cdpd.enable" shows cdpd are enabled on all nodes.

" system node run -node * options lldp.enable" shows are all off.

I don't know details how they are going to do on the switch. I am not positive, but I heard they couldn't put new switches into the same vPC with old switches due to some technical difficulties.

Are you sure of LIF migration is useless, and we don't need to do anything on the cluster? This sounds like what you are saying.

wheat palm
#

Stand alone ports are different. I almost always configure an active port channel (LACP) on each node and use vlan tags for everything. Simple and very effective

wheat palm
# devout axle I did all you've asked for. The following is the result: "ifgrp show" and "netw...

See my last post. If your controllers are in fact connected to both switches and the LACP port channel is using a link from each switch, you really should do nothing on your Netapp. This is the WHOLE POINT of using the vPCs. They can do maintenance on the switches without interrupting connected hosts (Netapp).

I would suggest to whomever is maintaining the switches to be sure this is part of the “vpc domain” configuration:

role priority 10 (should be different on each switch)

peer-switch
peer-gateway
auto-recovery
delay restore 150
ip arp synchronize

These are the settings we use when setting up FlexPods.

devout axle
#

@wheat palm I am not so familiar with vPC. Networking team definitely uses vPC here. For whatever reasons, they cannot just shut down one switch at a time, in another words, they cannot keep ifgrp alive at a time, and whole group on a node has to be down. They talked to CISCO vendor and made the conclusion.

They have to move all connections for both old switches to the new ones at the same time, which means they can not do one siwtch at a time, but only one node at a time. This is how I can explain why they ask us to do.

Can you think of any details why they cannot keep one switch alive or leave half ports in LACP alive? This is the part I dont' understand about.

gritty river
wheat palm
#

Ahhhh. I see says the blind man.
You are changing switches. If that is the case then absolutely migrate lifs from one node to another. Move the network connections and then verify before sending back. I would suggest temporarily disabling auto revert in case there is any network issue. After the ifgrp comes up in the new switch, you may wish to manually run the “network port reachability scan” command to verify before moving lifs to the new switches

#

I have seen place where they install new switches and then re-cable to the new.