Interconnect cluster connections are missing ! | NetApp | Page 1

dire slate Dec 4, 2024, 1:24 PM

#

We got a serious issue.The entire cluster is very slow. By checking the AIQUM, I found the red on Cluster Interconnect . The title is not acurate. I have not found connection are missing so far

How can I find the cause, cables connections? ports??? We use CISCO 3132q-v switches

I did the following, no issues
"net int show -role intercluster", "net port show",
"ifstat" on interconnect ports
"cluster ping -node nodes"

dire slate Dec 4, 2024, 2:13 PM

#

I found one thing. By running " network interface check cluster-connectivity show". I found "all" uner "packets loss". How do I identify what caused the packets loss?

brave dirge Dec 4, 2024, 2:44 PM

#

Check both ifstat on ONTAP and switchside, maybe proactively replace cables/SFPs to check if that helps. Otherwise create a NetApp case, I hope this cluster is under support.

mighty grotto Dec 4, 2024, 2:52 PM

#

Intercluster is not cluster interconnect though

blissful moss Dec 4, 2024, 2:57 PM

#

have you done an ontap upgrade lately ?

#

i am asking because i have seen something strange after an upgrade from 9.14p8 to 9.15p4 ; lifs did not dissapear but i had missing peering ip's in the cluster peering

mighty grotto Dec 4, 2024, 2:59 PM

#

Look at the Ports with role "Cluster", not "intercluster"

blissful moss Dec 4, 2024, 3:00 PM

#

ah they are missing cluster lifs ? never had that

dire slate Dec 4, 2024, 3:39 PM

#

What is the equivalent command of "ifstat" on Nexus OS?

dire slate Dec 4, 2024, 3:40 PM

#

blissful moss ah they are missing cluster lifs ? never had that

No recent upgrade. I miss spoke, I have not found missing connection.

dusty salmon Dec 4, 2024, 4:03 PM

#

Do this first

net int show -vserver Cluster
net port show -ipspace Cluster

Make sure they all look good

Then go to diag mode.

system switch ethernet <tab> show
You’re looking for the sub command to show performance or near that. Not seeing it in the online man page. I’ve used it and I know it’s there

dire slate Dec 4, 2024, 4:15 PM

#

I ran all commands you asked for, no issues found.
We were just told that it may be caused by MalwareBytes full scans, Window guys performed. That caused huge IOPS on all 7 nodes within the 8 nodes cluster.

If this is true, then probably there are too many indirect access's, and therefore data transmissions between nodes caused warning / Red on Cluster Interconnects. Make sense to you?

dusty salmon Dec 4, 2024, 4:42 PM

#

Absolutely
One of the “system switch ethernet” commands should show switch performance numbers that may corroborate that statement

dire slate Dec 4, 2024, 4:57 PM

#

Yeah, this is what I am doing, to corroborate that statement.

I don't see any options of the command "system switch ethernet” that can show me the performance numbers...

dire slate Dec 4, 2024, 5:40 PM

#

qos statistics volume latency show -iterations 100
this should be the command

mighty grotto Dec 4, 2024, 6:11 PM

#

blissful moss ah they are missing cluster lifs ? never had that

I don't think they are "missing" but the red part in the screenshot called "cluster interconnect" is almost certainly the cluster backend and not some intercluster connection

#Interconnect cluster connections are missing !