#CRC Errors

1 messages · Page 1 of 1 (latest)

kind condor
#

Hi all, would love some input on next troubleshooting steps to take if anyone has ideas! Basically I have an AFF-A220 that is acting as a datastore for our vCenter, pretty basic. On that NetApp i've begun getting alerts regarding CRC errors, and running "node run -node Node1 ifstat e0d" we can see about ~50 CRC errors coming in per minute. Networking wise, this is all on a backend network that is simply Netapp > Cisco Nexus switch > Dell Chassis w/ the ESXi hosts. On the switch interfaces facing our Node1 ports that our NFS data LIF sits on, we can see lots of outbound errors when running show interface. During our troubleshooting I migrated the LIF over to Node2, on another ifgroup we have going to that same switch (this ifgroup is unused, whole other story there), and rebooted Node1 to see if that would solve the CRC errors. While Node1 was rebooting, Node2 began getting the same CRC errors on the ports that our data LIF was sitting on. When Node1 finished rebooting, I migrated the LIF back to its home port and we continued getting errors on those Node1 ports, and ceased getting them on the Node2 ports. the This leads me to believe the issue does not lie within the cables/ports as a lot of digging online has suggested. Unfortunately this NetApp is at a site across the country and it will take us a week or two to get out there to physically look at it. Currently waiting a while for NetApp support to reply to my reply on my ticket, but wanted to see what you pros thought! Phew that was a mouthful!

novel ibex
#

if you are using cut-through switches (instead of store-and-forward), the CRC errors can come from anywhere in the path from source to destination, since the switches can only notice the errors after they have already sent the packet out so it's too late to drop it. So you really have to check the full network path for cable/transceiver issues

kind condor
#

our VMware guy ran some CLI commands and confirmed that there weren't any CRC errors popping up on the ESXi host ports. The only places we saw consistent errors were on the NetApp side, and also on the Switch interfaces facing the NetApp, reporting "outbound errors". Our theory is that the NetApp isn't receiving the frames properly and is telling the switch that

harsh kelp
#

Did you check the switch ports where the ESXi servers are connected? The usual problem with connections is that the sender can't detect if he is causing an issue. So your ESXi guy might not see any problems but is causing it anyway.

kind condor
#

Yes we did, no input or output errors there.

harsh kelp
#

You could try to evacuate the ESXi servers one by one. See if that stops the errors for one of the hosts.

kind condor
#

yeah, that was one of our next steps. i would think that the problem would stem from one of the VM's, not from the host itself. We may need to submit a CRQ to turn the VM's off/on one at a time, but saving that for last as our CRQ process would take a while to get approved

harsh kelp
#

So what kind of errors do you actually see? CRC is very low level, I think those can't be caused by a VM.

novel ibex
kind condor
kind condor
cunning spade
#

Wheat are the physical connections? Optical or twinax?
Where are the crc errors being reported? Netapp? Switch ports? Incoming or outgoing?
Version on Netapp? Are you using a port channel? So many questions

kind condor
cunning spade
#

You might want to upgrade. There were some bugs that related to this. Let me see if I can find something

novel ibex
#

these are generally quite tricky to troubleshoot. you could check the other switches that are connected to the one you've looked at, and see if any of those also report output CRC errors towards the downstream switch. note that the numbers could be smaller (like half or even less) than the numbers on the downstream server, depending on the pathc the packets take. If you are really curious, you can maybe (temporarily) convert the switch to store-and-forward and see which client is having his packets dropped more frequently... as long as the errors are not significantly higher this shouldn't affect any network traffic as TCP will simply retry sending the lost packets

kind condor
cunning spade
#

Ok. Are the crc errors actually on the port or the ifgrp?

#

Switch model? Are they doing vpc? Version of code?

#

What are the twinax cables you are using? Are they Cisco? Netapp? Or something wise?

kind condor
cunning spade
#

Cisco what?

kind condor
#

No, the CRC errors are only populating on the underlying e0d port of the ifgroup

kind condor
cunning spade
#

ifgrp show -instance ?

#

Node run -node xx ifstat e0d

kind condor
harsh kelp
#

Depending on the load maybe only e0d is used

#

You said it moves to the second node during failover. More likely it comes from the source

kind condor
cunning spade
#

Other errors, pause frames, xon/xoff

#

Depends on the the mode

#

Hence the instance if you can send the output

harsh kelp
#

Same source-target combination usually uses the same link.

kind condor
#

for e0d

#

don't see anything else

cunning spade
#

It depends on a number of factors. In FlexPod we set up everything in the switch and ONTAP to use src/dat/port which better distributes

#

OK since you can’t or won’t get send output, what ports are in your ifgrp? What distribution is it using? What mode is it using?
Are you using vlans?

#

Maybe the switch is not correct. If I can see how the ifgrp is configured I can offer suggestions for the switch side

kind condor
#

e0d/e0c for the ifgroup, distribution function is IP, create policy is multimode_lacp and yes there is a vlan sitting on top of the ifgroup we are using. The error populates for e0d, a0a, and a0a-30

cunning spade
#

Try shutting down e0d and verify the crc moves

kind condor
#

Unfortunately I can't copy paste the output, I'm using discord on personally device as we can't use it on network laptop

cunning spade
#

Ah ha

kind condor
#

I did do this two days ago - we rebooted the whole node that the ifgroup sits after manually migrating the LIF to the 2nd node. The CRC errors did populate on the second node while node1 was rebooting

cunning spade
#

Any chance you can get a look at the switch config for the Netapp?

kind condor
#

and e0c from the same ifgroup on node1 also has CRC errors, just not as many as e0d

kind condor
harsh kelp
#

Check the uplinks of your Dell server chassis.

#

Possibly the easiest to pull them one by one and see if it stops. 🙂

#

Don't overcomplicate it 😉

cunning spade
#

Try this

system node run -node * options cdpd.enable on

Wait about three minutes and do

Network device-discovery show -protocol cdp

Hopefully you can identify which ports you are in on the switch (which model again?). Maybe an ASIC issue on the switch and ask the ports are in the same ASIC?

#

@harsh kelp suggestion is really good also

kind condor
kind condor
#

TBH don't trust the one guy over there to pull the right cables 😅

cunning spade
#

For refence @harsh kelp , on a cisco switch I use:
port-channel load-balance src-dst l4port

#

really helps with distribution

#

On the NetApp for the ifgrp, waaaay better to use port for distribution
(Network traffic is distributed based on the transport layer (TCP/UDP) ports)

#

@kind condor , the switch config for each netapp should look something like this:

`interface Po121
description netapp-01
switchport mode trunk
switchport trunk native vlan <native-vlan-id>
switchport trunk allowed vlan 123,456
spanning-tree port type edge trunk
mtu 9216

interface Eth1/21
description netapp-01:e0c
channel-group 121 mode active
no shutdown

interface Eth1/22
description netapp-01:e0d
channel-group 121 mode active
no shutdown`

Please note that whatever your native VLAN is, should NOT be used as a tagged vlan. i.e. if Native is 123, then on the netapp If I need to use 123 I would use a0a (the base interface). If you need to use 123 as a tagged vlan, just make sure the native vlan is something unused.

#

(native default to 1 if not specified)

#

@kind condor is it a SINGLE 93180 or a pair in using Virtual Port-Channels (vPC)?

kind condor
#

why is it mtu 9216? we have it as 9000?

#

negative, just one

cunning spade
#

fix it!

#

Why: overhead

#

The switches should be 9216 (maximum). Different switch vendors have different max sizes as they treat overhead differently

#

all clients (netapp, esx) should be 9000. The vlan tagging adds extra and the 9216 account for this

kind condor
#

oh, the ESXi hosts should be set to 9000 as well? that is not something i verified with my VMware guy but that makes sense

cunning spade
#

This will defintely cause at least fragmentation as the netapp is sending 9000 and the switch will need to break that into two frames to pass along

kind condor
#

this NetApp was up for 2 years before this started so im sure it must be

cunning spade
#

This will not cause the CRC, you could potentially get a little faster with the correct switch mtu and prevent fragmentation

#

Any luck on identiifying which ports on the switch?

kind condor
#

Yes i know which ports go to the NetApp. Config for the 4 interfaces headed to our NetApp is:

#

switchport mode trunk, switchport trunk allowed vlan 30, spanning-tree port type edge trunk, spanning-tree bpduguard enable, mtu 9000, channel-group 1 mode active

#

and obv channel-group mode 2 for the 2nd port channel

cunning spade
#

what are the actual port numbers on the switch

kind condor
#

eth1/29-32

#

when you were asking about the ASIC, is that the backplane that the interface sit on? One of our theories was that the backplane was having issues, our switch guy was going to look into how to verify that but no dice yet

cunning spade
#

yeah, something like that

#

Looks like that switch has a single ASIC for all 48 10/25G ports

kind condor
#

Thank you both for your expertise - we think we may have something. We did find input errors on the switch stemming from one of the ESXi hosts. So looks like it is originating upstream. Also there is only on VM sitting on that host so we will troubleshoot the host and VM and i'll report back!

cunning spade
#

Excellent. Please "fix" the MTU on your switch ports when you can

kind condor
#

Apparently the NetApp implementation engineer that helped my predecessor install this box was adament about the 9000 mtu setting so my switch guy would prefer not changing it.

cunning spade
#

I think most of us here will tell you that "NetApp implementation engineer" was wrong. You more than likely are getting network fragmentation.

#

I would say....look at >any< cisco documentation about jumbo frames. Every singel time they tell you to use 9216 on the switch side. All clients stay at 9000

kind condor
#

I feel a little stupid reporting back that the issue seems to be resolved. I guess through lack of attention to detail on my part I missed the input errors on the switch coming from one of the ESXi hosts. Once we saw that we attempted to move the sole VM it had living on it off the host but it wouldn't go. We turned off the VM and it migrated just fine. Our VMware guy said he was seeing errors populating on the ESXi host(will need to get a debrief from him later on exactly what he saw), so they rebooted it and moved the VM back and viola, we are no longer getting any new errors on the host, the vm, the switch, or any NetApp ports.

fair mountain
#

we've one time got problem with several vms on iscsi datastore hosted from netapp. terrible performance, but vmotion solved this. after few days issues reoccured when vm admin was migrating. one link at esxi host negotiated 100mb throughput with switch instead of 10000mb

noble idol
#

I was gonna say...you probably gotta check somewhere else. Usually if I get something like that I like to involve all vendors, esp. the network vendor.

fair mountain