Investigation needed: CPU pegged | CrowdSec | Page 1

steep summit Feb 9, 2025, 4:41 PM

#

Twice in the last 2 days my opnsense router running crowdsec got "crazy" and blocked my entire network by simply being overloaded. As it is a VM on my only hypervisor and I do over provisioning it just pegged my CPU at almost 100% and to me the result is none of my services are working (home assistant, reverse proxy, ...)
Looking at what I could find it seems it's my opnsense VM and specifically crowdsec consuming most of it.
Not much I was able to look for but this seems like a lead : see picture
Do you have some help to guide me towards the next investigation steps to find out what's causing it and ultimately how I can avoid this?

indigo ginkgoBOT Feb 9, 2025, 4:41 PM

#

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

steep summit Feb 9, 2025, 5:38 PM

#

I believe I might have found the issue, but don't know how to confirm...
I think my mdns_repeater was configured to repeat mdns on an interface on which it was blocked and so many repeats caused the pflog to be filled with block events on port 5353 to 224.0.0.251
well to be honest I do have a rule to allow MDSN:5353 on this interface but it might be wrongly setup and not applied for some reason.

#

I'll be crossing my fingers for now...

#

seems all I can do is hope that the MDNS trafic was overwhelming the pflog and therefore the crowsdsec parser...

steep summit Feb 9, 2025, 8:44 PM

#

hmmm, it really seems to me now it's not related to crowdsec so I'm going to rsolve this I believe, my issue is not yet solved but I do believe this is due to mdns-repeater and udp-broadcast-relay that are not playing nice together, and my mistakenly enabling one while not diabling the other on my secondary and tertiary routers..

#

I guess since one of them is rewriting the origin IP address that a single mdns advertisment is snowballing into a storm of advertisment as the repeat by udp-broadcast-relay has a rewritten sourceIP that gets picked up again by mdns-repeater that is picked up again and re-written by udp-broadcast-relay... well snowballing from here

#

all this causing too many logs in my packetFilter and therefore crowdsec just hammering my CPU trying to handle all of it

steep summit Feb 9, 2025, 11:17 PM

#

yeah, on top of that, the fact that reflection was disabled for NAT on my failover firewall prevented my reverse-proxy to work leading me to believe I had even more issues when I shutdown the primary router...
Well, all seems fine now I even have all my chromecast back as available target thanks to diabling udp-broadcast-relay.

#

so nothing to do with crowdsec indeed, well crowdsec was only one symptom of a storm of mdns packets being self-repeating

indigo ginkgoBOT Feb 9, 2025, 11:18 PM

#

indigo ginkgo

Resolving Investigation needed: CPU pegged

#

This has now been resolved. If you think this is a mistake please run /unresolve

#Investigation needed: CPU pegged