CrowdSec Runtime Error: High CPU Load and Memory Errors Causing Restarts | CrowdSec | Page 1

pine cairn Jul 2, 2025, 11:11 AM

#

Hi team,

I’m encountering severe performance and stability issues when running CrowdSec with the AppSec component under high traffic conditions. Here are the details:

⸻

🔧 Setup:
• CrowdSec Engine: v1.6.9 (Docker)
• Subscribed to Console & Free Blocklist
• AppSec Component: Enabled with appsec-default rules using 4 routines
• Remediation: Nginx bouncer
• Traffic Load: ~1000 requests/sec
- Server: 8 CPU cores 16GB RAM

⸻

⚠️ Issue Summary:

After enabling the AppSec component:
• CPU usage spikes to 20–30% constantly.
• CrowdSec eventually consumes ~5GB of memory, then crashes and restarts.
• Observing frequent Nginx errors related to Lua timers.

⸻

🔍 Error Logs & Behavior:

NGINX Logs:
[error] lua entry thread aborted: runtime error: /usr/lib/crowdsec/lua/crowdsec.lua:305: Failed to create the timer: too many pending timers

Container logs before breaking:
fatal error: runtime: out of memory
...
runtime.stack: ... (full Go OOM trace follows)

Appsec config:
APPSEC_URL=http://127.0.0.1:7422
APPSEC_FAILURE_ACTION=passthrough
APPSEC_CONNECT_TIMEOUT=100
APPSEC_SEND_TIMEOUT=100
APPSEC_PROCESS_TIMEOUT=500
ALWAYS_SEND_TO_APPSEC=false
SSL_VERIFY=false

Could you help diagnose this?
• Is there a known leak or issue with AppSec timers or memory handling in this version?
• Any tuning suggestions to prevent crashes?
• Would upgrading or changing AppSec rules improve stability?

Thanks for your support — happy to provide more details or test patches!

strange flameBOT Jul 2, 2025, 11:11 AM

#

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

simple ledge Jul 2, 2025, 11:59 AM

#

pine cairn Hi team, I’m encountering severe performance and stability issues when running ...

How did you install the remediation, can you see which version it is? as this was an issue we had in one of the versions but we thought we fixed it.

Does your service have a lot of file uploads?

pine cairn Jul 2, 2025, 1:13 PM

#

I’m running CrowdSec NGINX Bouncer version 1.1.0 on Debian 11, installed through the apt package manager.
Due to dependency conflicts, I’m unable to install the latest version without first upgrading to Debian 12.
Yes, there are some file uploads as well.

dire oracle Jul 2, 2025, 1:47 PM

#

The timer issue was fixed in 1.1.1 (unfortunately, the fix makes it incompatible with nginx 1.18 due to a bug in the debian version of nginx, so we bumped the minimum required version in 1.1.2)

Regarding the file uploads, what's the average size ?

pine cairn Jul 2, 2025, 2:12 PM

#

Could the high CPU load and memory consumption be caused by a timer-related issue?
Would upgrading the machine along with a newer version of NGINX help mitigate this?

The file uploads are relatively small, approximately 50 KB to 150 KB each.

simple ledge Jul 2, 2025, 2:59 PM

#

pine cairn Could the high CPU load and memory consumption be caused by a timer-related issu...

Yes as nginx will create a new timer meaning cpu/ram is used to store the timer and this does cause exhaustion in Nginx itself. It wouldnt cause CrowdSec to OOM but the first step to debugging would be to ensure all components are up to date to ensure we are not debugging a fixed issue.

pine cairn Jul 3, 2025, 12:58 PM

#

After upgrading to Debian 12 (with NGINX v1.22 and nginx bouncer v1.1.2 now), the timer-related errors have been resolved. We’re no longer seeing memory exhaustion issues or frequent docker container restarts (previously occurring 3–4 times/hour).

However, we’re still noticing relatively high CPU usage, with around 20–25% more CPU load being consumed by a single routine in the AppSec component.
Do you have any suggestions for configuration optimizations to reduce the CPU impact of CrowdSec?

dire oracle Jul 3, 2025, 1:10 PM

#

How many requests per second are you processing ?
Which rules/collections have you enabled ?

pine cairn Jul 3, 2025, 1:17 PM

#

Approximately 700–1000 requests/sec.

For AppSec, we’re using the appsec-default appsec-config with the following enabled rules:

In-band rules:
• crowdsecurity/base-config
• crowdsecurity/vpatch-*
• crowdsecurity/generic-*

Out-of-band rules:
• crowdsecurity/experimental-*
• crowdsecurity/appsec-generic-test

dire oracle Jul 3, 2025, 1:25 PM

#

I'd need to try to do some more advanced benchmarking to see where exactly the time is spent, but I don't find it particularly high if you are analyzing 1k requests per second.

Were you using another WAF previously that used less CPU ? (we have definitely things to optimize, but again, I'm not really shocked at a 25% CPU usage)

pine cairn Jul 3, 2025, 1:46 PM

#

We weren’t using another WAF. The 20% additional CPU usage isn’t a major issue, but I wanted to double-check in case there are known optimizations that could help reduce the load.
The new crowdsec WAF component is really an amazing tool to add another security level on our infra. Keep going and provide more appsec rules for well known CVEs.

simple ledge Jul 3, 2025, 1:48 PM

#

pine cairn We weren’t using another WAF. The 20% additional CPU usage isn’t a major issue, ...

Thank you! glad to see you got value out of it, there are a few internal optimization that we can do inside the codebase but on a user end pretty much only loading what you need / general maintence is pretty much what you can do for now, obviously increasing the routines would save time spent with an increase in cpu usage, but since the problem isnt a bottle neck than this is fine for now.

strange flameBOT Jul 3, 2025, 1:51 PM

#

strange flame

Resolving CrowdSec Runtime Error: High CPU Load and Memory Errors Causing Restarts

#CrowdSec Runtime Error: High CPU Load and Memory Errors Causing Restarts