#Decisions from lists - processing, removing, notifications, checking

1 messages · Page 1 of 1 (latest)

cobalt kestrel
#

Hello, I have an instance of crowdsec, currently on premium with subscribed some lists, overall I have over 100k decisions from lists and over 40k decisions from local
I have few issues:

  1. When trying to check if IP is in decisions, if IP is from lists, it will not show
    cscli decision list --ip 34.83.216.252
    No active decisions
    But when I'm checking in mysql it is active:
    select * from decisions where value='34.83.216.252'\G
    *************************** 1. row ***************************
    id: 120066859
    created_at: 2026-04-20 08:23:06
    updated_at: 2026-04-20 08:23:06
    until: 2026-04-21 08:22:33
    scenario: threat_forecast_129d1330-4688-49b7-8b20-bb16f305266e
    type: ban
    start_ip: -9223372036278855427
    end_ip: -9223372036278855427
    start_suffix: -9223372036854775807
    end_suffix: -9223372036854775807
    ip_size: 4
    scope: Ip
    value: 34.83.216.252
    origin: lists
    simulated: 0
    uuid: NULL
    alert_decisions: 77350754
    Also it's being processed by bouncer but still I cannot get this decision using api nor I cannot find it via dashboard at app.crowdsec.com/decisions
  2. I cannot delete decision using cli nor API, that i recieved from lists. If I have some client complain, i need to delete it manually from endpoint firewall, if i try do thi normal way it just says that there is no decision regarding this ip
  3. Processing is very slow and prioritizes adding new decisions over deleting. To delete, I need to restart customer bouncer so it will pull deletions and process them.
  4. I have set up notifications using slack and elastic, neither works for decisions from lists.
  5. I had to fix indexes on decisions table in mysql, because some queries were extremely slow... That should be done by dev's not the end user.
    
    My Crowdsec version is 1.6.11, running on 8 cores and 40GB of RAM
deep hullBOT
#
Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

ivory spear
#

Hey,

THere's a known issue that --ip will only look at local decision, not decisions coming from blocklists.
You can workaround by doing cscli decisions list --all | grep IP : not pretty but it will work.
Decisions from blocklists are not shown in the console by design: it would just spam you for little value.

That being said, cscli decisions delete -i IP should work just fine with any IP, no matter where it is coming from.

1.6.11 is kinda old (almost 9 months), so we've had quite a few bug fixes and performance improvements since then.
The next release will bring a noticable improvement in performance and memory consumption when handling hundreds of thousands of decisions.

What do you mean by I have set up notifications using slack and elastic, neither works for decisions from lists. ? We don't notify you for any new IP added to a blocklist, it would be unmanageable (although admittedly, we could try to have a notification so that you know the blocklist was updated).

Do you have some examples for the queries that were slow, and which indices you created ?

cobalt kestrel
ivory spear
#

Some of the queries you've pasted look like they are from either cscli decisions list or cscli alerts list: for those, some indexes were skipped voluntarily. Those queries are (most of the time) ran by a human, not part of an automated process in crowdsec itself.
This is a trade-off: we could add indices of every fields or combinations of fields we use to query the DB, but it would most likely absolutely destroy the write performance (which is way more important if you want crowdsec to process everything in a timely manner).

I know we are missing some basic indexes on some tables (bouncers or machine for example): those would be a quick win from a correctness perspective, but in practice those tables are very very very small (a few hundreds records in the absolute worst case), so the gains would be minimal

cobalt kestrel
#

No, those queries were not executed by triggering "cscli decisions list", i'm the only one administrating this server and executing direct console requests, after fixing those indexes crowdsec stopped crashing everyday and cpu usage of mysql dropped from 200% to 60%.
Yet, there are still isues I have mentioned in points 1-4, grepping will not solve an issue that I cannot look for those decisions using your web dashboard (only 10k deciosions seen, while i have 40k active done by my own instance?), nor using lapi.
You ask me what do I mean by setting up notifications? I have used elastic plugin
https://docs.crowdsec.net/docs/local_api/notification_plugins/elastic/
to preserve ALL decisions, because we need history for our tech support to handle our clients, and i mean ALL decisions have to be stored, including those from capi or lists.
Also, how to delete decisions that were triggered by lists or capi? We have a system that communicates with lapi to manage (add, delete or look for current decisions)

CrowdSec can forward Alerts to Elasticsearch using the HTTP plugin. This guide will show you how to configure the plugin to send alerts to your Elasticsearch instance.

ivory spear
#

There's no plan at the moment to show decisions from blocklists in the console, nor to send notifications for every decisions coming from blocklists (it would be too much of a foot gun for 99% of the users)

You can delete decisions from lists or CAPI with a simple cscli decisions delete (or query directly the API exposed by LAPI if you have custom tools on this endpoint https://crowdsecurity.github.io/api_doc/lapi/#/watchers/deleteAlerts

#

Regarding the indexes, can you show me which one you created ?

#

Finally, for the 10k decisions, I'll need to check internally, but I think this is a limitation of the console (there's also a 10k alert limit per month per log processor, so you might be running into that)

ivory spear
#

For the decisions in the console, we don't have an explicit limit in our backend.
At the moment I see a bit under 15k decisions in our DB (but it changes a lot).
We do some deduplication in the console to not display too many redundant things, that could also be an explanation.
Was your 40k decisions without any duplicates ?

cobalt kestrel
# ivory spear There's no plan at the moment to show decisions from blocklists in the console, ...

its a shame that there is on option to enable notifications from lists, I'll have to figure something out on my own then. I think that you are underestimating how many users would like to have this feature 🙂
Regarding local decisions, right now i have way less of them after I have done some changes to the scenarios, but previously i had only 10k decisions visible at the dashboard (it was capped exactly at 10k).
Still, not being able to look for decisions from lists at the web dash is a problem for us and we will not be able to use dashboard 🙁
I have checked again and this time I was able to unban IP addresses fron lists using cscli and lapi, but when I'm trying to check single address via lapi, I'm getting full list of active desicions, i have tested with "include_capi=1" and "include_capi=0"
GET /v1/alerts?has_active_decision=1&include_capi=1&limit=100&ip=66.132.224.28
it's like those filters stopped working after conecting to the central API
When I set "limit=1" it started working for local decisions, but when checking central api's, still got all active ones listed out.