Selfblocking with unknown Reason

[fathom kraken]

Good evening everyone,
I have been using CrowdSec on a server for quite a while now.
Crowdsec runs in a container in combination with traefik and the corresponding bouncer. Supplemented with the bouncer for the host's firewall.

For a few days now, legitimate users of the server have been blocked by mistake and I have to unblock them manually.
When I tried to find the reason, the Alert and Decision List always showed LePresidente/http-generic-403-bf or LePresidente/http-generic-401-bf as the reason.

Unfortunately I could not find out where it comes from. I have not installed such a scenario, parser or collection. Does anyone know how to fix this?

Thank you very much

[blissful rivet]

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

© Created By WhyAydan for CrowdSec ❤️ · Dec 25, 2024, 7:30 PM

[blissful rivet]

Resolving Selfblocking with unknown Reason

This has now been resolved. If you think this is a mistake please run /unresolve

[blissful rivet]

Unresolving Selfblocking with unknown Reason

This has now been unresolved.

[fathom kraken]

Here is some more input:

[old pecan]

Hi,
I have the exact same issue. All started about 3 days ago, with the same list. Is is even banning one of my internal address, despite being whitelisted !

[delicate maple]

So we updated the following LePresidente/http-generic-403-bf and LePresidente/http-generic-401-bf to cover all verb instead of just POST this means GET requests also fall into the bucket now because technically speaking 401 and 403 requests should inform the client they need to reauthenticate or simply they are not allowed to request the resource.

However, note not all application use status codes correctly. From what I can see OP the ID 90999 is the first to trigger so would be good if we can get the context from cscli alerts inspect 90999 -d

[delicate maple]

The whitelist should prevent them from ever touching the scenarios pipeline, could you check via cscli explain if you see that the log line is indeed whitelisted.

[old pecan]

Here the result of /# cscli alerts inspect 90999 -d
FATA can't find alert with id 90999: API error: object not found

[old pecan]

Need to revert on some filters. I've added a specific expression on the whitelist for GET on /api request for my LAN IP's to see if it change the behavior.

[delicate maple]

That was for the OP original poster of the issue as you havent provided any alert ID's

[old pecan]

Sorry...

[old pecan]

I guess I need to wait until the next occurence to run this command again, right ?

[delicate maple]

all good if you provide some alert details we can go through them, however, to keep it cleaner might be best if you open a new thread.

[old pecan]

Will do if reoccurring now that I've modified the whitelist

[delicate maple]

Even if the decisions expires you should be able to see previous alerts in cscli alerts list unless the flush timer has been exceeded (by default 7 days)

[old pecan]

Here the result of the first event inspect command. All subsequent ones are the same.

[delicate maple]

and this machine weboth_crowdsec is the machine that you are currently on?

[old pecan]

It is the machine where CrowdSec and Swag are installed on (docker installation). 192.168.1.254 is the IP of my router.

[delicate maple]

and if you exec into the container and run cscli parsers list you see the default crowdsecurity/whitelists? Please note if you have loaded a custom whitelist using the same name key then this may cause an issue as it will override the default LAN whitelist.

[old pecan]

Yes, as well as the newly created on for the expression filtering

[delicate maple]

Yeah but since its named crowdsecurity/whitelists it means the LAN whitelist is overriden by your custom one (or your custom one does include these also), you should unique names.

[old pecan]

So I renamed it differently to avoid confusion now, but still have the same file content as the crowdsecurity/whitelist file. Guess I don't need to duplicate by adding the "official" file too, correct ? Or would it be better to remove the local one and reinstate the clean crowdsecurity/whitelist one ?

[fathom kraken]

Tanks for the Responses

I can not inspect these old events. But i blocked now 😉

If i inspect he first event today i get this output:
crowdsec:/# cscli alerts inspect 95732

################################################################################################

• ID : 95732

• Date : 2024-12-30T14:27:15Z

• Machine : localhost

• Simulation : true

• Remediation : true

• Reason : crowdsecurity/http-crawl-non_statics

• Events Count : 63

• Scope:Value : Ip:84.152.228.140

• Country : DE

• AS : Deutsche Telekom AG

• Begin : 2024-12-30 14:27:02.525805936 +0000 UTC

• End : 2024-12-30 14:27:15.148205688 +0000 UTC

• UUID : 71cd87bf-edf1-4735-b5ed-013af6b0db1e

• Active Decisions :
  ╭──────────┬───────────────────┬────────┬────────────┬──────────────────────╮
  │ ID │ scope:value │ action │ expiration │ created_at │
  ├──────────┼───────────────────┼────────┼────────────┼──────────────────────┤
  │ 37436798 │ Ip:84.152.228.140 │ ban │ 3h54m40s │ 2024-12-30T14:27:15Z │
  ╰──────────┴───────────────────┴────────┴────────────┴──────────────────────╯

• Context :
  ╭────────────┬─────────────────────╮
  │ Key │ Value │
  ├────────────┼─────────────────────┤
  │ method │ GET │
  │ status │ 200 │
  │ target_uri │ /api/v1/request/157 │
  │ target_uri │ /api/v1/request/136 │
  │ target_uri │ /api/v1/request/123 │
  │ target_uri │ /api/v1/request/129 │
  │ target_uri │ /api/v1/request/77 │
  │ target_uri │ /api/v1/request/147 │
  │ user_agent │ - │
  ╰────────────┴─────────────────────╯

This is the result of today's latest event.
I don't quite understand the problem, because the answers are all in the 200s.

I assume the application here is Overseer and the amount of API calls at this point are normal.

[delicate maple]

200 is normal but I guess overseer requests alot of resources in individual requests at once (cause that scenarios is a capacity of 40 ). So I would recommend to whitelist those requests patterns.

[fathom kraken]

Okay, here is the full inspect of every persistent alert:

[fathom kraken]

Thank you for your feedback. How do I implement this?

[delicate maple]

here some resources:
https://docs.crowdsec.net/docs/next/log_processor/whitelist/create_expr
https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/nextcloud-whitelist

The former is the docs, and the latter is an example showing how people whitelisted expressions in nextcloud

[fathom kraken]

Thanks

[rough orbit]

This caused a lot of false positives here with git UIs that use challenge-response flows like Fork because those flows always start with a 401 and the client resends the request with proper authorization headers resulting in a successful 200 (which crowdsec ignores). The false positive blocks by crowdsec happen about every 10 minutes when using a git client with ~10 background fetched repos.

Whitelisting the IP is not an option because the blocked IP isn't static.

What other options do I have?

Would you consider a more sensible default (like ignoring a 401 if it is succeeded by a 200)?

[delicate maple]

The bucket is filering down to 401 codes only so it doesnt know it was succeeded by a 200, the other option you have is to whitelist the expression of the request sort of similar to how we did nextcloud: https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/nextcloud-whitelist

[rough orbit]

I see. The log that causes this is formed like this:

xxx.xxx.xxx.xxx - - [date:time timezone] "GET /gitlab/username/reponame.git/info/refs?service=git-upload-pack HTTP/1.1" 401 358 "-" "git/2.47.1.windows.1"

Any suggestion what I could whitelist so that it's not simply disabling the brute force check for /gitlab/?