#Unable to persist Local API key on container restart

1 messages · Page 1 of 1 (latest)

gray ermine
#

I am running caddy with caddy bouncer build from hslatman.
When I run sudo docker exec crowdsec cscli bouncer add caddy-bouncer I get an API key. This is good, then I enter it into my docker compose and Caddyfile and restart Caddy. No issues and crowdsec is working normally.

Unfortunately, whenever there is an update or I run sudo docker compose down that API key needs to be reset. I have looked on forums and though this discord to find fixes but I am unable to find any solutions.

The error which appears only after a container down or update:
"msg":"auth-api: auth with api key failed return nil response, error: dial tcp 172.30.0.2:8080:
I know this means it cant connect to the local API, but it does work once I rerun cscli add caddy-bouncer and replace the API key.

What I've tried:

sudo cscli lapi register doesn't seem to fix anything.

This looks promising but I cannot find the caddy-bouncer remediation file.
https://docs.crowdsec.net/docs/next/local_api/authentication/#:~:text=dedicated documentation.-,Remediation Components (Bouncers)​,API token to use it in your Remediation Component configuration file.,-Log Processors (machines

My compose:

  crowdsec:
    container_name: crowdsec
    hostname: crowdsec
    image: crowdsecurity/crowdsec:latest
    expose:
      - 8080
    restart: always
    environment:
      GID: "${GID-1000}"
      BOUNCER_KEY_CADDY: <KEY>
      COLLECTIONS: <COLLECTIONS_HERE>
    volumes:
      - ./logs:/var/log/caddy
      - ./crowdsec/crowdsec-db:/var/lib/crowdsec/data/
      - ./crowdsec/crowdsec-config:/etc/crowdsec/
    labels:
      - com.centurylinklabs.watchtower.enable=true
    networks:
      caddy:
        ipv4_address: 172.30.0.2

Caddyfile:

    crowdsec {
        api_url http://crowdsec:8080
        api_key <KEY>
    }
}```
hallow latchBOT
#
Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

gray ermine
#

@ me and I will respond asap, I need to figure this out

gray ermine
#

Anyone help?

heavy smelt
# gray ermine Anyone help?

first check if any bouncers are already there
docker exec crowdsec cscli bouncers list
if there remove first
docker exec crowdsec cscli bouncers delete "bouncer name"
then register bouncer
docker exec crowdsec cscli bouncers add caddy-bouncer
get the key
and place it in caddy file. restart caddy.

gray ermine
#

You didn’t really read my message did you, sorry that was a bit harsh lol,

#

This works and yes the bouncer shows up, it’s that on container restart, or update, the api key will reset it self and I need to re add the bouncer manually every time

#

Rather than just doing it once, I have to add the bouncer many times,

mild zodiac
#

A few things to check:

  • Can you confirm the crowdsec.db file is in the crowdsec/crowdsec-db directory ?
  • If it's there, run the command sqlite3 ./crowdsec/crowdsec-db/crowdec.db then SELECT * FROM bouncers;: just to make sure the data is indeed in the database
  • Can you provide the logs of the crowdsec container ? Maybe for some reason, crowdsec does not find the DB and create a new one on each startup
heavy smelt
#

if that doesn't work. we can see the db is persistence or not.

gray ermine
#

I am seeing the db file I guess I dont have SQLite installed? Or do I need to execute on the cscli?

gray ermine
#
    build:
      context: .
      dockerfile: ./Dockerfile
    depends_on:
      - crowdsec
    container_name: caddy
    hostname: caddy
    restart: always
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"```
#

It now depends on crowdsec

#
Check if lapi needs to register an additional agent
sqlite database permissions updated
/etc/crowdsec was found in a volume
Running hub update
Downloading /etc/crowdsec/hub/.index.json
/var/lib/crowdsec/data was found in a volume
Running hub upgrade
WARN parsers:crowdsecurity/whitelists is tainted, use '--force' to overwrite
Running: cscli  parsers install "crowdsecurity/docker-logs"
Nothing to do.
Running: cscli  parsers install "crowdsecurity/cri-logs"
Nothing to do.
Running: cscli  collections install "crowdsecurity/caddy"
Nothing to do.
Running: cscli  collections install "crowdsecurity/http-cve"
Nothing to do.
Running: cscli  collections install "crowdsecurity/appsec-generic-rules"
Nothing to do.
Running: cscli  collections install "crowdsecurity/appsec-virtual-patching"
Nothing to do.
Running: cscli  collections install "crowdsecurity/sshd"
Nothing to do.
Running: cscli  collections install "crowdsecurity/linux"
Nothing to do.
Running: cscli  collections install "crowdsecurity/base-http-scenarios"
Nothing to do.
time="2025-03-13T02:36:07Z" level=info msg="Enabled feature flags: none"
time="2025-03-13T02:36:07Z" level=info msg="Crowdsec v1.6.5-72b4354b"
time="2025-03-13T02:36:07Z" level=info msg="Loading prometheus collectors"
time="2025-03-13T02:36:08Z" level=info msg="Loading CAPI manager"```
#

logs on restart

#

after restart it now needs a new api key once again

gray ermine
# mild zodiac A few things to check: - Can you confirm the `crowdsec.db` file is in the `crow...

when lapi key is working: 11|2024-11-01 21:35:58.106627865+00:00|2024-11-01 21:35:58.106627985+00:00|CADDY|f292e4b7ceaa6bb6be4e5f3384a2b9f61c6f50e62b3da72f2ede4cee24cd45ae740710df9e5f916e35f946da0cbeeeea27fc82959286d3764a29942316e2fa3b|0|||||api-key||||0 26|2025-03-13 02:47:49.14646011+00:00|2025-03-13 03:18:37.54483813+00:00|caddy-bouncer|0326c1a66fe92320396c76d8ba74b8d7abf9adb940e2529b751bd4a50deda32c8ee7e0e207c7d9a442177cebd924615488837a585a6674892efe388e003b2b2f|0|172.30.0.3|caddy-cs-bouncer|v0.7.0|2025-03-13 03:18:07.543987004+00:00|api-key|Alpine Linux|3.20.3||0

#

after container down and restart (lapi key not working)

11|2024-11-01 21:35:58.106627865+00:00|2024-11-01 21:35:58.106627985+00:00|CADDY|f292e4b7ceaa6bb6be4e5f3384a2b9f61c6f50e62b3da72f2ede4cee24cd45ae740710df9e5f916e35f946da0cbeeeea27fc82959286d3764a29942316e2fa3b|0|||||api-key||||0
26|2025-03-13 02:47:49.14646011+00:00|2025-03-13 03:19:52.952697715+00:00|caddy-bouncer|0326c1a66fe92320396c76d8ba74b8d7abf9adb940e2529b751bd4a50deda32c8ee7e0e207c7d9a442177cebd924615488837a585a6674892efe388e003b2b2f|0|172.30.0.3|caddy-cs-bouncer|v0.7.0|2025-03-13 03:19:42.949496154+00:00|api-key|Alpine Linux|3.20.3||0```
heavy smelt
#

my final head scratch, what i do for traefik

services:
  crowdsec:
    # ... existing configuration ...
    healthcheck:
      test: ["CMD", "cscli", "lapi", "status"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

  caddy:
    # ... existing configuration ...
    depends_on:
      crowdsec:
        condition: service_healthy

if this doesn't work it upto the developers

mild zodiac
#

Can you paste the full logs of the crowdsec container ?

gray ermine
# mild zodiac Can you paste the full logs of the crowdsec container ?
Check if lapi needs to register an additional agent
sqlite database permissions updated
/etc/crowdsec was found in a volume
Running hub update
Skipping hub update, index file is recent
/var/lib/crowdsec/data was found in a volume
Running hub upgrade
WARN parsers:crowdsecurity/whitelists is tainted, use '--force' to overwrite
Running: cscli  parsers install "crowdsecurity/docker-logs"
Nothing to do.
Running: cscli  parsers install "crowdsecurity/cri-logs"
Nothing to do.
Running: cscli  collections install "crowdsecurity/caddy"
Nothing to do.
Running: cscli  collections install "crowdsecurity/http-cve"
Nothing to do.
Running: cscli  collections install "crowdsecurity/appsec-generic-rules"
Nothing to do.
Running: cscli  collections install "crowdsecurity/appsec-virtual-patching"
Nothing to do.
Running: cscli  collections install "crowdsecurity/sshd"
Nothing to do.
Running: cscli  collections install "crowdsecurity/linux"
Nothing to do.
Running: cscli  collections install "crowdsecurity/base-http-scenarios"
Nothing to do.
time="2025-03-13T11:15:10Z" level=info msg="Starting community-blocklist update"
time="2025-03-13T11:15:10Z" level=info msg="attempt 1 out of 2"
time="2025-03-13T11:15:13Z" level=info msg="capi/community-blocklist : 0 explicit deletions"
time="2025-03-13T11:15:13Z" level=warning msg="sqlite is not using WAL mode, LAPI might become unresponsive when inserting the community blocklist"
time="2025-03-13T11:15:14Z" level=info msg="crowdsecurity/community-blocklist : added 15000 entries, deleted 14985 entries (alert:1821)"
#

this is all it shows if i do sudo docker logs > message.txt

#
time="2025-03-13T07:15:11Z" level=info msg="capi/community-blocklist : 0 explicit deletions"
time="2025-03-13T07:15:11Z" level=warning msg="sqlite is not using WAL mode, LAPI might become unresponsive when inserting the community blocklist"
time="2025-03-13T07:15:12Z" level=info msg="crowdsecurity/community-blocklist : added 15000 entries, deleted 14995 entries (alert:1818)"
time="2025-03-13T07:15:15Z" level=info msg="172.30.0.3 - [Thu, 13 Mar 2025 07:15:15 UTC] \"POST /v1/usage-metrics HTTP/1.1 201 27.585665ms \"caddy-cs-bouncer/v0.7.0\" \""
time="2025-03-13T07:15:25Z" level=info msg="172.30.0.3 - [Thu, 13 Mar 2025 07:15:25 UTC] \"POST /v1/usage-metrics HTTP/1.1 201 26.295471ms \"caddy-cs-bouncer/v0.7.0\" \""
time="2025-03-13T07:15:30Z" level=info msg="Ip 199.45.155.74 performed 'crowdsecurity/http-bad-user-agent' (2 events over 5.572427664s) at 2025-03-13 07:15:30.399608511 +0000 UTC"
time="2025-03-13T07:15:30Z" level=info msg="(localhost/crowdsec) crowdsecurity/http-bad-user-agent by ip 199.45.155.74 (US/398722) : 4h ban on Ip 199.45.155.74"
time="2025-03-13T07:15:30Z" level=info msg="127.0.0.1 - [Thu, 13 Mar 2025 07:15:30 UTC] \"POST /v1/alerts HTTP/1.1 201 82.55284ms \"crowdsec/v1.6.5-72b4354b-docker\" \""
time="2025-03-13T07:15:31Z" level=info msg="received signal for discord config" @module=http-plugin
time="2025-03-13T07:15:35Z" level=info msg="172.30.0.3 - [Thu, 13 Mar 2025 07:15:35 UTC] \"POST /v1/usage-metrics HTTP/1.1 201 57.089802ms \"caddy-cs-bouncer/v0.7.0\" \""
time="2025-03-13T07:15:38Z" level=info msg="Signal push: 1 signals to push"```
#

i removed alot of the "time" lines to show anything valid

#

Unable to persist Local API key on container restart

gray ermine
#

It could be the WAL mode for sql but I dont think that will fix anything

gray ermine
#

@mild zodiac sorry for the ping, if you have a moment can you look at the posted logs?

iron pawn
gray ermine
#

I will try this when I am home today. Thanks

gray ermine
#

Unless I am doing it wrong, maybe the docker compose needs to point at that ip or in the caddy file.

gray ermine
gray ermine
#

{"level":"error","ts":1742442088.1099384,"logger":"crowdsec","msg":"failed to connect to LAPI, retrying in 10s: Get \"http://crowdsec:8080/v1/decisions/stream?startup=true\": dial tcp 172.30.0.2:8080: connect: connection refused","instance_id":"786a8844","address":"http://crowdsec:8080/","error":"failed to connect to LAPI, retrying in 10s: Get \"http://crowdsec:8080/v1/decisions/stream?startup=true\": dial tcp 172.30.0.2:8080: connect: connection refused"}

It does use correct address

gray ermine
heavy smelt
#

30s is the key

#

I have made it standard

gray ermine
#

yeah definitely, i tried with 45 and was not needed, ten failed also

#

now it starts every time

#

life saver for real, had this issue for so long

heavy smelt
#

I couldn't share it before because it's a closed forum. Some might think it's promoting

gray ermine
#

I signed up now, I will leave comment, I feel this needs to be added to crowdsec docs or somthing

heavy smelt
#

I use this principal in traefik also

gray ermine
#

Love the site by the way

heavy smelt
#

Thank you 🙏

hallow latchBOT
# hallow latch

Resolving Unable to persist Local API key on container restart