ingress nginx kubernetes + custom logs | CrowdSec | Page 1

warm bobcat Sep 1, 2025, 11:00 PM

#

The logs are being parsed, but no further events occur.
Nginx has custom logs, but I created a parser.

Could you please advise me on how to resolve this issue?

torpid shadowBOT Sep 1, 2025, 11:00 PM

#

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

warm bobcat Sep 1, 2025, 11:07 PM

#

I have followed the troubleshooting steps, but unfortunately the issue still persists.

worn ore Sep 2, 2025, 7:02 AM

#

warm bobcat The logs are being parsed, but no further events occur. Nginx has custom logs, b...

Can you provide the full output with -v passed to the explain command, this will show the log line details so if you have any PII you would need to sed the output or sanitize before sending.

warm bobcat Sep 2, 2025, 8:11 AM

#

here is the output of log processing

📎 explain_v.log

#

and custom parser

📎 custom_parser.yaml

worn ore Sep 2, 2025, 8:25 AM

#

warm bobcat here is the output of log processing

Okay parsing seems fine, but there no matching scenarios, so if you run cscli scenarios list from within the pod (agent) do you see any installed?

warm bobcat Sep 2, 2025, 8:35 AM

#

but if I check the raw log from Docker, the transition to other parsers does not happen. Could that be the issue?

📎 explain_docker.log

#

scenarios

#

📎 scenarios.log

worn ore Sep 2, 2025, 8:55 AM

#

warm bobcat but if I check the raw log from Docker, the transition to other parsers does not...

That is not docker logs that is containerd

#

so you should be running cscli explain ... --type containerd --labels "program:nginx"

ref: https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/cri-logs

#

Also your parser needs to use evt.Parsed.message instead of evt.Line.Raw as the raw line include the containerd prefixes

warm bobcat Sep 2, 2025, 9:32 AM

#

update parser

📎 cri.log 📎 metrics.log 📎 nginx-type.log

#

cri log - cscli explain ... --type containerd --labels "program:nginx"
nginx log - cscli explain ... --type nginx

#

kubectl -n crowdsec logs pods/crowdsec-agent-q4txc --tail 10
Defaulted container "crowdsec-agent" out of: crowdsec-agent, wait-for-lapi-and-register (init)
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key proxy does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key upstream does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key http does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key http does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key proxy does not exist"
time="2025-09-02T09:36:07Z" level=error msg="JsonExtractObject: key upstream does not exist"

#

create evt.Parsed.http_referer : [32mhttps://new.site.online/ship <-- from "http": {"referer": "https://new.site.online/ships/celestyal-discovery","user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36","accept_encoding": "gzip, deflate, br, zstd"}

((((((

worn ore Sep 2, 2025, 9:39 AM

#

let me cook up a parser using UnmarshalJSON

#

its faster, and more runtime efficient

worn ore Sep 2, 2025, 10:13 AM

#

Hey @warm bobcat

what is the intention from these groks?

nodes:
    - grok:
         pattern: '%{GREEDYDATA}(?<endpoint>Error obtaining Endpoints for Service)%{GREEDYDATA}'
        expression: evt.Line.Raw
    - filter: evt.Parsed.endpoint != nil
       onsuccess: next_stage
       statics:
         - meta: log_type
            value: dev-null-endpoint

warm bobcat Sep 2, 2025, 10:17 AM

#

this is an attempt to not process logs containing the string Error obtaining Endpoints for Service

worn ore Sep 2, 2025, 10:20 AM

#

warm bobcat this is an attempt to not process logs containing the string Error obtaining End...

ahhh, okay my updated parser will just ignore them anyways

warm bobcat Sep 2, 2025, 10:22 AM

#

this line appeared out of desperation and attempts to fix log parsing (

worn ore Sep 2, 2025, 10:24 AM

#

filter: "evt.Parsed.program startsWith 'nginx' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') in [nil, '']"
name: crowdsecurity/custom-nginx-logs
description: "Parse nginx access logs in JSON format"
onsuccess: next_stage
statics:
  - target: evt.StrTime
    expression: evt.Unmarshaled.nginx["@timestamp"]
  - parsed: status
    expression: int(evt.Unmarshaled.nginx.status)
  - parsed: body_bytes_sent
    expression: evt.Unmarshaled.nginx["body_bytes_sent"]
  - parsed: request_length
    expression: evt.Unmarshaled.nginx["request_length"]
  - parsed: request_time
    expression: evt.Unmarshaled.nginx["request_time"]
  - parsed: request
    expression: evt.Unmarshaled.nginx.request
  - parsed: remote_addr
    expression: evt.Unmarshaled.nginx["remote_addr"]
  - parsed: verb
    expression: evt.Unmarshaled.nginx["request_method"]
  - parsed: time_local
    expression: evt.Unmarshaled.nginx["time_local"]
  - parsed: http_version
    expression: evt.Unmarshaled.nginx["server_protocol"]
  - parsed: target_fqdn
    expression: evt.Unmarshaled.nginx.vhost
  - parsed: http_referer
    expression: evt.Unmarshaled.nginx.http.referer
  - parsed: http_user_agent
    expression: evt.Unmarshaled.nginx.http["user_agent"]
  - parsed: proxy_upstream_name
    expression: evt.Unmarshaled.nginx.proxy["upstream_name"]
  - parsed: proxy_alternative_upstream_name
    expression: evt.Unmarshaled.nginx.upstream.addr
  - parsed: port
    value: "-"
  - parsed: remote_user
    value: "-"
  - meta: source_ip
    expression: "evt.Parsed.remote_addr"
  - meta: http_status
    expression: "evt.Parsed.status"
  - meta: http_path
    expression: "evt.Parsed.request"
  - meta: http_verb
    expression: "evt.Parsed.verb"
  - meta: http_user_agent
    expression: "evt.Parsed.http_user_agent"
  - meta: target_fqdn
    expression: "evt.Parsed.target_fqdn"
  - meta: service
    value: http
  - meta: log_type
    value: http_access-log

#

This works for me and output is

$ cscli explain -f test.log --type containerd --labels "program:nginx"
line: 2025-09-02T08:27:15.486046104Z stdout F {"@timestamp": "2025-09-02T08:27:15+00:00", "_ym_uid": "1755531811525907277" , "request_id": "386ac2fb4058ca99df87c92fca92dfaa", "remote_addr": "45.82.64.192","time_local": "02/Sep/2025:08:27:15 +0000","request": "/discounts?_rsc=1omfx","request_length": 1002,"request_time": 0.020,"request_method": "GET","status": 200,"vhost": "new.site.online","body_bytes_sent": 2034,"namespace": "site-online","ingress_name": "new-site-online","server_protocol": "HTTP/2.0","service": {"name": "ithaca-production","port": "3000"},"proxy": {"protocol_addr": "-","upstream_name": "site-online-ithaca-production-3000","add_x_forwarded_for": "45.82.64.192"},"http": {"referer": "https://new.site.online/ships/celestyal-discovery","user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36","accept_encoding": "gzip, deflate, br, zstd"},"upstream": {"addr": "10.112.189.44:3000","response_length": 2051,"response_time": 0.006,"status": 200,"location": "-","connect_time": 0.001,"response_length_string": "2051","response_time_string": "0.006","status_string": "200","connect_time_string": "0.001"}}
        ├ s00-raw
        |       └ 🟢 crowdsecurity/cri-logs (+10 ~9)
        ├ s01-parse
        |       └ 🟢 crowdsecurity/custom-nginx-logs (+25 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
        |       ├ 🟢 crowdsecurity/public-dns-allowlist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                └ 🟢 crowdsecurity/http-crawl-non_statics

#

ohh wait I just need to update it

#

updated it, by default all numbers are floats so we have to wrap status in int() call

warm bobcat Sep 2, 2025, 10:50 AM

#

The parser worked perfectly! Thank you!
What was my mistake? Why didn’t it move on to the next parser?

#

and...
crowdsec-agent-6vn8b:/# cscli decisions list
No active decisions

#

📎 metrics.log

#

time="2025-09-02T10:56:43Z" level=warning msg="failed to run filter : unexpected end of JSON input (1:42)\n | evt.Parsed.program startsWith 'nginx' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') in [nil, '']\n | .........................................^" id=wandering-snow name=crowdsecurity/custom-nginx-logs stage=s01-parse

warm bobcat Sep 2, 2025, 12:30 PM

#

Hooray! You’re amazing! We managed to fix it! HUGE THANKS!

We needed to override the variable in the Helm chart container_runtime: containerd.
Unfortunately, I couldn’t find any documentation on this issue.

worn ore Sep 2, 2025, 12:50 PM

#

warm bobcat Hooray! You’re amazing! We managed to fix it! HUGE THANKS! We needed to overrid...

Sorry yes I should have added this part, as I did mention it wasnt docker 😄

warm bobcat Sep 2, 2025, 12:58 PM

#

in the agent logs there are errors
is this normal or should it be fixed?

time="2025-09-02T12:54:04Z" level=error msg="UnmarshalJSON : invalid character 'W' looking for beginning of value" line="W0902 12:54:04.346346 7 controller.go:1107] Error obtaining Endpoints for Service "logs/devnull": no object matching key "logs/devnull" in local store"

time="2025-09-02T12:54:04Z" level=warning msg="failed to run filter : invalid character 'W' looking for beginning of value (1:42)\n | evt.Parsed.program startsWith 'nginx' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') == nil\n | .........................................^" id=throbbing-sound name=crowdsecurity/custom-nginx-logs stage=s01-parse

worn ore Sep 2, 2025, 1:00 PM

#

warm bobcat in the agent logs there are errors is this normal or should it be fixed? time="...

Probably you have non json logs being printed?

if so you can add to the filter like

filter: "evt.Parsed.program startsWith 'nginx' && trim(evt.Parsed.message) startsWith '{' && UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, 'nginx') in [nil, '']"

so in short we check the message starts with { indicating a json log, depending on it also we can check for stdout or stderr from containerd

warm bobcat Sep 2, 2025, 1:12 PM

#

yes, the agents are monitoring nginx through which many services pass. It turned out that there are non-normalized logs there.

warm bobcat Sep 2, 2025, 1:32 PM

#

cool! the fix works
once again, huge thanks

torpid shadowBOT Sep 2, 2025, 1:32 PM

#

torpid shadow

Resolving ingress nginx kubernetes + custom logs

#ingress nginx kubernetes + custom logs