OTEL exporter 429 responses for every request

[torn tapir]

Hi, I have a service that is setup to sent otel traces and logs to sentry through a collector sidecar (specifically the AWS ADOT container), however we aren't getting any traces through into the dashboard and the only error I'm seeing in the collector is sentry returning a 429 for every request.

I have tried to change the rate limit on the DSN through the project settings but its made no difference (seems to only be a rate limit for exception captures?) and we're batching all otel data before its sent to sentry so I'm at a bit of a loss for why its happening. Any help would be appreciated.

Extra info:
ASP.NET API with otel pipeline setup with exporter sending to the sidecar
ADOT container is pulling from latest version
Using sentry SaaS product.

[soft perch]

That is weird... are you sending to the regular DSN or the OTel endpoint?

[torn tapir]

through the otel endpoint like suggested here, https://docs.sentry.io/concepts/otlp/forwarding/pipelines/collector/ we are still on the trial but I dont imagine that should prevent us from going down the collector route

[soft perch]

@torn tapir how are you batching all otel data before sending it to sentry?

[torn tapir]

heres the full config setup, batch was copied from some blog I've lost now

extensions:
  health_check:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  filter/healthcheck:
    error_mode: ignore
    traces:
      span:
        - '(IsMatch(attributes["url.path"], "/health.*") or IsMatch(attributes["http.target"], "/health.*")) and status.code != STATUS_CODE_ERROR'
  probabilistic_sampler:
    sampling_percentage: ${env:TRACE_SAMPLE_PERCENTAGE}
  batch/traces:
    timeout: 10s
    send_batch_size: 2048
    send_batch_max_size: 4096
  batch/metrics:
    timeout: 60s
  batch/logs:
    timeout: 10s
    send_batch_size: 2048
    send_batch_max_size: 4096

exporters:
  otlphttp/sentry:
    endpoint: ${env:SENTRY_OTLP_ENDPOINT}
    headers:
      x-sentry-auth: "sentry sentry_key=${env:SENTRY_AUTH_KEY}"
    timeout: 30s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/healthcheck, probabilistic_sampler, batch/traces]
      exporters: [otlphttp/sentry]
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [otlphttp/sentry]
    logs:
      receivers: [otlp]
      processors: [batch/logs]
      exporters: [otlphttp/sentry]

  extensions: [health_check]

[soft perch]

What’s you org slug @torn tapir?

[torn tapir]

exizent

eu region

[soft perch]

@torn tapir I see there are spans coming through, so you are getting data in

try turning off spike protection and see if that helps: https://exizent.sentry.io/sentry.io/settings/spike-protection/

but you should turn it back on later, you don't want to keep spike protection off

[torn tapir]

oh yeah can see some now from an hour ago, okay will look into otel logs again tomorrow to double check. I had been stupid and had setup both the otel collector and the direct push through the sdk so perhaps I just got in touch too soon after turning the direct push off

thanks for help so far

[dusty gazelle]

sentry will also reply with a 429 when you are out of spans quota.

[torn tapir]

yeah did double check that, but we are on trial plan atm so still have like 80 million spans available

it seems to have resolved itself overnight, I can see traces coming through on the apps I expect

I am suspecting that it might be due to me being daft and enabling both the SDK tracing AND the collector tracing and the 429s were due to us sending every trace twice. Now the SDK has been disabled and its had time to resolve the rate limit the collector isnt reporting any 429s and we can see traces now

so, tldr: user error 😂 thanks for the help