405 method not allowed

[rapid bay]

👋 sorry about this! A few quick questions:

• Do you have anything in your network setup that might explain the issue? Like a weird corporate proxy, slow internet or equivalent?

• Did you have any calls that did not experience the 405 errors, and if so, any changes that could explain it?

Thanks!

[near cape]

I'm using "Little Snitch" but had it disabled during the testing. I'm in my home network which is very standard.

Using Fiber internet in Berlin.

[rapid bay]

🤔 @versed river @naive zodiac @spring sun @red frost are we aware of any reliability issues at ingestion?

[versed river]

not aware of any, no - just tried now and it seems OK

from the screenshot, I see it's hitting http://api.dagger.cloud instead of https://api.dagger.cloud for some reason

i don't see anything in our code that returns 405, so maybe it's hitting a http->https redirect handler that only wires up GET? just a guess. my main question is what's switching it from https:// to http://

[maiden shard]

@near cape does this work for you?

curl https://api.dagger.cloud/health

[maiden shard]

can you please set the env variable DAGGER_CLOUD_URL to https://api.dagger.cloud and try again please? Just trying to rule out if that will fix it 🙏

[maiden shard]

ok, this is definitely strange... @near cape any chance you have a few minutes in 10 to check it together in #911305510882513037?

[red frost]

Nothing that a backoff & retry wouldn't fix.

[red frost]

The fact that this is trying to connect to http is suspicious, but even then the redirect should catch this: httpstat http://api.dagger.cloud/v1/traces -X POST

@near cape what does dig api.dagger.cloud +short return for you?

FWIW: https://status.dagger.io/ (no issues reported for Dagger Cloud API in the last 90 days)

[maiden shard]

yeah, strange. First time I've seen this kind of issue. Makes me think that it must be related to the user setup somehow 🤔

[red frost]

I have seen something similar for Daggerverse.dev, where the users were connecting via ISP proxies, not the real origins. If you are seeing anything other than these 2 IPs for api.dagger.cloud @near cape, then there is some other endpoint in-between which is misbehaving:

dig api.dagger.cloud +short
3.95.80.43
52.20.62.189

[maiden shard]

right, but I'd have expected the curl to also fail in this case 🤔

[red frost]

Only if curl was sending a POST request to the http://api.dagger.cloud/v1/traces endpoint

[maiden shard]

yeah.. I see. I guess we need to check what this returns instead? curl -v -L -X POST http://api.dagger.cloud/v1/traces ?

[near cape]

can't do calls right now, sorry

[maiden shard]

np @near cape ! thx for sharing this 🙏

by any chance you don't have any OTEL_* env variables set in your environment, don't you?

[near cape]

I do

[maiden shard]

I see.. can you try unsetting those?

[near cape]

Yay, that fixed it. I do need those env vars for my app though.

[versed river]

what vars were set? maybe disabling TLS / overriding the protocol?

[near cape]

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://${DEVSERVER_HOSTNAME}:4318/v1/traces"
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://${DEVSERVER_HOSTNAME}:4318/v1/traces"

[maiden shard]

ok so in summary, as Dagger also uses those variables, I guess the best solution here is to make sure that those variables are set only for your app and not exported globally so other processes also don't use them

[near cape]

Given I'm using direnv in the root of my monorepo this would cause quite a bit of inconvenience. Maybe dagger could prefix their env vars with DAGGER_?

[maiden shard]

we could add a prefix but we'd like to stick to the standard as much as we can and also chaning the variable, means breaking backwards compatibility with existing tools . I guess the other option is unsetting the variables before calling dagger? i.e: set -u OTEL_EXPORTER_OTLP_TRACES_ENDPOINT -u OTEL_EXPORTER_OTLP_METRICS_ENDPOINT dagger. Not sure if direnv allow defining aliases / functions to overcome this 🤔