#Dependency upon registry.dagger.io

1 messages · Page 1 of 1 (latest)

arctic hound
#

I've got a bunch of Dagger pipelines (using the Go SDK) that have been running great for weeks, and suddenly several of them started failing this morning with the following error:

failed to resolve image digest: GET https://registry.dagger.io/token?scope=repository%3Aengine%3Apull&service=ghcr.io: unexpected status code 503 Service Unavailable
falling back to leftover engine
Error: no fallback container found

Is there any way to remove this dependency from our Dagger pipelines, or maybe I should be adding a fallback container to the pipelines for when this service is down?

high prairie
#

Sorry about that.. You can provision the engine yourself and configure your client to connect to it. It’s an experimental env variable for now (still finalizing the UI) but works fine

arctic hound
#

Ah, cool. No problem. Thank you, @high prairie !

sick mantle
#

Hi 👋 ,
Sorry to hear that. Everything seems to be working fine on our side, especially as it runs from several regions. Maybe a ghcr issue this morning ? Or a Fly one ?

I am checking the logs atm, and the only odd thing comes from these requests:

2023-03-15T16:18:30.855 app[9336a7f4] ams [info] {"level":"warn","ts":1678897110.8551507,"logger":"fallback","caller":"redirect/redirect.go:218","msg":"request without Authorization header, getting auth"}

2023-03-15T16:18:31.024 app[9336a7f4] ams [info] {"level":"info","ts":1678897111.024534,"logger":"fallback","caller":"redirect/redirect.go:222","msg":"Error response getting token: 403 403 Forbidden"}

2023-03-15T16:18:31.419 app[9336a7f4] ams [info] {"level":"warn","ts":1678897111.419208,"logger":"fallback","caller":"redirect/redirect.go:218","msg":"request without Authorization header, getting auth"}

2023-03-15T16:18:31.567 app[9336a7f4] ams [info] {"level":"info","ts":1678897111.5675333,"logger":"fallback","caller":"redirect/redirect.go:222","msg":"Error response getting token: 403 403 Forbidden"}

This comes directly from https://github.com/dagger/registry-redirect/blob/main/pkg/redirect/redirect.go#L218. Is your docker client authenticated when downloading the engine ?

Otherwise, we'll continue exploring 🙏

image.png
maiden bobcat
#

Sorry for the trouble @arctic hound, we are looking into improving this service. I can confirm that this is the second time that we have seen this issue in the last two weeks. I will dig further into the root cause.

_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://ghcr.io/dagger/engine:v0.4.0 <COMMAND-THAT-INVOKES-SDK> is the workaround command. It is important that the Engine version is the correct one for the SDK. This is where that is declared for the Go SDK: https://github.com/dagger/dagger/blob/sdk/go/v0.5.0/sdk/go/internal/engineconn/version.gen.go

image.png
arctic hound
#

No problem. These were all initiated from GitHub Actions, so maybe the GitHub server was having outbound issues.

maiden bobcat
#

I am digging further into our own event streams to confirm this.

arctic hound
#

Thank YOU @maiden bobcat and team! You all are amazing! Keep up the great work.

maiden bobcat