#Pulling dagger engine images occasionally fails

1 messages · Page 1 of 1 (latest)

meager ether
#

Hey there, running dagger in a github action and the past week or so I occasionally see errors when pulling the engine image. Ex

14
2   : ┆ starting engine
3   : ┆ ┆ create container
6   : ┆ ┆ ┆ exec docker pull registry.dagger.io/engine:v0.19.6 ERROR [41.7s]
6   : ┆ ┆ ┆ ! failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: exit status 1
6   : ┆ ┆ ┆ [41.7s] | unknown: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://registry.dagger.io/v2/engine/blobs/sha256:1bba5a5f9f9aaab6f5226b7272c0d6f5848d3e7c8886631d2854d415e98124bd: 500 Internal Server Error

2   : ┆ starting engine
3   : ┆ ┆ create container
6   : ┆ ┆ ┆ exec docker pull registry.dagger.io/engine:v0.19.6 ERROR [9m60s]
6   : ┆ ┆ ┆ ! failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: signal: killed
3   : ┆ ┆ create container ERROR [10m0s]
3   : ┆ ┆ ! failed to pull image: failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: signal: killed
2   : ┆ starting engine ERROR [10m0s]
2   : ┆ ! failed to pull image: failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: signal: killed
1   : connect ERROR [10m3s]
1   : ! start engine: failed to pull image: failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: signal: killed

I don't have exact logs atm but I've also seen 504 gateway timeouts as well.

Wondering if there's any known issues with the registry, or if perhaps we're abusing some rate limiting that we're unaware of.

cinder wasp
meager ether
#

Will do. It happens as our end to end tests run throughout the day and it has been sporadic over the last week and a half or so. All that to say is I’ll keep an eye on it over the rest of the week and let you know

meager ether
#

just got another one

2   : ┆ starting engine
3   : ┆ ┆ create container
6   : ┆ ┆ ┆ exec docker pull registry.dagger.io/engine:v0.19.6 ERROR [1m0s]
6   : ┆ ┆ ┆ ! failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: exit status 1
6   : ┆ ┆ ┆ [1m0s] | Error response from daemon: unknown: failed to resolve reference "registry.dagger.io/engine:v0.19.6": unexpected status from HEAD request to https://registry.dagger.io/v2/engine/manifests/v0.19.6: 500 Internal Server Error
cinder wasp
meager ether
#

Thank you! I’ll trigger a re run of our test suite. It’s intermittent so no guarantees it’ll trigger any issues but worth a shot

cinder wasp
#

I've disabled the offending region there and around 20m ago and haven't seen more 500's since then

image.png
meager ether
#

Awesome, thank you very much. The last tests I ran all made it past the dagger engine's image pull so looks like it's working well here at the moment

meager ether
#

bumping this again, but we're seeing intermittent time outs on the registry this morning

6   : ┆ ┆ ┆ exec docker pull registry.dagger.io/engine:v0.19.6 ERROR [15.0s]
6   : ┆ ┆ ┆ ! failed to run command [docker pull registry.dagger.io/engine:v0.19.6]: exit status 1
6   : ┆ ┆ ┆ [15.0s] | Error response from daemon: Get "https://registry.dagger.io/v2/": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
cinder wasp
cinder wasp
#

@meager ether I think should be better now. LMK how's that looking