Re-resolving docker.io images is slow | Dagger | Page 1

thorny temple Oct 8, 2023, 1:39 PM

#

I'm working through the demos, of just running npm test / npm build in dagger, and noticing that everytime I run dagger run ... build.mts, that resolving the node:20.8.0-slim to a hash/image (which has already been cached) can take up to 10 seconds.

Which is weird but then also in the output, it's incorrectly attributed as only having taken 0.1s:

│ ▽ │ from node:20.8.0-slim
│ █ │ [0.11s] resolve image config for docker.io/library/node:20.8.0-slim
│ █ │ [0.02s] pull docker.io/library/node:20.8.0-slim
│ ┣ │ [0.01s] resolve docker.io/library/node:20.8.0-slim@sha256:8d26608b65edb3b0a0e1958a0a5a45209524c4df54bbe21a4ca53548bc97a3a5

So that looks fast, but when I ran it, the "resolve image config" timer actually spun up to ~10 seconds. I'm kinda wondering if docker.io has rate limiting / throttling--I know they have the "100 pulls / 6 hours", but it seems like even resolving node:20.8.0-slim --> sha256:... is taking awhile, even though I'm avoiding the actual image pull (as expected).

Usually it's ~3-5 seconds, but sometimes 10s. Fwiw I'm in the US midwest, with generally good Internet access.

Is there a way to cache the label -> sha resolution?

I'm hoping for docker-compose-ish speed, but so far each dagger run ... invocation takes ~30s each, even re-running "echo hello world" actions that are 100% cached, and this ~10s "resolve image config" is the biggest part of it.

Thanks!

west elm Oct 8, 2023, 2:14 PM

#

Hey Stephen, have you tried using the image digest instead of the label to validate that if that effectively reduces the resolution times?

west elm Oct 8, 2023, 3:13 PM

#

It also seems to me that the issue doesn't seem to be the image resolve itself but what actually happens before that (engine session creation). Could you share a bit more about your setup? Which OS and OCI runtime (Docker, lima, Rancher, etc) you're using?

west elm Oct 8, 2023, 3:43 PM

#

I'm also assuming you're on the latest v0.8.7 SDK. Is that correct?

thorny temple Oct 8, 2023, 8:29 PM

#

Yep, I'm on v0.8.7 👍

#

And yeah, that's a lot faster, if I switch between these two:

// const nodeImage = "node:20.8.0-slim"
const nodeImage = "node@sha256:c704ff2a5e27c3fc990418bf1f6c1b1ac6eb75471a7a2e2f192f401d211d2101";

the sha256 one is much faster, entire cached reruns complete in sub-two seconds, which is what I was expecting.

#

i'm on linux, ubuntu 23.04:

[I] $ docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:40 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:40 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[I] $ uname -a
Linux sh12 6.2.0-33-generic #33-Ubuntu SMP PREEMPT_DYNAMIC Tue Sep  5 14:49:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

west elm Oct 8, 2023, 8:33 PM

#

👍 we can check it out tomorrow to see what's happening with the label resolution.

thorny temple Oct 8, 2023, 8:48 PM

#

cool, thanks! fwiw here is a asciinema: https://asciinema.org/a/bM79lQjS58TeDoJRCUUMSqmC3 (actually see the 2nd recording, it shows the behavior better)

#

in this 1st recording, the label only takes up to ~5 seconds (i had been seeing 10s yesterday). and then immediate re-runs are actually as fast as the digest. but if i wait another ~few minutes, it'll take ~5 seconds again.

#

ah yeah, here is one with 3 runs of dagger run, where the 1st one takes 10 seconds on the label -> digest resolution (afaict), then the 2nd & 3rd runs get it cached and are quick: https://asciinema.org/a/mRLIaZ5R7eJbvcyNF6pQB8idA

west elm Oct 8, 2023, 9:43 PM

#

awesome, thx for sharing! will try to repro tomorrow. By any chance are you logged in to your dockerhub account via the docker login command?

#

as you mentioned before, just trying to rule out the dockerhub rate-limiting thing

west elm Oct 10, 2023, 4:44 PM

#

hmm still wasn't able to repro this. Were you able to confirm if you were logged in to docker hub @thorny temple ?

thorny temple Oct 13, 2023, 3:47 AM

#

hi @west elm , sorry for the delayed reply! have been playing with dagger in my free time; you're right, i was not logged into docker hub, which fwiw/afaict had never really affected my docker-compose usage (i very rarely run docker-compose build new images, only when we bump versions of node/pg, and then otherwise just mount code into the cached images for my day-to-day workflow)

#

but, yeah, doing docker login made all of my dagger run invocations way, way faster. and now every run is fast, instead of the sporadic "sometimes 2 seconds, sometimes 10 seconds"

#Re-resolving docker.io images is slow