Wondering if there is any prior art or experience with this.
The idea is to have a local registry, push with dagger, pull from kind
related links:
#dagger + kind, best setup for a great dx?
late ingot
I wonder, if instead of starting the registry on the dagger container network, we could get the same result in 4579 by doing a docker network connect like the kind page does to make the registry available to the cluster, but also still available on localhost (from the outside)
modern canopy
you can do that, but you'll have to change the engine configuration to push to a non-tls registry since dagger's engine (buildkit) config doesn't allow it by default
IMO, the simplest solution until we have c2h network is:
- Start registry locally
- Export image to tmp dir
- Call
docker import(or use the docker SDK ) - Call
docker push(or use docker SDK) to push to your local registry
it's more steps but doesn't require custom fidding with networks or dagger engine configs
late ingot
yeah, just hit the non-tls registry, I don't mind the network config'n, how can I set that for dagger?
I've been curious about how to configure the engine more generally
modern canopy
https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md
^ look for engine.toml
late ingot
does the default engine container contain both config files, and I will need to FROM that image and add the config files (or just mount them at runtime?)
oh, wait, the dagger.toml is a buildkit.toml
modern canopy
oh, wait, the dagger.toml is a buildkit.toml
yes, this
late ingot
hmm, getting a dns failure after adding the registry config, rather than a failed request
/ # cat /etc/dagger/engine.toml
debug = true
insecure-entitlements = ["security.insecure"]
[registry."registry:5000"]
http = true
insecure = true
[registry."local-registry:5001"]
http = true
insecure = true
docker network connect --alias registry "kind" "${reg_name}"
(reachable in k8s as local-registry:5001)
modern canopy
yes, but the engine won't resolve the registry by "local-registry` since it's a completely separate network
did you connect the registry to the engine network also?
late ingot
I connected both the registry and engine to the kind network
and use the --alias registry for the local-registry pod when connecting
so in theory, it should be reachable via registry:5000
I added local-registry as a check since k8s can reach it there and maybe DNS is now going through kind b/c docker network connecting
modern canopy
hmmm not sure if that will work since the engine has its dnsmasq server for Dagger service discovery
and that's probably avoiding the registry resolve that goes through docker's service discovery
can you ping the registry container ip from the engine?
and curl to the ip port 5000?
late ingot
ping ok, curl 200
modern canopy
yeah.. it's the DNS thing.. well.. I have a better hack for you
are you in linux?
or mac?
late ingot
yes, sort of, chromeos debian
modern canopy
k, if you start the engine with --add-host=host.docker.internal:host-gateway I think you can reach your registry with host.docker.internal as long as it's listening to 5000 in your local machine
you'll also need an exception in the engine.toml file
I recall doing that before Dagger services were a thing. I believe it should still work
late ingot
I haven't started the engine by hand before
modern canopy
I haven't started the engine by hand before
oh.. I was assuming you were doing that for the engine.toml thing
late ingot
docker exec && docker restart lol
my kind setup requires docker exec to add the registry setup, don't mind if dagger ends up the same way
modern canopy
docker run -d --add-host=host.docker.internal:host-gateway --privileged --name dagger-engine-691afa1f6ab7b8d7 registry.dagger.io/engine:v0.6.4
late ingot
oh, I could even remove the extra random chars in the container name
modern canopy
oh, I could even remove the extra random chars in the container name
the SDK's use that to detect if the engine is running
otherwise they'll start a new one
late ingot
is that hash tied to the engine version?
modern canopy
yes
just checked and host.docker.internal still works
so that should unblock you
late ingot
will I still need the engine.toml changes?
late ingot
hmm, not working for me, cannot curl either
modern canopy
ping host.docker.internal inside the engine works?
late ingot
yes
curl host.docker.internal:5000 -v
* Trying 172.17.0.1:5000...
* connect to 172.17.0.1 port 5000 failed: Connection refused
-p 5001:5000 since that was what the kind docs have for local registry setup
late ingot
curl fails for that too
/ # curl host.docker.internal:5001 -v
* Trying 172.17.0.1:5001...
* connect to 172.17.0.1 port 5001 failed: Connection refused
* Failed to connect to host.docker.internal port 5001 after 0 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to host.docker.internal port 5001 after 0 ms: Couldn't connect to server
curl localhost:5001 in your machine works?
late ingot
docker run -d -p "127.0.0.1:5001:5000" --restart=always --name local-registry registry:2
modern canopy
oh, remove the 127.0.0.1
late ingot
I can push an image with docker for host
modern canopy
since that makes it listen on the local interface only
late ingot
hmm, wonder if that will break kind
modern canopy
yes, since kind uses the internal port
late ingot
I didn't quite follow if the inc was needed
late ingot
pag!!! it works now
late ingot
ok, so new inception experiment, can I run the dagger engine / buildkit inside of kind?
so then I would have one outside of the kind cluster (attached as above), and then another inside the cluster, and then ideally I can then point dagger cli at either one?
I'd like to then see if I can run dagger as a k8s job, which would use the in cluster k8s buildkit / engine
modern canopy
ok, so new inception experiment, can I run the dagger engine / buildkit inside o...
I'd think so since kind containers are privileged
so it's like doing dagger-in-dagger which we're doing ourselves
modern canopy
so then I would have one outside of the kind cluster (attached as above), and th...
yep, you should be able to do this
late ingot
following this? https://github.com/dagger/dagger/pull/5446/files
modern canopy
we don't have official docs for that yet but yeah.. information is scattered across multiple places
but yeah.. you basically need to run the Dagger engine in a DaemonSet and then configure your jobs to use that engine
cc @little prawn might be the best person to ask about this since he has deep knowledge on self-hosted k8s dagger engines
@little prawn TL;DR about running Dagger inside kind
late ingot
well, the daemonset seems to be running, but I don't know how to use it (no ports exposed, do I have to mount a shared hostPath volume into a client container? or expose the port and create a service as well)
modern canopy
@little prawn would know 🙏
little prawn
well, the daemonset seems to be running, but I don't know how to use it (no port...
Yeah, we mount a shared hostPath volume. I'll grab a snippet.
I add this to the daemonset:
volumes:
- name: varrundagger
hostPath:
path: /var/run/dagger
And this mount:
volumeMounts:
- name: varrundagger
mountPath: /var/run/buildkit
And add it on the client container, as well.
You'll also want this env on the client container: _EXPERIMENTAL_DAGGER_RUNNER_HOST: "unix:///var/run/buildkit/buildkitd.sock"