Dagger engine registry

[hasty crag]

Hello there,

I am currently playing around with Dagger within our companies Github Enterprise where we are limited to the companies docker registry. So my very basic Dagger pipeline fails with

Downloading CLI... OK!
Creating new Engine session... OK!
Establishing connection to Engine... 1: connect
1: > in init
1: starting engine 
1: starting engine [6.89s]
1: connect ERROR: new client: failed to run container: Unable to find image 'registry.dagger.io/engine:v0.8.4' locally

For me it looks like there is no access to images from foreign registries but the companies registry is able to proxy images from some of the more widespread registries like dockerhub or ghcr. So my question is.

1. Is the docker engine image available on other registries than registry.dagger.io?

2. Is there a way to instrument the dagger client to pull the engine from another location?

Thanks for that awesome project! I already had a lot of fun with it and could do some proof of concepts for some of our ci problems we face here every day. Regards, Sebastian

[cerulean ore]

Hi! There's information how setting up registry mirrors in https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#registry-mirrors.

[hasty crag]

Hello @cerulean ore thank you for the link. I think I basically understand how to run a runner at our site. I must admit I do not fully understand what the error message means. Unable to find image 'registry.dagger.io/engine:v0.8.4' locally Is this the docker pull from within the runner that has no access to the engine image? Sorry for my lack of understanding but I am not so experienced with docker internals and not sure if I understand the concepts explained in operator_manual correctly.

[tulip hatch]

Hey @hasty crag!

Is the docker engine image available on other registries than registry.dagger.io?

Yes, the image is published to ghcr and you can see it here: https://github.com/dagger/dagger/pkgs/container/engine

Is there a way to instrument the dagger client to pull the engine from another location?

When the CLI starts it tries to create the default runner, the error that you are seeing means that its not able to find the base image due to your internal registry restrictions.

It looks like right now this registry is hard coded in the CLI so if you want to use a runner outside of this registry it looks like you need to start up a custom image and then set the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable to point to the new runner as shown here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#connection-interface

Once that is done you should be able to run dagger commands and things should work as expected.

Let me know if that makes sense and works for you, if you are still running into issues could you share your entire Dagger pipeline so that I could try to reproduce it?

[hasty crag]

Hello @tulip hatch, many thanks for this detailed description. It makes totally sense to me now. I will try to implement those steps in our companies setup but that might take a while. I will definitely respond here if this works or not. I was wondering if it would be possible to replace the hard coded reference to the registry with a further dagger.ClientOpt field in an upcoming version? But I don't know if this would be the correct place.

[tulip hatch]

Hey @hasty crag my pleasure, glad that this is a bit more clear now. Please do let us know if this works for you in your environment.

I think making this configurable is a good idea, I started a GitHub issue to discuss this idea here: https://github.com/dagger/dagger/issues/5775

Please follow along and add any additional comments that you might have about my proposed approach.

[hasty crag]

As promised, I've tried installing the Dagger engine in our company's Kubernetes cluster. Fortunately an official Helm chart came out in the meantime that I used with the official guide (https://docs.dagger.io/194031/kubernetes/). Everything worked fine so far. The engine pods are up and running. My question now is: is it possible to address the Dagger engine (pods) over TCP instead of a unix socket (for temporary testing purposes until I have adjusted the github runner part)? Does the Dagger engine run on a specified TCP port which i can expose?

[crystal meadow]

yes, you can also access it through a TCP port if needed. You need to add the argument --addr $IP:PORT to your engine pod and then you can connect through your Dagger pipelines by setting the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable to tcp://$HOST:$PORT

[hasty crag]

Hi @crystal meadow as statet [here](#1144765715174391879 message) I can also put the tcp addr inside the toml file right? Sorry should have used the discord search earlier.