#Making proxy easy?

1 messages · Page 1 of 1 (latest)

distant crag
#

I'm trying to onboard two teams into using dagger. The issue: README.md is starting to get too complicated.

Since there's no dagger configuration that would inject http_proxy vars into the engine or the "runners"(?) you need to bake a custom docker-engine. Since I still don't have a registry I've included a Dockerfile with BUILD_ARG into the .dagger directory and ask people to build the engine locally. So far, almost so good.

Issue is in WSL, not to pass username/pass around, people install local proxies in their Windows host. The only way I've found to access the host from docker on WSL/Linux is to run docker with the dreaded --add-host=host.docker.internal:host-gateway. So they will always have to run the engine "manually" and it would be awesome to configure some aspects of this in the dagger.json file, for example.

Just for future reference, since some people may also need more special configurations, you would need to ask people to run something like:

# every time you would like to run dagger, please be sure that the engine is running with:
docker run --rm -d \
  --name dagger-engine-custom \
  --privileged \
  -v /path/to/cacert.pem:/usr/local/share/ca-certificates/proxy.crt:ro \
  -v /path/to/engine.toml:/etc/dagger/engine.toml \
  --add-host=host.docker.internal:host-gateway
  -e http_proxy=http://host.docker.internal:3128 \
  -e https_proxy=http://host.docker.internal:3128 \
  -e no_proxy="host.docker.internal,*.internal.corp,localhost,127.0.0.1" \
  -e _DAGGER_ENGINE_SYSTEMENV_GOPROXY="https://internal-artifactory.corp/go-proxy/" \
  -v /var/lib/dagger \
  registry.dagger.io/engine:v0.18.14

And then probably:

export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine-custom

I know it's 2025... But people don't know how to use Docker still. 😐

How would you simplify these for the general mass?

potent karma
distant crag
#

Corporate, yeah. You don't have to installl a local proxy but that means exposing your password in environment vars - therefore: cntlm or px or whatever.

WSL can, yes. Only for docker to access the windows host is a bit tricky. Need to add that --add-host option.

potent karma
# distant crag Corporate, yeah. You don't have to installl a local proxy but that means exposin...

is setting the proxy username+password in the engine directly is a no-go? first time I personally see this cntlm double proxing approach to avoid passing the credentials to the underlying system that needs them (Dagger in this case)

we could add a way to easily set the http_proxy env var in the engine.json while also automatically adding the --add-host flag to the engine container given that by default that's the way it works in Docker Desktop. I think that'd help to simplify your current setup doesn't it?

distant crag
#

Yes,adding http[s]_prpxy vars to the json would help. And yeah, adding the host automatically on WSL (not sure about macos) would also help a lot too.

#

If I could reference the http_proxy as an environment variable on the json would be better since they can differ by os and everyone is used to setting the var up - env vars are the norm for proxy setting. Don't get me wrong, I'm not talking about just using the http_proxy env var directly since in WSL it would be localhost:3128 and inside a container host.docker.internal:3129 or something else. But just being able to use whatever env var like $proxy, for example.

potent karma
# distant crag If I could reference the http_proxy as an environment variable on the json would...

I was mostly thinking of the same approach docker uses to enable http_proxy as defined here: https://docs.docker.com/engine/cli/proxy/#configure-the-docker-client

I guess we could also add a way to use existing vars if present withing the same context the Dagger CLI is invoked when launching the engine

Docker Documentation
Proxy configuration

How to configure the Docker client CLI to use a proxy server

distant crag
#

Rebuilding the dagger image for every dagger new version is proving to be a concrete blocker to adopt dagger in the department. I'm unable to handle the load of people having issues with the building process... 😕

Without having this config in the engine.json I would need to create a general mirror image repository and somehow make some magic to standardize the way people configure their proxies. It's close to impossible...

potent karma
#

is that because you're bundling your certificates within the image?

distant crag
distant crag
potent karma
distant crag
#

As a local docker image… Also after a system prune… So, yes, very disruptive.

distant crag
#

@restive nexus hi, yes, I'm using docker – colima actually. why do you ask?

potent karma
# distant crag <@310795041931264001> hi, yes, I'm using docker – colima actually. why do you as...

I think he's refering to this PR (https://github.com/dagger/dagger/pull/11621/changes) which seems to also address your needs @distant crag

GitHub
feat: engine client docker driver proxy env handling by frantjc · ...

allows Dagger to &quot;just work&quot; with its default configuration on machines that need http(s) proxies e.g.
HTTPS_PROXY=http://localhost:8080 dagger
will spawn an engine container with...

restive nexus
#

yeah, I was checking which CLI you use to see if the above PR would work for you. It doesn't try to solve this problem for nerdctl or finch, but should for docker and podman

distant crag
#

interesting, I think it would work for mac but not sure about WSL. there seem to be some issue with host.docker.internal... I think window devs on my team decided to port forward using iptables or something and refer to the WSL host with its own IP or something like that.

#

else every time you run any docker image you need to add some --some_parameter host.docker.internal

#

(on WSL that is)

#

or can you pass the hostAddr somehow...? I think I didn't get the usage howto; sorry

restive nexus
#

you can't pass hostAddr in the current implementation in that PR: that was one of the shortcomings that @potent karma pointed out. I'll get around to another implementation sometime soon, hopefully!

distant crag
potent karma
distant crag
distant crag
#

Better than needing to do docker build though.

restive nexus
distant crag
restive nexus
#

If you are interested, my actual solution was to use mise to manage this env var on a per-directory basis. That was easy for me because we had already adopted mise (and actually use it to install dagger for our project, too!)

#

I know that such tooling bloat may not be desirable; just offering options 🙂

distant crag
distant crag
#

somehow stopped working on 0.20.6... didn't test since 20.3 I think

# 
failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:8b3bb9f277c68dec5095a27186575d58a544cc6ac09de1e0fd9129a6ae87b5f2?se=2026-04-16T14%3A25%3A00Z&sig=sMzflArabq37n8On05ciub4d5SnVWWNCuV
Im0zkMDA0%3D&ske=2026-04-17T04%3A17%3A59Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-04-16T04%3A17%3A59Z&sktid=398a6654-997b-47e9-
b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=[somebigstring]
061d9df042bc0a": Parent proxy unreacheable
restive nexus
#

I looked briefly and didn't see anything rm'd in v0.20.6 that would have caused this. Can you docker inspect the v0.20.6 to see if the ?env= from your _EXPERIMENTAL_DAGGER_RUNNER_HOST made it into the container config?

distant crag
distant crag
#

some issue with colima/docker and proxy it seems, sorry for the false alarm... it was only with a specific request

distant crag
# restive nexus If you are interested, my actual solution was to use mise to manage this env var...

How do you set the _EXPERIMENTAL_DAGGER_RUNNER_HOST to have the dagger version in the mise.toml file? Probably you would use that:

[env]
DAGGER_VERSION = { value = "{{ tools.dagger.version }}", tools = true }
_EXPERIMENTAL_DAGGER_RUNNER_HOST = "image://registry.dagger.io/engine:${DAGGER_VERSION}?env=https_proxy=http://localhost:3128"

I'll need to test it soon... But if you have the solution ready, would be great if you could paste it here. Also the localhost:3128 will change per person... I'll need to find a way so that each individual would be able to create a local env file with the proxy string.

restive nexus
#

I can't paste the exact solution here as the machine that has access to it does (can) not have Discord on it.

But, to give you an idea, I personally used something like:

[env]
_.source = "./my-script.sh"

and:

#!/usr/bin/env sh
# my-script.sh
export _EXPERIMENTAL_DAGGER_RUNNER_HOST="image://registry.dagger.io/engine:$(dagger version | awk-or-something)"

With that said, I like your idea better. Let me know if you get it to work. Additionally, you could use something like this to inject your per-user proxy variable:

[env]
_.file = '.env'
distant crag
#

Thanks for the idea – you're great. 🙂
I'll post my version here for future ref. too when I'm done.

distant crag
#

it's excruciating harder to work behind a proxy...

potent karma
distant crag
#

also the vendoring of the golang modules when dagger is starting could be precompiled/predownloaded, no?

#

probably not call vendoring... there was something like this in the past. anyhow, when you download modules for the engine runtime or something like that

potent karma
#

I guess the right solution here is to make the engine proxy configuration as smooth as possible. What is it not working for you currently?

distant crag
#

Currently I'm facing two issues, one edge case with colima that one of the layers of the engine cannot be downloaded:

failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:1fa4f3ae8c534bbaa1ac2ca503c1485bcf05e0a69a0960e3df4b857f3be1ba03?se=2026-04-24T17%3A10%3A00Z&sig=yada,yada": Parent proxy unreacheable

I'm trying to debug this.

The other is during dagger initialization:

go: dagger.io/dagger@v0.19.11: Get "https://proxy.golang.org/dagger.io/dagger/@v/v0.19.11.mod": proxyconnect tcp: dial tcp: lookup host.internal.docker on 10.x.z.1:53: no such host

Probably also something with colima having issues to find out about host.internal.docker... The strange thing is that the virtual machine can reach these contents using the proxy with curl/wget... So it's just that the way these are called from within docker runtime and the go get way are somehow not working.

#

Let me try podman...

#

Podman solved the one layer download issue.

#

The golang issue persists... I'll debug it more.

#

any way for me to try to execute this part of the build in isolation? I mean, in a shell?

codegen generate-typedefs --module-source-path /src --module-name python-sdk --introspection-json-path /schema.json --lib-version 7058e9313c720d82c6a07fefb6ce3fab60c7ec4e --output typedefs.json
#

(thanks for minding) 🙂

potent karma
#

once you do that, the engine should inject those variables in all the containers that it starts

#

including the Go dagger initialization ones

potent karma
distant crag
#

mise.toml

[env]
MISE_EXPERIMENTAL = 1
DAGGER_VERSION = { value = "{{ tools.dagger.version }}", tools = true }
DOCKER_PROXY = { required = "Read docs/container runtime.md I'm using http://host.internal.docker:3128" }
NO_PROXY = "localhost,127.0.0.0,127.0.1.1,127.0.1.1,local.home"
_EXPERIMENTAL_DAGGER_RUNNER_HOST = { value = "image+docker://registry.dagger.io/engine:v{{env.DAGGER_VERSION}}?env=https_proxy={{env.DOCKER_PROXY}}&env=http_proxy={{env.DOCKER_PROXY}}&env=HTTPS_PROXY={{env.DOCKER_PROXY}}&env=HTTP_PROXY={{env.DOCKER_PROXY}}&env=no_proxy={{env.NO_PROXY}}&env=NO_PROXY={{env.NO_PROXY}}", tools = true }
distant crag
potent karma
distant crag
#

it's not...

#

the other one may be though.

#

Let me bake the image locally...

potent karma
distant crag
#

With just the _EXPERIMENTAL_lalala the image doesn't have the proxy vars, as expected. Somehow it's injected at runtime, should be fine.

❯ docker image inspect registry.dagger.io/engine:v0.20.6 | jq '.[] | .Config.Env'                                                                                                                                       feature/mise !2
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "DAGGER_GO_SDK_MANIFEST_DIGEST=sha256:d6615bb79f543d3b049233456b6f5c988e92b8d5c34ca22ccfec21165a4769a4",
  "DAGGER_PYTHON_SDK_MANIFEST_DIGEST=sha256:118dc4c28da806bb5ae745a9812efa083c22551911eadbbfa67a15b93ffa7dcb",
  "DAGGER_TYPESCRIPT_SDK_MANIFEST_DIGEST=sha256:2494b010f864e03e018eb58c42a087e26d51023bf59955567baae080d70ed967",
  "_EXPERIMENTAL_DAGGER_RUNNER_HOST=unix:///run/dagger/engine.sock"
]
#

The issue is that it's not working... So I need to try something.

potent karma
distant crag
potent karma
#

ok so it should still work. Let me try really quick

#

have you tried removing your engine container and trying again?

distant crag
#

Trying this now

#

It pulls the image fine but fails in the generate-typedefs. Not sure why it would do this dns lookup on this 10.x.x.x:53...

#

I'll try with orbstack...

potent karma
#

@distant crag just tried the env stuff in the image and seems to be working ok

distant crag
#

It is, indeed

potent karma
distant crag
# 
go: dagger.io/dagger@v0.19.11: Get "https://proxy.golang.org/dagger.io/dagger/@v/v0.19.11.mod": proxyconnect tcp: dial tcp: lookup host.internal.docker on 10.x.x.1:53: no such host
Error: ensure dagger package: failed to get dagger.io/dagger@7058e9313c720d82c6a07fefb6ce3fab60c7ec4e: exit status 1
#

✘ withExec codegen generate-typedefs --module-source-path /src --module-name python-sdk --introspection-json-path /schema.json --lib-version 7058e9313c720d82c6a07fefb6ce3fab60c7ec4e --output typedefs.json (

potent karma
#

are you setting host.internal.docker as the http proxy?

distant crag
#

http://host.internal.docker:3128, yes

potent karma
#

that's the issue then

#

host.internal.docker doesn't work in linux unless you add a specific flag to the container

#

it doesn't resolve to anything

distant crag
#

On windows I knew there were issues with it... --add-host=host.docker.internal:host-gateway was needed... But on macOs always worked... I'll try to find out the host IP and see how it goes.

#

With the linux VM, I mean... Somehow colima/podman solved this.

potent karma
distant crag
potent karma
#

try setting your host's IP. That should work

distant crag
#

Worked... 😐
That's sad. The linux machine is able to use thi host alias... Not go, though.

distant crag
distant crag
#

From the chapter: "it's always dns"