#[SOLVED] Dagger fails to run with an EngineSessionError

1 messages · Page 1 of 1 (latest)

tardy haven
#

I was running my dagger pipeline just fine a few days ago. The trouble started when I updated from 0.9.7 to 0.9.9 and now my pipeline will no longer run. I removed the dagger-engine container and pruned all volumes to try to get dagger to a clean slate but that did not work. This is the output I get when trying to run the pipeline:

Downloading CLI... OK!
Creating new Engine session... OK!
Establishing connection to Engine... failed to resolve image digest: error getting credentials - err: exec: "docker-credential-pass": executable file not found in $PATH, out: ``
falling back to leftover engine
Error: new client: no fallback container found
OK!
file:///var/home/bojan/projects/lambda-sol/mind-central/mind-central-web/node_modules/@dagger.io/dagger/dist/provisioning/bin.js:187
                throw new EngineSessionError(e.stderr);
                      ^

EngineSessionError
    at file:///var/home/bojan/projects/lambda-sol/mind-central/mind-central-web/node_modules/@dagger.io/dagger/dist/provisioning/bin.js:187:23
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: undefined,
  code: 'D105'
}

Node.js v20.9.0

I should note that my host OS is fedora silverblue with podman as the container engine and I have symlinked podman to docker.

#

Running from an ubuntu distrobox gives me a different error:

⬢ [Systemd] ❯ dagger run node ops/test.mjs
┻                                                                                                                                                 
█ [0.70s] ERROR connect
┣ [0.01s] pulling registry.dagger.io/engine:v0.9.9
┻ 

⧗ 0.74s ✔ 1 ✘ 1
new client: failed to pull image: Cannot connect to the Docker daemon at unix:///run/podman/podman.sock. Is the docker daemon running?
: exit status 1

Is it possible this is a breaking change in podman and this started happening after a system update I did?

thin latch
#

@tardy haven what happens if you manually run docker pull registry.dagger.io/engine ?

tardy haven
# 
❯ docker pull registry.dagger.io/engine
Trying to pull registry.dagger.io/engine:latest...
Error: initializing source docker://registry.dagger.io/engine:latest: reading manifest latest in registry.dagger.io/engine: manifest unknown

Hmm, this is weird

thin latch
#

try pulling :v0.9.9?

tardy haven
#

That worked but it looks like I still have issues running dagger, seems to be related to docker socket tho. I am not sure how/why/when this stopped working for me but podman desktop is also reporting that docker socket is not reachable.

podman info contains this bit:

  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock

So depending on how I try to run dagger I get different errors, I am kinda stuck here tbh. If I try running from my host:

❯ dagger run ops/test.mjs 
failed to resolve image digest: error getting credentials - err: exec: "docker-credential-pass": executable file not found in $PATH, out: ``
                                                                                                                                            falling back to leftover engine
                         ┻                                                                                                                        █ [0.03s] ERROR connect
┻ 

⧗ 0.07s ✔ 1 ✘ 1
new client: no fallback container found

If I try running from an ubuntu distrobox container:

⬢ [Systemd] ❯ dagger run ops/test.mjs 
┻                                                                                                                                                 
█ [0.61s] ERROR connect
┣ [0.01s] pulling registry.dagger.io/engine:v0.9.9
┻ 

⧗ 0.66s ✔ 1 ✘ 1
new client: failed to pull image: Cannot connect to the Docker daemon at unix:///run/podman/podman.sock. Is the docker daemon running?
: exit status 1
#

I even spun up a local k8s cluster via minikube (using the podman driver) and installed dagger via helm onto that, then made this small shell script, called kdagger:

#!/bin/bash

DAGGER_ENGINE_POD_NAME="$(kubectl get pod \
    --selector=name=dagger-dagger-helm-engine --namespace=dagger \
    --output=jsonpath='{.items[0].metadata.name}')"

_EXPERIMENTAL_DAGGER_RUNNER_HOST="kube-pod://$DAGGER_ENGINE_POD_NAME?namespace=dagger" exec dagger $@

Trying to run that I get:

❯ kdagger run ops/test.mjs 
█ [0.00s] ERROR ops/test.mjs
┻ 
• Engine: dagger-dagger-helm-engine-hvg4v (version v0.9.3)
⧗ 0.20s ✔ 3 ✘ 1
fork/exec ops/test.mjs: permission denied

Also looks like the helm chart is a bit behind the latest release? (0.9.3 vs 0.9.9)

#

Does dagger depend on more than just the docker CLI? Looks like it needs docker-credential-pass now? Is this new?

thin latch
tardy haven
#

well dang

#

I just can't make sense of this

thin latch
#

did you upgrade podman recently?

#

Might be worth looking at your local docker config file, too

thin latch
#

this kind of error reminds me of when I run the docker cli from a linux container, with the docker config mounted from a mac host. The helpers configured on the mac are not available in the container, resulting in a similar error. I don't know how this applies to your situation, though.

#

I'm looking for evidence of a discrepancy between the docker CLI's environment (os, available helpers) and its configuration file

tardy haven
#

Yes, I did update podman, I noticed it was part of a system update I did recently

#

Thanks @thin latch your suggestion of looking into the docker config helped me figure it out. I had tried out rancher-desktop recently by installing it into my ubuntu container and looks like that messed up my docker config. Deleting the ~/.docker directory did the trick and everything is working again now. Sorry for the false alarm, this had nothing to do with dagger.

#

[SOLVED] Dagger fails to run with an EngineSessionError

thin latch