#timeout connecting to dagger engine in WSL2/Ubuntu with podman/cgroup2/crun/netavark/pasta stack

1 messages · Page 1 of 1 (latest)

jaunty verge
#

Hello! I'm seeking guidance on troubleshooting dagger engine in a simple use case below.

I am seeing persistent timeouts when attempting to daggerize any golang app/module. I can pull/build/run other containers in this environment, (hello-world, ubuntu), and this problem occurs even for the hello-dagger project in the Quickstart:

tim@thinky:~/src/hello-dagger$ dagger init --sdk=go --source=./dagger
✘ connect 10m2.1s
! start engine: new client: context deadline exceeded
  ✔ starting engine 2.0s
    ✔ create 2.0s
      ✔ exec docker start dagger-engine-d95f50c1aa49026a 1.0s
      ┃ dagger-engine-d95f50c1aa49026a
  ✘ connecting to engine 10m0.0s
  ! new client: context deadline exceeded
Error: start engine: new client: context deadline exceeded

This environment is a WSL2 Ubuntu machine configured with a fairly bleeding-edge container stack composed of:

tim@thinky:~/src/wupdedup$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

tim@thinky:~/src/wupdedup$ dagger version
dagger v0.12.3 (registry.dagger.io/engine) linux/amd64

tim@thinky:~/src/wupdedup$ podman version
Client:       Podman Engine
Version:      5.1.2
API Version:  5.1.2
Go Version:   go1.22.5
Git Commit:   94a24974ab345324db1a1489c924af4b89d2d0e9
Built:        Sat Jul 27 18:46:31 2024
OS/Arch:      linux/amd64

tim@thinky:~/src/wupdedup$ crun --version
crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

tim@thinky:~/src/wupdedup$ pasta --version
pasta 2024_07_26.57a21d2
Copyright Red Hat
GNU General Public License, version 2 or later
  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Thanks!

Daggerize an example application | Dagger

Daggerize an example application

jaunty verge
# jaunty verge Hello! I'm seeking guidance on troubleshooting dagger engine in a simple use cas...

Forgot to mention that I have docker linked to podman per the podman instuctions and am using cgroups v2 exclusively:

tim@thinky:~/src/hello-dagger$ ls -la $(which docker)
lrwxrwxrwx 1 root root 15 Jul 27 16:52 /usr/local/bin/docker -> /usr/bin/podman
(failed reverse-i-search)`mount': con^Cn --version

tim@thinky:~/src/hello-dagger$ mount | grep -i cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

I wonder if there is more to getting dagger engine working with podman than linking docker-->podman client binaries. Could dagger be trying to access the docker.socket directly? (which of course, doesn't exist in my environment).

Podman | Dagger

Podman is CLI-compatible with Docker and therefore can be used by creating a symbolic link to the Podman executable in your system path and naming it docker:

sour venture
#

@jaunty verge I think the issue is that the engine is not being able to start at all

#

are you around now to help you troubleshoot?

#

dagger should work fine with podman in a linux machine. Having said that, you're using WSL2 and we currently are not testing this in our integrations tests so itis possible that something could have stopped working because of a Dagger or podman update

#

after you run the dagger init can you see a dagger engine container with podman ps -a?

there should be a dagger-engine-$id running. If that container is stopped, try getting the logs and sharing them here as it probably failed to start for some reason.

pallid tangle
#

I had a similar issue a while back which turned out to be because I was running Podman in rootless mode. I don't know if the state of Podman in WSL has improved since then. I seem to remember that rootful Podman in WSL straight up wasn't supported back then.

#

Based on your OP you seem to be running in rootful mode because netavark is not available without it

jaunty verge
#

So far I have never seen the d-e container listed in podman ps..

#

@sour venture i will try the test again and report back (with logs if possible). I'm heading into a phone call for 30min.

@pallid tangle I have not done all the steps required for rootless yet in this env, so I'm assuming it's still rooftul.

#

Ah, so I'm not seeing the dagger-engine container appear when I dagger develop or dagger init, however I am noticing something very strange when I watch -n1 podman ps: even before running dagger, every few seconds I see a dagger container pop up and disappear, a zombie of some sort. Will report more later..

sour venture
#

@jaunty verge you should see it with podman ps -a

jaunty verge
#

Indeed, after cleaning up podman containers, I'm able to connect to dagger-engine. 🙂 Thanks.

Now onto the next error. 😉

sour venture
#

Oh, ok. You probably had an old container in a weird state that was causing the issue

sour venture
jaunty verge
#

I'm now hitting what may be a golang problem. I've been seeing this in my project since before upgrading the env to bleeding-edge:

tim@thinky:~/src/wupdedup$ dagger init --sdk=go --source=./dagger
Setup tracing at https://dagger.cloud/traces/setup. To hide: export SHUTUP=1

✔ connect 9.7s
✔ moduleSource(refString: "."): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
✔ ModuleSource.withName(name: "wupdedup"): ModuleSource! 0.0s
✔ ModuleSource.withSDK(sdk: "go"): ModuleSource! 0.0s
✔ ModuleSource.withSourceSubpath(path: "dagger"): ModuleSource! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.3s
✘ ModuleSource.asModule(engineVersion: "latest"): Module! 4.7s
! failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "codegen --output /src --module-context-path /src/dagger --module-name wupdedup --introspection-json-path /schema.json" did not complete successfully: exit code: 1
Error: failed to generate code: input: moduleSource.withContextDirectory.withName.withSDK.withSourceSubpath.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "codegen --output /src --module-context-path /src/dagger --module-name wupdedup --introspection-json-path /schema.json" did not complete successfully: exit code: 1
#

Pretty sure it's not dagger issue, will dig in. I just updated my project from a 2 year old golang version to latest.

sour venture
#

Is there a chance you can connect your CLI with Dagger Cloud and send us a trace link?

#

That'd be easier to troubleshoot. You can get a free individual account

jaunty verge
#

Sure, it's a good time for me to learn how Dagger cloud works..

#

@sour venture does Dagger Cloud only do GitHub auth right now?

#

I didn't see any other options

sour venture
jaunty verge
#

So, I exported DAGGER_CLOUD_TOKEN and reran my dagger init command that failed, but don't yet see any traces in my account..

#

My org name is timblacksoftware

#

I suspect it's bc I'm still not yet running any pipeline/functions, and just running init

sour venture
#

Oh, I see. Well init should not fail though...

#

Are you running init in an empty directory?

jaunty verge
#

It's running in an existing golang module that I am "daggerizing"

sour venture
#

Gotcha. Can you check if running init in a new empty directory works? Just want to rule out if it's a specific error or something it fails all the time

jaunty verge
#

Sure

#

Also "failed to generate code" when running in an empty dir:

tim@thinky:~/src/wupdedup$ cd ..
tim@thinky:~/src$ mkdir dagger-test
tim@thinky:~/src$ cd dagger-test/
(failed reverse-i-search)`gar': cd dag^Cr-test/
tim@thinky:~/src/dagger-test$ dagger init --sdk=go --source=./dagger
Full trace at https://dagger.cloud/timblacksoftware/traces/9d94f8ef1a676a53bd3083fc158d7d15

✔ connect 2.1s
✔ moduleSource(refString: "."): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
✔ ModuleSource.withName(name: "dagger-test"): ModuleSource! 0.0s
✔ ModuleSource.withSDK(sdk: "go"): ModuleSource! 0.0s
✔ ModuleSource.withSourceSubpath(path: "dagger"): ModuleSource! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.1s
✔ ModuleSource.asModule(engineVersion: "latest"): Module! 7.7s
✔ Module.generatedContextDiff: Directory! 0.0s
✘ Directory.export(path: "/home/tim/src/dagger-test"): String! 31.8s
! failed to export: rpc error: code = Canceled desc = grpc: the client connection is closing
  ✘ export directory / to host /home/tim/src/dagger-test 31.8s
  ! failed to export: rpc error: code = Canceled desc = grpc: the client connection is closing
Error: failed to generate code: input: blob.diff.export resolve: failed to export: rpc error: code = Canceled desc = grpc: the client connection is closing
#

I had the suspicion that it's related to the --source=./dagger part. Some examples I've seen have that, and some do not. It's not creating this directory, either way.

#

Update: it works fine in a clean directory without the --source arg.

# 
tim@thinky:~/src/dagger-test$ cd ..
tim@thinky:~/src$ rm -rf dagger-test
tim@thinky:~/src$ mkdir dagger-test
tim@thinky:~/src$ cd dagger-test/
tim@thinky:~/src/dagger-test$ dagger init --sdk=go
Full trace at https://dagger.cloud/timblacksoftware/traces/001176f89eb3eee9eb0b31809e9d3011

✔ connect 1.9s
✔ moduleSource(refString: "."): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
✔ ModuleSource.withName(name: "dagger-test"): ModuleSource! 0.0s
✔ ModuleSource.withSDK(sdk: "go"): ModuleSource! 0.0s
✔ ModuleSource.withSourceSubpath(path: "."): ModuleSource! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.asModule(engineVersion: "latest"): Module! 1.3s
✔ Module.generatedContextDiff: Directory! 0.0s
✔ Directory.export(path: "/home/tim/src/dagger-test"): String! 0.3s

Initialized module dagger-test in .
tim@thinky:~/src/dagger-test$ l
LICENSE  dagger.gen.go  dagger.json  go.mod  go.sum  internal/  main.go
#

I've seen considerable conflicting, or at least confusing, advice in the docs WRT the first step in "daggerizing" a project. Basically, it's dagger init vs dagger develop and then there's the issue of whether --source is supposed to be supported or not. I read best practice is to use --source to isolate the "dagger things" from the rest of the project, which is what I wanted to do, but I cannot get it to work as shown above.

sour venture
jaunty verge
#

You mean --source without any dir specified?

sour venture
jaunty verge
#

Ah, I've done that, and it failed in my project dir.

sour venture
#

@jaunty verge any chance you can jump into a quick #911305510882513037 chat to see this together?

jaunty verge
#

Yep

#

Having trouble figuring out how to invite you..

#

I'm a little discord challenged..

#

I was able to open dev-audio, but couldn't see how to include you..

sour venture
#

Lol, for some reason the Discord app doesn't seem to be working for me. 1 sec

#

Lol Discord is having an outage

#

Let me send a meet link

jaunty verge
#

So, regardless of how I dagger init in my non-empty golang project, I get the "failed to generate code" issue, and StdErr shows:

Stderr:
Error: load package "dagger": err: exit status 1: stderr: go: inconsistent vendoring in /src:
        github.com/99designs/gqlgen@v0.17.49: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/Khan/genqlient@v0.7.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/cenkalti/backoff/v4@v4.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/go-logr/logr@v1.4.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/go-logr/stdr@v1.2.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/google/uuid@v1.6.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/grpc-ecosystem/grpc-gateway/v2@v2.20.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
.
.
.
sour venture
jaunty verge
#

So, it could perhaps be some problem with the vendor module versioning?

sour venture
#

we've managed to make it work. Seems like an intermittent timing issue with grpc connections when using podman + dagger + wsl2 and performing dagger init on an alredy existing codebase which makes the code generation take a bit.

Filing an issue now

#

I'll add it to the issue