[Solved] Dagger call can't connect to engine

[pseudo goblet]

hello, there 👋

I'm getting the these debug messages on repeat on the Dagger Engine.
(log attached as it was too long)
Running a Dagger engine on a kubernetes cluster (_EXPERIMENTAL_DAGGER_RUNNER_HOST flag set) and whenever I make a dagger call it can't connect to the engine.
Any ideas on how to solve the issue?

thank you!

Dagger call can't connect to engine

Running dagger v0.13.0

[strong ferry]

I'd def move to v0.13.1 that was released today.

[rich musk]

could you please describe a bit more your setup? Where are you trying to connect to the engine from? Is that a pod within the same k8s cluster? Or from your machine via the kube-pod scheme?

[pseudo goblet]

The dagger call is being made in a pod from a argo workflow on the same cluster as the engine.
It was all running fine until a few hours ago.
Bumped the dagger version to v0.13.1, removed the /var/lib/dagger after deleting the daemonset and then reapplying the helm chart but the problem still happens.

[rich musk]

It's the KCD module we were trying the other day?

[pseudo goblet]

It is not.
Since we couldn't get it to work as intended I moved to plan B.

[strong ferry]

Sorry! Let us know if we can help with anything else.
I just spun up a k3s cluster on rancher desktop, applied the Dagger helm chart, installed argo workflows, and used the sample here https://docs.dagger.io/integrations/argo-workflows/ (removed the Dagger Cloud token bit).

end of argo submit -n argo --watch ./workflow.yaml 👇

kubectl logs -n argo -f dagger-in-argo-lbvl5 👇

[pseudo goblet]

I should have explained a bit better :
Right now i'm running a talos linux cluster on digital ocean, with the dagger helm applied as well as argo workflows and events.
a workflow template gets triggered everytime a github trigger happens (in this case a push to a certain branch).
I haven't changed the machine configurations from the time it was working till now.

is there a way to change log level from the engine logs?

[rich musk]

hey @pseudo goblet if we make the k3s module do you still need this?

I can allocate some time in checking the KCD module again today

[pseudo goblet]

Buenos dias, @rich musk
I think that if we make the KCD module work we can make for a better demo, but I think that sending events from GitHub to a local cluster might get a bit trickier, altough not impossible.

[rich musk]

ok, gotcha. Lets get you unblocked first with your current cluster and then we can check the k3s stuff

do you have a few minutes to jump into #911305510882513037 to check it out together?

[pseudo goblet]

I can't right now, but if you can arrange a space in about 2 hours I can join you

[rich musk]

yes, sure. I'll be around

just let me know

in the meantime I'll check the KCD module again

the issue was that the client was taking too long to connect to the engine, correct?

[pseudo goblet]

It takes a bit too install the modules at start

I was testing out the KCD module and i'm getting the same logs in the engine as reported here

could it be related to dagger cloud cache?

[rich musk]

that's strange. Try removing the DAGGER_CLOUD_TOKEN and see if that fixes it?

[pseudo goblet]

removed right after the comment, but still getting the error

[rich musk]

👍 checking now

the KCD module works fine for me in v0.13.1 @pseudo goblet

[pseudo goblet]

🤔
can you run the dagger-workflow?

[pseudo goblet]

@rich musk can you spare a few minutes to debug this on #911305510882513037 ?

[rich musk]

I can now.

still around?

[pseudo goblet]

i have 30 min more or less

[rich musk]

@pseudo goblet managed to get the K3s module running way faster now

sending a patch here

here's what I changed:

diff --git a/KCD/main.go b/KCD/main.go
index aff37e1..df1785d 100644
--- a/KCD/main.go
+++ b/KCD/main.go
@@ -30,7 +30,10 @@ func New(
 }
 
 func (c *Kcd) CreateCluster(ctx context.Context) *Kcd {
-    c.KCDServer = dag.K3S(c.Name).Server()
+    kc := dag.K3S(c.Name).Container()
+    kc = kc.WithMountedCache("/var/lib/dagger", dag.CacheVolume("varlibdagger"))
+
+    c.KCDServer = dag.K3S(c.Name).WithContainer(kc).Server()
     return c
 }

LMK if that makes your module run faster 🙌

it works pretty fast to me

[pseudo goblet]

@rich musk, I managed to solve the error on this thread.
It was indeed the HostPath volume being misconfigured.
damn this yaml...

I'm marking this thread as solved and try the solution you proposed for the KCD module and continue in the thread #1284289915734982790

Thank you! 🙏