Error `Error Post http dagger query EOF` | Dagger | Page 1

wanton burrow Feb 3, 2023, 2:38 PM

#

With new go 1.20 release our dagger pipelines started to fail and when we rollback go version to 1.19 everything returned normal. I tried to reproduce same error with new pipeline it didn't happend again.

#

go version go1.19.3 linux/amd64

› mage -v test:unit
Running target: Test:Unit
...
#1 DONE 0.1s

#2 copy / /
#2 DONE 0.3s

#3 copy /go.mod /
#3 CACHED

#4 copy /go.sum /
#4 CACHED

#5 resolve image config for docker.io/library/golang:1.19
#5 DONE 1.7s

#6 mkdir /meta
#6 DONE 0.1s

#7 docker-image://docker.io/library/golang:1.19
#7 resolve docker.io/library/golang:1.19
#7 resolve docker.io/library/golang:1.19 0.2s done
#7 DONE 0.4s

#7 docker-image://docker.io/library/golang:1.19```

#

This is a normal execution with 1.19

#

go version go1.20 linux/amd64

› mage -v test:unit                                   
Running target: Test:Unit
...
#1 DONE 2.0s

#2 copy / /
#2 DONE 0.3s

#3 copy /go.mod /
#3 DONE 0.1s

#4 copy /go.sum /
#4 DONE 0.6s
Error: Post "http://dagger/query": EOF
Please visit https://dagger.io/help#go for troubleshooting guidance.```

#

Exact same code when we run with go 1.20

#

CONTAINER ID   IMAGE                          COMMAND                  CREATED              STATUS              PORTS     NAMES
a1201e79b1f5   ghcr.io/dagger/engine:v0.3.7   "buildkitd --oci-wor…"   About a minute ago   Up About a minute             dagger-engine-9ad18aca71b1b1ba```

it seems I didn't  bump dagger version for the repo. I'll try to update dependencies latest version and try again

rough hound Feb 3, 2023, 2:59 PM

#

cat_confused

wanton burrow Feb 3, 2023, 3:06 PM

#

I forget to mention docker image version also same. Only bumped local go version

rough hound Feb 3, 2023, 3:12 PM

#

let me do a test really quick with Go 1.20

#

a very simple "hello world" program works in Go 1.20 and latest Dagger engine.. I'll try running our test suite

wanton burrow Feb 3, 2023, 3:19 PM

#

Bumping dagger engine to latest version fixed this issue. I didn't get same issue with ghcr.io/dagger/engine:v0.3.10

rough hound Feb 3, 2023, 3:21 PM

#

which version were you using before? Trying to repro

wanton burrow Feb 3, 2023, 3:21 PM

#

but I still wonder why we're getting this error. 1.20 shouldn't introduce any breaking changes as far as I can see.

wanton burrow Feb 3, 2023, 3:21 PM

#

rough hound which version were you using before? Trying to repro

let me check

scarlet blade Feb 3, 2023, 3:22 PM

#

I think another user has an issue related to this one. I thought it was windows stuff but this might be the root cause

wanton burrow Feb 3, 2023, 3:22 PM

#

dagger.io/dagger v0.4.2

#

I had another issue related with go 1.20 from a library we're using. This could be the similar dependency issue

#

we had panic: Something in this program imports go4.org/unsafe/assume-no-moving-gc to declare that it assumes a non-moving garbage collector, but your version of go4.org/unsafe/assume-no-moving-gc hasn't been updated to assert that it's safe against the go1.18 runtime. If you want to risk it, run with environment variable ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH=go1.18 set. Notably, if go1.18 adds a moving garbage collector, this program is unsafe to use.

#

one of our dependencies was indirectly using go4.org/unsafe/assume-no-moving-gc in their code and this was causing panix

rough hound Feb 3, 2023, 3:36 PM

#

Just tried hello-world against 0.4.2 and seems to be running ok as well 🤔

scarlet blade Feb 3, 2023, 3:37 PM

#

scarlet blade I think another user has an issue related to this one. I thought it was windows ...

Fake alert on my side, the other user had an unrelated issue

wanton burrow Feb 3, 2023, 3:38 PM

#

let me do the same with mesosphere/daggers as well, maybe it's something we're injecting

rough hound Feb 3, 2023, 3:38 PM

#

this user has the same issue @scarlet blade (https://discord.com/channels/707636530424053791/1070997951503151104). Also in v0.4.2

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

#

however, I couldn't repro with a simple example and 0.4.2

wanton burrow Feb 3, 2023, 3:43 PM

#

I just tried with our repo and didn't able to produce it again

rough hound Feb 3, 2023, 3:43 PM

#

😕

#

maybe some weird go pkg compatibility between go 1.19 and go 1.20 🤷‍♂️

scarlet blade Feb 3, 2023, 3:45 PM

#

wanton burrow I just tried with our repo and didn't able to produce it again

Was the engine re-spawned in between ? The last time we had such error message was because of the state of the container engine (stopped for example). But since then, we added the fix to print the error messages

wanton burrow Feb 3, 2023, 3:47 PM

#

I tried multiple times with new engine or exisiting engine instances. Error was consistent for that project

#

but couldn't re-produce in other projects

rough hound Feb 3, 2023, 3:49 PM

#

https://giphy.com/gifs/the-x-files-3oEdv5M3JrgUqJ3VQc

Giphy

The X-Files

wanton burrow Feb 3, 2023, 3:52 PM

#

I just rollback the fix in the repo and re-produce the error.

#

Do you have any hint to debug this error

rough hound Feb 3, 2023, 4:00 PM

#

@wanton burrow by rollback the fix what do you mean exactly?

#

the unsafe lib thing you mentioned above?

wanton burrow Feb 3, 2023, 4:01 PM

#

I'm just curious and returned to using dagger v0.4.2

#

trying to see what is break to app.

rough hound Feb 3, 2023, 4:01 PM

#

ok, I see so it happens on a single repo only?

wanton burrow Feb 3, 2023, 4:02 PM

#

yes, I'm able to reproduce the error because I forget to bump dependencies here

rough hound Feb 3, 2023, 4:02 PM

#

can you try go clean -cache and see if still happens?

wanton burrow Feb 3, 2023, 4:05 PM

#

interesting, it passed previous error point first but later it failed with same error.

#

But now again after cleaning everything including docker, error consistent

#

time="2023-02-03T16:10:49Z" level=debug msg="remove snapshot" key=a3pnf2678hnh2qw8fsy4dyuws snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="remove snapshot" key=ky13d6x635szkhqgvp2u076q9 snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="remove snapshot" key=m4ifraund99pseap731l6ubyw snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="remove snapshot" key=w5ogr03j22gzyqg5wzkg0fh35 snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="remove snapshot" key=x7qr5jqyi5jht905l8idi9vzd snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="schedule snapshotter cleanup" snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="removed snapshot" key=buildkit/26/m4ifraund99pseap731l6ubyw snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="removed snapshot" key=buildkit/31/ky13d6x635szkhqgvp2u076q9 snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="removed snapshot" key=buildkit/34/x7qr5jqyi5jht905l8idi9vzd snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="removed snapshot" key=buildkit/45/a3pnf2678hnh2qw8fsy4dyuws snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="removed snapshot" key=buildkit/41/w5ogr03j22gzyqg5wzkg0fh35 snapshotter=overlayfs
time="2023-02-03T16:10:49Z" level=debug msg="snapshot garbage collected" d=102.083581ms snapshotter=overlayfs
time="2023-02-03T16:10:50Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/ReadFile returned error: rpc error: code = Canceled desc = context canceled\n"```

#

I runned couple this sequentially. It started to execute the tests and in the middle it failed again. This is the last logs from engine

rough hound Feb 3, 2023, 4:15 PM

#

strange that this is happening in go 1.20 only 🤔

wanton burrow Feb 3, 2023, 4:15 PM

#

anyway, at least it's solved by new version. If you want to look something I can recreate issue anytime

wanton burrow Feb 3, 2023, 4:15 PM

#

rough hound strange that this is happening in go 1.20 only 🤔

yes, it's pretty unstable in 1.20 for our workflow

rough hound Feb 3, 2023, 4:16 PM

#

I'll defer this to @old matrix since it seems to be happening in the internals of buildkit for some reason

wanton burrow Feb 3, 2023, 4:17 PM

#

salute sure, just let me know if you want to debug this again

old matrix Feb 3, 2023, 6:29 PM

#

I think those buildkit logs are just saying that the client closed its connection, which in this case is probably the dagger session binary. The stuff about snapshot removal and garbage collection are normal.

#

But yeah very strange, glad that the latest version fixes it somehow but still would love to know why

wanton burrow Feb 3, 2023, 7:32 PM

#

didn't deep dive the debugging but I saw biggest difference between 0.4.2 and 0.4.4 seems dependencies used in dagger. I think one of this dependencies is the issue.

#Error `Error Post http dagger query EOF`