#"grpc: received message larger than max"

1 messages · Page 1 of 1 (latest)

vapid basin
#

I have a pipeline running with dagger 0.9.6 as the client against a dagger-engine running 0.9.7. I saw this error, even when the job was repeated multiple times upon failure, but runs based on new commits didn't show this again, so far:

goroutine 324 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x10df4a0?, 0xc0002e0208?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
github.com/dagger/dagger/engine/client.(*Client).ServeHTTP(0xc000002000, {0x1403010, 0xc0005e80e0}, 0xc000330100)
    /app/engine/client/client.go:535 +0xbf2
net/http.serverHandler.ServeHTTP({0x13ff0b8?}, {0x1403010?, 0xc0005e80e0?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0006aa870, {0x1407f48, 0xc000020e10})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 176
    /usr/local/go/src/net/http/server.go:3086 +0x5cb
panic: unexpected EOF

looking at my code, this pipeline was my first attempt at using dagger, and it's doing a dagger.Container.File().Contents() to fetch a very large file. I'm probably going to switch this to FIle().Export() because I can simplify this functionality but wanted to report it.

#

I also saw a different error for that same HTTP server in another pipeline:

goroutine 27325 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb0
panic({0xbbed00?, 0x1800cc0?})
    /usr/local/go/src/runtime/panic.go:920 +0x26c
github.com/dagger/dagger/engine/client.(*Client).ServeHTTP(0x400021a480, {0xf4ec40, 0x4000830380}, 0x4001216100)
    /app/engine/client/client.go:535 +0xa3c
net/http.serverHandler.ServeHTTP({0xf4ac78?}, {0xf4ec40?, 0x4000830380?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0xbc
net/http.(*conn).serve(0x4000700bd0, {0xf53be8, 0x40004074d0})
    /usr/local/go/src/net/http/server.go:2009 +0x518
created by net/http.(*Server).Serve in goroutine 52
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

in this pipeline, my code is publishing a container to a remote gitlab registry:

                if err != nil {
                        panic(err)
                }
                fmt.Println("packaged " + label + " published to: " + addr)

the "published to:" line is printed out and images appear to be uploading correctly (still need to verify this) so it doesn't look like my code is panic'ing based on an error returned from Publish.

let me know if you want a bug report on this, or more information. I'd be interested if there are configuration tweaks as far as remote publishing retries/timeouts as I can imagine that as more pipeline in parallel that timeouts and such may increase.

livid merlin
#

hm this sounds quite suspicious to me - out of curiosity, do you know if this was an issue on older dagger versions?

#

huh, this is actually surprising to me - i would have thought if this issue was going to show up, it would have showed up in v0.9.8

#

since we had some changes to the grpc message limits then

#

could you try upgrading the client+server to v0.9.8 to see if you see the same issue?

vapid basin
#

didn't see it before, the difference is the dagger-engine daemon / gitlab-runner combo that we just setup last week

vapid basin
#

updated client and server to 0.9.8. haven't seen it so far

livid merlin
#

ok, phew ❤️

#

if you do see it again, then please let me know! (or open a github issue)

#

this kinda thing is exactly the kind of stuff i love lol 😄

vapid basin
#

yeah I didn't change my client code to verify this, but I want to. it's pulling a base64 encoded file out, and it really doesn't need to be base64 encoded..

vapid basin
#

I'm seeing this again @livid merlin . Running 0.9.11. I'll open an issue because discord keeps crashing on my desktop

livid merlin
#

aha thanks ❤️
i'll try and set aside some time to investigate this, i've got a little pile of bugs to dive into once the next release lands 😄

livid merlin
#

this should make it to the next release (which should be some time next week)
but in the meantime, as a workaround, you can Export the file to the local filesystem, and then use os.ReadFile to get the raw contents