#dockerfile2llb2dagop

1 messages · Page 1 of 1 (latest)

mystic oyster
#

good suggestion on the sigquit, looks like we have a bunch stuck evaluating references:

goroutine 14904 gp=0xbcb8f54a960 m=nil [select, 1 minutes]:
runtime.gopark(0xbcb91a169c0?, 0x2?, 0xe0?, 0x8?, 0xbcb91a1694c?)
    /usr/lib/go/src/runtime/proc.go:462 +0xce fp=0xbcb91a167d0 sp=0xbcb91a167b0 pc=0x48e2ee
runtime.selectgo(0xbcb91a169c0, 0xbcb91a16948, 0xbcb914e4810?, 0x0, 0x366ae60?, 0x1)
    /usr/lib/go/src/runtime/select.go:351 +0xaa5 fp=0xbcb91a16900 sp=0xbcb91a167d0 pc=0x46a725
github.com/dagger/dagger/internal/buildkit/util/flightcontrol.(*call[...]).wait(0x4065a20, {0x4030bd8, 0xbcb91503170})
    /app/internal/buildkit/util/flightcontrol/flightcontrol.go:185 +0x606 fp=0xbcb91a16a50 sp=0xbcb91a16900 pc=0x29f15e6
github.com/dagger/dagger/internal/buildkit/util/flightcontrol.(*Group[...]).do(0x4065aa0, {0x4030bd8, 0xbcb91503170}, {0x3c8c96e, 0x6}, 0xbcb91508060)
    /app/internal/buildkit/util/flightcontrol/flightcontrol.go:96 +0x234 fp=0xbcb91a16a90 sp=0xbcb91a16a50 pc=0x29f0574
github.com/dagger/dagger/internal/buildkit/util/flightcontrol.(*Group[...]).Do(0x4065aa0, {0x4030bd8, 0xbcb91503170}, {0x3c8c96e, 0x6}, 0xbcb91508060)
    /app/internal/buildkit/util/flightcontrol/flightcontrol.go:54 +0x99 fp=0xbcb91a16b00 sp=0xbcb91a16a90 pc=0x29f09d9
github.com/dagger/dagger/internal/buildkit/solver/llbsolver.(*resultProxy).Result(0xbcb9133cee0, {0x4030bd8, 0xbcb91503170})
# 
...
github.com/dagger/dagger/engine/buildkit.(*filteringGateway).Solve(0xbcb91503470, {0x4030bd8, 0xbcb91503170}, {0x1, 0xbcb915027b0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...}, ...)
    /app/engine/buildkit/client.go:1067 +0x257 fp=0xbcb91a17000 sp=0xbcb91a16e30 pc=0x1584057
github.com/dagger/dagger/engine/buildkit.(*Client).Solve(0xbcb8ed58240, {0x4030c10?, 0xbcb91366000?}, {0x1, 0xbcb915027b0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
    /app/engine/buildkit/client.go:199 +0x5eb fp=0xbcb91a17238 sp=0xbcb91a17000 pc=0x157bd2b
github.com/dagger/dagger/core.(*Directory).Evaluate(0xbcb91368780, {0x4030c10, 0xbcb91366000})
    /app/core/directory.go:191 +0x14d fp=0xbcb91a172d0 sp=0xbcb91a17238 pc=0x2399d0d
github.com/dagger/dagger/core.getRefOrEvaluate[...]({0x4030c10, 0xbcb91366000}, 0xbcb91368780)
    /app/core/util.go:593 +0x5b fp=0xbcb91a172f8 sp=0xbcb91a172d0 pc=0x24c093b
github.com/dagger/dagger/core.(*Directory).WithDirectory(0xbcb9129a3c0?, {0x4030c10, 0xbcb91366000}, {0xbcb913cd500, 0x5}, 0xbcb914e3140, {{0x70f1780, 0x0, 0x0}, {0x70f1780, ...}, ...}, ...)
    /app/core/directory.go:1091 +0x130b fp=0xbcb91a176d0 sp=0xbcb91a172f8 pc=0x23a276b
github.com/dagger/dagger/core/schema.(*directorySchema).withDirectory(_, {_, _}, {{0xbcb914c4c40, 0x0, 0x1, 0x0}, {0x0, 0x1, 0xbcb8edd2030, ...}}, ...)
    /app/core/schema/directory.go:495 +0x1c9 fp=0xbcb91a17810 sp=0xbcb91a176d0 pc=0x2b359c9
github.com/dagger/dagger/core/schema.(*directorySchema).withDirectory-fm({_, _}, {{0xbcb914c4c40, 0x0, 0x1, 0x0}, {0x0, 0x1, 0xbcb8edd2030, 0xbcb8e69c018, ...}}, ...)
...
#

(I had to remove a few other layers in between since my message was too long.

hidden topaz
#

Ya, it's hard to tell in isolation whether those are truly part of the deadlock or not, but in the deadlocks I would hit there would be a couple stack traces similar to that where it seems like somehow two results were in a circular dependency somehow

mystic oyster
#

something circular does seem likely since I can't trigger it via a single test

hidden topaz
#

I'd just assume it's that for now. If we are creating a true circular dep in IDs somehow, if nothing else the new code will more concretely identify that and error out with a message that has the actual path that created the circular dep (IIRC, I added that 2 weeks ago I think)

#

But also highly possible it's not even a "real" circular dep and something is just going haywire in dagop or the buildkit solver, both of which are gone

mystic oyster
#

if you did want to squash my branch and try pulling it in that would be interesting to see if it goes away or not.

#

I moved some of the tests from util/llbtodagger/*_test.go into the TestDockerfile/ integration tests, and simply deleted the others. Not sure how useful it was to have the generic llb to dag tests vs an integration test that capture what our users will be using the code for (i.e. submitting a dockerfile vs custom llb)

hidden topaz
# mystic oyster I moved some of the tests from `util/llbtodagger/*_test.go` into the TestDockerf...

No the tests were added over time at each stage, but we don't need all of them at the end. Really the most important tests by far are the actual DockerfileSuite integration tests that use real dockerfiles. If at the end we had all of the same coverage provided by the other tests but purely as Dockerfile integ tests, that would be ideal.

If it makes sense to leave around a couple unit tests for quicker validation that's fine too, but not a big deal either way

hidden topaz
mystic oyster
#

sweet, in the meantime I'll take a final look to see if any of the tests I deleted could be moved over.

mystic oyster
#

dockerfile2llb2dagop

#

I came across an inconsistency this morning: currently COPY --exclude=... is broken in this branch due to excludePatternsEnabled = false. buildkit has https://github.com/alexcb/dagger/blob/795db6d39cbae4dbe75dad16129affaa0d3f09d7/internal/buildkit/frontend/dockerfile/instructions/exclude_pattern_feature.go#L7 which gets toggled at buildtime via a dfexcludepatterns flag; however it's unclear if we're setting that flag (at least we aren't when I run hack/build)

GitHub
dagger/internal/buildkit/frontend/dockerfile/instructions/exclude_p...

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - alexcb/dagger

mystic oyster
#

toolchains/go/main.go doesn't seem to support tags -- do we want to shave that yak, or should we just change the exclude_pattern_feature.go source code to always set this to true?

hidden topaz
mystic oyster
#

🪒

hidden topaz
#

If it's a rabbit hole then yes we can still maintain a fork for now

mystic oyster
#

yeah it shouldn't be that much of a detour.

#

I'll give it a go

hidden topaz
#

(aside: WHY are build tags the way this is configured... WHY??!! but that's neither here nor there)