Hey, I have the ExperimentalPrivilegedNesting: true as I am invoking the underlying dagger engine for my unit tests. It looks like when I have this flag enabled, it never caches the tests. As soon as I remove the dagger dependent tests and remove ExperimentalPrivilegedNesting flag, buildkit starts caching that step. Could this be a bug? I validated that it's not something my test is doing by disabling the test and keepin the flag on, it still didnt' cache. I had to remove the flag for it to cache again.
#`ExperimentalPrivilagedNesting` doesn't honor cache
brave arrow
round plinth
it shouldn't be the case @brave arrow.. since we're using ExperimentalPrivilegedNesting heavily in Dagger tests and we haven't observed that behavior. It'd be great if you could send us a repro case so we can look beter
brave arrow
Alright, let me put something together 🙂
brave arrow
Here it is. Apologies if it's too big. I'm re-using the same repro we had on another question I asked.
Notice that every run re-runs the tests. I tried
- Commenting out the test logic (empty test) - didn't cache
- Removing
ExperimentalPrivilegedNesting- cached
sharp yarrow
@brave arrow using the comments in the rtf file, it looks like this is the directory structure (??)
maybe you could send a tar or zip file 🙂
How should I run your project?
From the root with dagger run go run ci/main.go or from ci dir with dagger run go run main.go?
brave arrow
Hey @sharp yarrow , this is how it looks like. You can do go run main.go from the ci folder.
brave arrow
LMK if you want a zip or a tar also 🙂
sharp yarrow
LMK if you want a zip or a tar also 🙂
@brave arrow I ran your pipeline three times 👆
I saw what looked like caching on the second two.
Might be just caching of go build and not go test, is that what you mean? Everything finishing quickly in this example so a little tough to know what you're seeing.
I'm using Dagger Engine v0.6.2 and Go SDK v0.7.2
With ExperimentalPrivilegedNesting: false I can't run the pipeline
brave arrow
With `ExperimentalPrivilegedNesting: false` I can't run the pipeline
Try commenting out the content of the test and running with ExperimentalPrivilegedNesting: false . You may see what I mean
In the above video, I am seeing the behavior I'm referring to. In your subsequent runs, you can see the tests are being run again and again. I am expecting buildkit to be caching that WithExec and not re-run the test
With ExperimentalPrivilegedNesting: false I see that WithExec test step as CACHED
In your video, the subsequent test steps run fast because it's go caching them.
sharp yarrow
@brave arrow I'm thinking the thing that might be changing in your tests and thus invalidating cache is the tempfile name?
Error: Not equal:
┃ expected: "./settings3124339281.
┃ xml"
┃ actual : ""
ts, err := os.CreateTemp(".", "settings*.xml")
brave arrow
I thought the same. But this even happens if you change that test to print hello world
As long as I have ExperimentalPrivilegedNesting: true, I have not been able to get that Exec to cache. Even with simplifying the test.
high shell
@brave arrow what version of Dagger is this? we found an issue with cache busting due to host paths recently, but I think that's only in main. cc @grizzled bough
sharp yarrow
@high shell I'm on engine v0.6.2 and am seeing same issue (see video above) where everything is cached except for the go test ...
though for me, that's with ExperimentalPrivilegedNesting: true
(doesn't run at all for me with ExperimentalPrivilegedNesting: false from my laptop, I think Nipuna is running in a Docker agent in Jenkins)
@brave arrow if you can share that tarball now, would prob be ideal 🙂
brave arrow
Sure! I am running the same version as @sharp yarrow . And I am seeing this on my local laptop. Our Jenkins instance doesn't have local cache so this isn't relevant there yet.
high shell
ack, @grizzled bough found the issue - it's indeed a bug with nesting, we mount in a socket whose path changes on every run:
https://github.com/dagger/dagger/blob/0c8da0a136047f58930ba69d0b5779cfcffb93b7/core/container.go#L1056
-> https://github.com/dagger/dagger/blob/0c8da0a136047f58930ba69d0b5779cfcffb93b7/engine/engine.go#L423
brave arrow
That means your engine tests aren't actually caching either right? Fixing this could speed up your CI 😃
high shell
true! it would probably allow the tests to skip for non-code changes.
serene cloak
@coral fable ^^
coral fable
<@412836691720536065> ^^
Thanks for the mention! @high shell, did you have any ideas for what to name that socket, so it's more stable?
high shell
Thanks for the mention! <@108011715077091328>, did you have any ideas for what t...
We were just discussing modifying the API to support setting a name for sockets, instead of just using the path as the LLB ID. That way we can just set some internal const like progrock-socket here. Users might find it useful too, for the same reason, if they're forwarding a host socket in that has an unstable path
the socket path itself should probably remain as-is; it's like that to avoid conflicts, since it's ephemeral and created by every Dagger session
coral fable
👍🏻
sharp yarrow
Thanks again for reporting this @brave arrow looks like its going to be a great improvement for everyone! 🔍
high shell
opened https://github.com/dagger/dagger/issues/5373 for this
