let’s start a thread 🙂
#Using Dagger for Nix CI?
topaz sable
Lots of output -> input connections between CUE values.
modest urchin
Using Dagger for Nix CI?
topaz sable
@vast lichen Nixery is interesting (almost done with your video)
modest urchin
Realizing that I don’t have a good understanding of the use case. What would dagger do in the end?
vast lichen
I think the most natural mapping would be that nodes in the DAG correspond to individual Nix builds enriched with dependency information, as well as what we currently call "extra steps" (doing something impure with the build results).
We could then use Dagger to portably execute our CI DAG
modest urchin
I actually think you could pull this off without code generation
vast lichen
via the JSON route or something else?
an example of what we currently output might be useful, it's the *-chunk-*.json files that are listed in the "Artifacts" tab on the llama here: https://buildkite.com/tvl/depot/builds/12085#96097023-a872-4c94-bb5e-91171617b9bd
The format of what comes out there is essentially a Buildkite build step; that'd probably map to some Dagger-native structure instead.
modest urchin
I think you could manually author a DAG for Dagger , that dynamically fetches the list of artifacts to build and test, and then goes ahead and does it.
So in that sense, yes it would be like gg, dynamically adding nodes to the DAG
for example, you could fetch a git repo (or load a local checkout), scan for those chunk-*.json files (or if they’re generated, call the tool that generates them), then iterate over them, parse each into the cue tree and execute the corresponding build & test tasks
vast lichen
interesting, and the code that does that would also be written in Cue?
(Nix generates the chunk files)
modest urchin
Well it’s wired together declaratively in CUE, but it can wrap shell scripts, invocation of nix tools etc
inside containers run by buildkit per the CUE spec
You can move data back and forth between containers and cue as the DAG is executed
run a container that fetches the repo, export a reference to the git directory back to cue, mount that into another container which executes nix and produces a json file; export that json file into a string value in the cue tree; unmarshal it and iterate over its contents; for each iteration, run a buildkit job with the correct parameters; export each newly built image back as a reference in the cue tree; and so on
vast lichen
Okay, thanks, I get it conceptually. Will find some time to experiment with this. It could also be an interesting stress-test (TVL build graph has ~500 nodes, nixpkgs has >40k) for solving logic etc.
there'll be some minor hurdles to overcome (dealing with the Nix store, which would be nice to just bind-mount initially while testing on a single node; also the fact that everything TVL does is public and I'll need to find a way to prototype something non-publicly), but those should be fine 🙂
modest urchin
You can bind mount local directories at will 🙂 It’s supported
let us know how we can help! And yes it will be quite the stress test 😅
vast lichen
Thanks, will do!