Zenith Development Planning

[cinder swallow]

Goal: continue rapid iteration on Zenith and related SDK support, merging incremental commits into main as much as possible to avoid eternal rebase nightmares.

Problem: the base commits of Zenith are massive and also fundamental pre-reqs that can't be separated, namely:

1. The session frontend changes
2. The follow-up plumbing that deletes projects and adds the new environment concept.

Following is my current thinking for how to approach this; I will convert everything to linear issues with more detailed subtasks once others give a SGTM. Feel free to disagree about anything and everything though of course.

Active Quests:

1. Get the session frontend PR (without Zenith changes) into main ASAP
   1. There are a few features left to add, then a ton of cleanup and finally lots of testing
   2. e.g. the oci store fix @plain merlin is working on should be backported here
2. In meantime, continue to iterate on the Zenith PR branch, primarily:
   1. UX/DX polish of tool+check entrypoints, reconciliation w/ Zenith Design proposal
   2. Python SDK support, cc @crimson niche
   3. Updating integ tests with barebones coverage of new stuff
   4. (If bandwidth) Addition of basic artifacts API
3. Figure out how services v2 integrates with session frontend + what upstream changes are still needed
   • Seems the answer may be no relation, which simplifies things

Deferred Quests (to be restarted once the above are in main):

1. Any other SDK support for environments besides Go+Python

• cc @fleet lance @gleaming dirge while sad I think it will work out best in the end to hold off a bit longer

2. More complicated entrypoints like shell, deployment, etc.
3. Filling out more universe envs
   • I think the ones we have there are fine on a provisional basis since they help demos and validate our ideas+work
4. Codegen support for environments

cc @shell jay @pliant latch

(sorry for the wall of text 😅, I was overwhelmed with the amount of stuff, so this helped sort through, hope it's not too much)

[plain merlin]

SGTM, thanks for writing this up!

re: "Active Quests" point 3, I took a look the other day and don't think the services v2 changes benefit a whole lot from the session frontend changes. It still requires plumbing all the way down to the executor.Meta type for configuring networks in a way that doesn't put session-ephemeral info into LLB, and also requires changes to the HTTP and Git sources, so I didn't see any new shortcuts available to us. Let me know if you had something in mind though, maybe I overlooked something.

I dropped a ping in the Buildkit slack, but no bites yet

[cinder swallow]

Okay cool! I will go refresh myself on the services PR to do a double check, but lets assume there's no impact for now until proven otherwise

Also, I didn't include any context on the timing here, I can get the session frontend PR to a mergeable state next week, but given the size it will take a bit to document and review too. So the pessimistic estimate would be to merge the week of the 24th.

I'll do whatever I can to speed the process up, but that's probably the right estimate to go with for now.

[shell jay]

Thanks, @cinder swallow ! We also need to help tell the story. How can I help on docs?

but given the size it will take a bit to document and review too.

I'll have more time to help with the docs and comms next week

[gleaming dirge]

I have a general idea of what you want to achieve. The rust sdk is somewhat stable atm. i dont see the need for massive changes until zenith is in place. If you've got some more specific to share then that would be appreciated otherwise ill wait until there is a reference implementation to follow

[pliant latch]

I would start with the README and keep it all there until a) the general structure and story stabilize and b) it’s really too large for one document.

I think we’re at least 1 month away from outgrowing the readme

[cinder swallow]

Agree with Solomon, also think we could just start with the parts we have: tools (still named commands, but can be reconciled in the code soon) and checks. And also perhaps focus more on the robinhoods than tophats, which I think is likely easier to get going with

[cinder swallow]

Yep that sounds perfect, that's why I want to start with just Go and Python. Once we have two SDKs aligned we'll be in a good place to get the rest in

[gleaming dirge]

I will also hold off of major refactorings id planned for my summer break. As the session frontend potentially makes my plans redundant in the best kind of way xD

It SGTM atm. Reach out once you feel the tires are ready to be kicked at (don't know how well that translate to english)

[cinder swallow]

Will do! And it translated perfectly 🙂

[cinder swallow]

For the record, took a look again and 100% agree with what you said. Once we have a network session attachable we'll need to integrate that with the session frontend, but should fit in perfectly with the rest of the session handling there

[fossil sparrow]

With zenith UX of the gale going to have massive shift. So I would need hold back some refactorings and make some changes.

In the mean time, I could create some timeslot for supporting this development effort if it make sense for you as well. Please feel free to throw and development / testing etc. work to my side.

[plain merlin]

😂 I suddenly remembered pipeline labels and wondered how that worked with session-frontend (we need a cool name for it btw)....

go pipeline.LoadRootLabels("/", "da-engine")

[cinder swallow]

That sounds great Ali, thank you! At this moment things are still a bit too in-flux probably, also don't want you to waste time on something that ends up radically changing again, but that should improve over the course of the next week or two, so we will let you know once we've got something just solid enough for Gale to start dogfooding 🙂

[cinder swallow]

😁 Yeah as you can see I deferred that problem for later, I think the solution will be for clients to just send those to the server as frontend opts, so should be pretty straightforward to re-incorporate

Also yeah session frontend is not a good name. The fact that we are using frontends is more incidental than fundamental

I like "dagger server", but that kind of leaves out that it technically is just a component of the overall daemon, which includes all the other buildkitd components we import too

[plain merlin]

yea true. hmm

btw, I was able to implement a (*buildkit.Client).CacheKey(context.Context, *pb.Definition) (digest.Digest, error) just now, so we can compute the actual cache key for things now (even SourceOps). pretty useful! maybe we can also use that for check if something is cached ahead of time?

use case here was to fix the Container.import caching scheme; previously it computed a cache key based on FileID, which is obviously not stable, and technically the cache keys in Buildkit are more "stable" than the vertex digest itself. for example llb.ExtraHosts is omitted from an ExecOp's cache key.

here's the implementation (haven't pushed yet): https://gist.github.com/vito/cff6ac28aa27cccc8e599c0bc0c05bc2 - that's on our *buildkit.Client

[crimson niche]

SGTM. I'm going to review your PR more closely tomorrow and understand what needs to change.

[fossil sparrow]

👍 sounds good.

[cinder swallow]

Great! I still owe you and everyone updated dev docs on modifications to how runtimes get invoked, it's on the (long) TODO list, but in the meantime feel free to just reach out, I'm happy to go over it directly too

[crimson niche]

For sure. I like reading the code anyway because I get a much deeper understanding of how things work, so you can focus on the more pressing issues.

[fleet lance]

LGTM, let me know if I can help on anything!