Last task of the day write an updated

[dense remnant]

In terms of Python+Node SDKs having a dependency on the CLI, you are referring to them needing the local engine right? Or is there some other dependency on the CLI I'm not thinking of.

If so, just FYI I just finished prototyping a solution to how they can obtain the engine binary, so we may not need to package it directly with the SDKs (they can just get it from our engine image): https://github.com/dagger/dagger/pull/3647

Sorry if you're aware of all this already, just double checking we're all on the same page

[dense knoll]

Thanks - I'm not aware, and good to know. We urgently need to go over every aspect of every CLI that needs to be released (including your prototoype), to make sure we have a consistent design.

Oh sorry - the -v /:/ thing. Yes I'm aware of that. That is one option. (I'm very skeptical that we will ship this implemention fwiw)

[dense remnant]

Oh no I didn't do that, it doesn't work on macos

[dense knoll]

Yeah any context relevant to the SDK's dependency (or lack of dependency) on any binary release, please send my way so I can make an informed summary 🙂

[dense remnant]

In that PR I have a dagger-engine image which includes both buildkitd+cloak. When an SDK starts, it grabs the cloak engine binary out of that image (equivalent of docker cp). Then it can use it locally (caches it in local filesystem for reuse, but with versioning and all that).

Pros:

• Makes packaging way easier (there's just one engine image, not an image and then also figuring out how to package a binary for python, nodejs, etc.)
• Also makes it easier to transition to the future where the engine binary never has to leave the image at all (i.e. fully remote engine)

Cons:

• SDKs have to each have code for obtaining the cloak binary from that image and safely caching it, which is slightly annoying but the implementation is pretty straightforward so far. Works in Go, think it should be easy to translate to Python

More details in the description of the PR

[dense knoll]

I see, clever! I remember a comment by @grim canopy that this approach might go against developyexpectations that we integrate in their native package manager instead of sneaking in a binary. Definitely should be listed as a downside IMO

[dense remnant]

Totally agree that some users will have that concern. My immediate response is that it's not all that different from running binaries inside a privileged container that can import/export files to your client's host. And we are doing that anyways.

However, that's obviously not a very "user facing" statement, if we go this route need to clean it up

[pine hedge]

Yeah, just chatted with @grim canopy about that concern -- I explained the alternatives and he now likes the idea 😄

[dense knoll]

lol

what’s the alternative?

is there an issue for this thread? otherwise I’ll just create that since it covers the short term (python & nodejs sdk)

either way we still need to define the name & syntax of that binary , too

[dense remnant]

I was gonna use my PR as the discussion post for more (just wanted to make sure it actually worked before asking everyone for feedback), but a dedicated issue sounds better

[dense knoll]

good timing then 🙂

[dense remnant]

JIT code generation 😎

[pine hedge]

1. embedding the binary in the SDK code

2. user is responsible to install a binary on the side, distributed in a different way

[dense knoll]

what about 3) use package manager’s ability to install the binary? I think pip & npm both support that

[dense remnant]

We can but at least for npm it isn't totally free:

1. We have to do stuff with custom install and post-install scripts (pretty useful article on this: https://blog.xendit.engineer/how-we-repurposed-npm-to-publish-and-distribute-our-go-binaries-for-internal-cli-23981b80911b)
2. We have to package binaries for each os+arch combo. Looking now, our cloak bins are 18MB for Linux, 21MB for Darwin on arm64. So lets say it's ~20 MB x {x86, aarch64} x {linux,darwin,windows} = 120MB to pull the package down every time.
   • The image technique lets us optimize this more (only have to pull for your arch for example, maybe even only your OS if client+server match)

I don't know what the situation is w/ pypi

[dense knoll]

ok I understand, thanks

Seems like your approach has the least one way doors

If devs complain we can always add native packaging later. Harder to remove it once it’s out there downstream (ask me how I know…)

@dense remnant do you want to do the issue? Otherwise I’ll do it tomorrow to summarize the above + current consensus

• next steps of defining name & syntax of the binary in question

[pine hedge]

That’s 1

[dense remnant]

Yeah sounds good, I'll do that. Things get blurry again once we have to think about an actual CLI (i.e. for graphql API like we talked about the other day). I'll at least throw out some ideas on that there in terms of the naming + distribution

[pine hedge]

(That’s the only way to embed in the SDK code in Python / Node)

[pine hedge]

Python is similar — there’s a way but not straightforward

[dense knoll]

Assuming we agree on proposed terminology for engine architecture, then this binary would be the “Dagger API router”, itself a component of the engine. So my best guess would be dagger-engine-router ? Descriptive, and long enough to imply that it’s plumbing

[dense remnant]

I was going to suggest dagger-helper since even though today it will have the router it's possible as we migrate functionality from it -> remote engine that it no longer is a router and instead just has utils for filesync and similar stuff.

dagger-engine-helper makes sense too