bundling

[atomic ice]

🧵

if I may @sour rivet @errant prawn @proud citrus

[sour rivet]

If it's just our deps, I don't see any reason to not do it

But I dunno about it being a "static compilation bundle" - I'd see it as instead being a different install mode for dagger, like a "fat" dagger with all the stuff pre bundled

[atomic ice]

yes but all your stuff for a specific pipeline

so you're effectively bundling a pipeline + all its dependencies, including dagger itself, into an artifact

with no dependency other than an oci runtime

[errant prawn]

Was just thinking how nice it would be to have some form of pre-baking module dependencies if not just to speed up our test suite. We download Go module dependencies a ton of times 🫠

[sour rivet]

I don't see why this solves a lot of the corporate module issues though

[atomic ice]

(heads up I am taking off and airplane wifi is apparently broken. I will be back online in approximately 90mn)

[sour rivet]

Agreed, I think we should do this for at least the python + typescript sdks

[atomic ice]

because all the go dependencies currently being blocked from downloading would be pre-downloaded.

[sour rivet]

Ideally we could even pregenerate and precompile the go code

Right but now you need to solve the bundling problem - you still need proxy logic, you just have it in a different place

[atomic ice]

that's fine, because you can build your pipeline in a less restrictive environment

for example @proud citrus can use his laptop (not blocked by proxies apparently)

the issue is production CI

[sour rivet]

But are companies okay with running opaque bundled artifacts from dev laptops in production CI?

[atomic ice]

let's give them the choice

🛫

[sour rivet]

Could we maybe start with the smaller problem of bundling just the dependencies just together? Not necessarily bundling dagger in there as well

Something like a vendor directory I guess (or an advanced version of that

[solid hamlet]

(On vacation but in car ride checking messages, may not be back online til next week)

I’m pretty vehemently opposed to doing anything that packages intermediate artifacts into opaque blobs. The model is and should be that everything is content defined as a DAG and performance optimizations are via generic caching of all operations in the DAG. Trying to create separate concepts like pre-compiled modules that black box the underlying dag is only going to create more complications for users and ourselves and break all the various things we can do with fine grained caching and supply chain tracking.

I also don’t understand how this solves any problems with knobs since it just means we need to add knobs to this separate “pre-compiled module” as opposed to knobs on the engine, which is how our existing support for GOPROXY works. It’s the same knobs either way

[atomic ice]

🛬

[atomic ice]

first: this is not urgent so don't feel the urge to reply during your vacation.

But I do think it's important so I will try to keep this discussion alive.

[atomic ice]

I’m pretty vehemently opposed to doing anything that packages intermediate artifacts into opaque blobs. The model is and should be that everything is content defined as a DAG and performance optimizations are via generic caching of all operations in the DAG. Trying to create separate concepts like pre-compiled modules that black box the underlying dag is only going to create more complications for users and ourselves and break all the various things we can do with fine grained caching and supply chain tracking.

I understand the concern. I don't think of it as adding new concepts, more like decoupling two existing concepts (building a module & loading a module) that are currently combined in a single opaque "build and load". It's not like we don't build a runtime image every time we load a module. We just never let the user see it or interact with it. That's not necessarily a bad thing; but the decision is not whether to create new concepts or not, rather it's how much of the existing concepts to show the user.

I don't see how decoupling the "build module" & "load module" phases necessarily hurts our DAG model, or our ability to take advantage of caching, traceability etc. After all we already embrace the fact that users will not always be willing or able to run their entire delivery pipeline, end to end, in a single call to a single dagger engine. The "one DAG" is already broken down into islands, interconnected by registries and other means of artifact transfer. We're happy to daggerize one island, and wait for the chance to daggerize another. This is dictated by the user's constraints - whatever makes their life easier. If the user's constraints happen to be "it would be easier for me to build my dagger module in one environment, then load and execute it in another, more constrained environment", that's just another island in our already fragmented DAG. We already build the module's runtime as an OCI image. What's the harm in letting the user export it from one engine, and load it into another, with a registry interconnect in between? Both engines will still have a cache; they will still send traces to various central location; I just don't see what will break.

I also don’t understand how this solves any problems with knobs since it just means we need to add knobs to this separate “pre-compiled module” as opposed to knobs on the engine, which is how our existing support for GOPROXY works. It’s the same knobs either way

The point of loading a pre-built module is to decouple the module's build and production environment. In the production environment, you don't have to build the module, so you don't need the knobs related to building modules. Sure, you may still need them in the build environments, but less users will need those knobs, and they will need them less urgently, because they will have an escape hatch: finding a less restrictive environment to build their module. This is easier because as a rule of thumb, dev environments are less locked down than production environments.

A few more considerations:

1. Maybe this doesn't have to be a core feature? Since on the one hand this is only needed by production teams that can afford a corporate proxy in the first place, and on the other hand we don't want to disrupt the core experience where build & run are seamlessly combined, and "cache is king"... Maybe this feature could be layered above the core, as an optional commercial tool? Like CI/CD for your CI/CD 😁

2. Maybe instead of building from the runtime image, we could generalize a solution based on cache snapshots? If the user could get a cache export from building the module, in a stable supported format, then pass that cache export to their production CI along with the module source , and trust with certainty that the result will be exactly the same as loading from a binary, that could be a happy compromise? Higher-level tools could still create the same experience of "bundling the pipeline into an artifact" but the artifact would simply be source+cache snapshot. It's just that today our caching system doesn't have that capability. The closest thing would be EXPERIMENTAL_DAGGER_CACHE_EXPORT (env var name from memory) but I just don't trust that it can deliver the level of reliability required to create the illusion of a pre-built artifact. Maybe that could be fixed though?

🛫 again

fyi @tropic grove I know you have thoughts on this

[primal sand]

Just FYI we have this exact use case. We have build envs with step down access to the internet. Not laptop but different enclaves.

[solid hamlet]

I'm getting the sense we are conflating separate problems here. From what I read previously, users just want to be able to set env vars during module loading, but now we're talking about users who need to run builds and loads in separate environments. I get that there's a some connection points potentially, but these problems can and should be viewed orthogonally I think. There is not a one-to-one mapping between users who want to set custom env vars and users who need to build+load in separate environments. There's plenty of users who just want an env var set and do not care about anything else.

We already build the module's runtime as an OCI image. What's the harm in letting the user export it from one engine, and load it into another, with a registry interconnect in between? Both engines will still have a cache; they will still send traces to various central location; I just don't see what will break.
The packaging in the engine as layers is an implementation detail and for non-Go SDKs it's literally just packaging source code. The fact that it's being done as layers is just an internal performance optimization (layers are faster to load and easier to cache loading of than arbitrary directories).

The harm in doing this is that it creates an enormous amount of work for users to solve the otherwise simple problem of "set env vars on module containers" if this is the only way to do so. It requires users setup separate builds of their modules, setup a registry, setup automation so they are updated whenever needed (including on any and all dependency modules in their whole DAG of deps, etc.).

I get that there's users who do need separate environments and we should support that, but we can do it in such a way that it's completely orthogonal.

Maybe instead of building from the runtime image, we could generalize a solution based on cache snapshots? If the user could get a cache export from building the module, in a stable supported format, then pass that cache export to their production CI along with the module source , and trust with certainty that the result will be exactly the same as loading from a binary, that could be a happy compromise? Higher-level tools could still create the same experience of "bundling the pipeline into an artifact" but the artifact would simply be source+cache snapshot. It's just that today our caching system doesn't have that capability. The closest thing would be EXPERIMENTAL_DAGGER_CACHE_EXPORT (env var name from memory) but I just don't trust that it can deliver the level of reliability required to create the illusion of a pre-built artifact. Maybe that could be fixed though?
Yes when we have discussed this at various points in the past the idea is to support something like pinned cache that isn't just "best effort until pruned".

I greatly prefer that sort of model in combination with the "magic background" caching that magicache + storage drivers provide, as compared with the EXPERIMENTAL_DAGGER_CACHE_EXPORT model. I think it keeps the amount of complexity for users minimal

[atomic ice]

yeah setting env variables in the module is 100% orthogonal

[solid hamlet]

Sorry if I'm confused, back home now but was previously reading everything while stuck in traffic in a cramped/hot car so not sure if I was following everything correctly 😄

[atomic ice]

No problem. There is a connection, which is that sometimes the underlying problem (can't run the dagger pipeline in production CI because of restrictive networking) can be solved by configuring the SDK with an env variable (eg. to disable use of the go proxy).

But only sometimes. Other times, we don't have workarounds. For example, one user disabled the go proxy only to find that go.opentelemetry.io (go dep of our sdk) was also blocked. So there's a game of wack-a-mole that IMO requires a more general solution.

[solid hamlet]

But only sometimes. Other times, we don't have workarounds. For example, one user disabled the go proxy only to find that go.opentelemetry.io (go dep of our sdk) was also blocked. So there's a game of wack-a-mole that IMO requires a more general solution.
The point I'm trying to make is that to solve that problem, we need to expose some part of the module building process (whether it's just a config setting or something deeper), but that should always be possible to configure without needing separate registries or other intermediate state. I can't see why that would be a hard requirement (even if separating building+loading across environments is a worthy separate feature).

Anything you can do that involves building a module into an OCI image, pushing it somewhere and then using it should also be possible to do within the confines of a single engine.

The nice part is that if you take that ability to build a custom module runtime and combine it with "cache pinning" like I mentioned above, then you have solved the problem, but in such a way that it's just the combination of two separate features that are useful both by themselves and together

[atomic ice]

I like this as a principle, but in practice how do you apply this principle to "I can't connect to go.opentelemetry.io from my production CI"?

[solid hamlet]

I don’t know the details of the issue, so I can’t say yet the precise way we will fix that one. But in terms of general principles here; we need to give users some way of solving this (maybe it’s just a config, maybe we give them even more control over the module runtime in our core API, etc.). They will use whatever the mechanism is to build a module runtime in a custom way and specify that as the runtime container.

Right now, the type we expose for the runtime container is literally just the core.Container type. Sure, you can publish that if you want, but it’s also just another Container that is going to be in the engine cache. If you only want to use it in the engine, you don’t need to publish it; it’s just a Container so it can just be plugged back into the core API.

And since it’s in the cache, it’s also integrated with all the current and future remote caching features.

Quite a bit of this is also possible already today by writing a custom SDK that wraps an existing one, but that DX is bad since it wasn’t really optimized for wide consumption yet

I’m back on mobile atm but when Im back at the computer I can try to provide more concrete examples