SDKs roadmap

[balmy mauve]

@cerulean oyster @valid heath @vast bear

Marcos told me you talked about SDK roadmap & priorities today, and there are unresolved questions?

[balmy mauve]

I am free to chat live if you want

[cerulean oyster]

Tom started some discussions, I'm currently writing some more detailing what we could do and to make more visible the different benefits.
I think the main question is the priority of all that and where to focus.
Should we focus on performances only? On maintainability of SDKs? On ease of writing new SDKs?
This could have an impact on the steps. For instance if the goal is only performances of the three main SDKs, we can think about doing improvements on each of them independently without changes on the SDK interface. It will not give the full benefits but it will improve things. It means to re-do it for each SDK.
If we change the SDK interface more, this will drive the work on SDKs as it will provide a framework to change them, and might be one step further in term of performances. This will also help a lot writing new SDKs as the interface will be cleaner, you know what to expect from each step. But the effort is bigger.

So basically the question is where to stop, what benefit we want to get and for what reason (what user problem we want to solve). In in front the effort we agree to spend.

[balmy mauve]

personally I think we should follow the Theseus model:

• start working right away on the highest-impact change, even if it's a long-term effort. eg. unify the 2 caching systems

• do it in a way that you can ship early and often (not a multi-month mega-branch). eg. theseus PRs already get merged regularly

• Go on side quests along the way, to solve short-term problems for users. eg. add function caching without waiting for theseus

cc <@&946480760016207902> and in particular @rich mauve @void star @quiet willow 👆

In the case of SDKs that becomes IMO:

1. Start work on the new interfaces, which will be a multiplier on everything else

2. Bikeshed the interface design upfront, but implement it incrementally

3. Fix DX and performance issues along the way, as side quests

That's my thinking so far, based on the deep dive + observing Theseus.

What do you think we should prioritize @cerulean oyster ?

[valid heath]

For now I'm working on a PoC to benchmarks SDKs with au automatic tools, then we shall start with the split on the init function, that's fairly simple and can already increase the dagger init speed

[cerulean oyster]

Fix DX and performance issues along the way, as side quests
I wonder if we shouldn't do the opposite, start from DX and performances, have them as main goal and from there construct backward all we need. But maybe that's just two readings of the same thing.
I do agree we need to have something iterative (way more than the type defs/self calls)

[balmy mauve]

yeah better DX and performance are the goal. but... now what? 😁

[cerulean oyster]

On priorities, what could improve a lot the DX would be to focus on reducing the things we are generated.
For instance "when a body function change but not the signatures, the core + dependent types should never be re-generated"
This means a dagger develop should have no impact, and a dagger call should only recreate the final stage of the runtime container, building the module before to run it (no codegen at all)
All that based on cache of course.

Use case:

• dagger init --sdk=go
• change container-echo function, for instance to add something in the echo command
• dagger call container-echo should have almost no overhead up to the call of the function itself (in the case of the template the container | from alpine | ...)
  Today, on my machine, to load the module takes 1.5s. We should go down to ~0.4s and only rebuild, no other steps
  (this is just example with a very empty module and no deps)