#XDX
strange bear
Thread here
Context: I'm juggling many inter-dependent design threads and looking for a way to simplify things end to end.
In the beautiful world of README-driven development, I can say:
Each pipeline has an API. Pipelines can be shared and reused across projects, teams or the entire community.
The Dagger API allows composing a large pipeline from smaller ones. The component pipelines may be defined inline, or imported from an extension
An extension is a collection of pipelines which can be imported into any Dagger project, and used to compose larger pipelines in the usual manner.
earnest notch
Depends on precisely what you mean, but I did have some thoughts on that area yeah.
Scripts and code-first extensions look extremely similar now: they are both just main funcs now. In fact, if you made a code-first extension that takes no args and returns nothing, it would really really look like a script.
The main difference is that scripts run on the host and extensions run containerized. But I think we can also bridge the gap there. The fact that extensions are just main funcs means that it actually should be possible to execute them directly on the host too.
We'd need to decide what exactly should happen when you execute an extension main func directly on the host (there's a few possibilities). But it should be possible to basically unify these two concepts I think.
I don't I'm having trouble turning the idea in my head into words at the moment, it may be more clear to just give it a shot in a prototype. But let me know if that makes any sense
strange bear
But what about the stuff inside the main funcs.
- Your script calls functions that encapsulate graphql queries
- Your extension registers functions that resolve graphql queries
Could we make it so there is only one kind of function to write, and it can be interchangeably used for one or the other?
Kind of blurry but it feels very similar to the buildkit gateway pattern
maybe your callback runs in your client... or in a frontend container... but in the end it's the same code wrapped differently
earnest notch
Yeah I'm imagining something sort of along these lines. One simple possibility is that right now in my PR dagger.Serve expects a struct with fields+methods, but it could also accept a plain func(ctx dagger.Context) error, which is essentially the same as what scripts do today with engine.Start.
strange bear
could we make a Go SDK that requires you to write such a function to define your pipelines, making each of your pipeline usable either 1) inline in your script, or 2) packaged in an extension
in the current terminology, it would be the "everything is an extension" pattern you described a while back. Except we wouldn't use that exact word to describe it
earnest notch
I'm just imagining something along these lines:
embedded:
// existing code
dagger.Pipeline(func(ctx dagger.Context) error {
// do dagger stuff
})
// more existing code
script:
func main() {
dagger.Pipeline(func(ctx dagger.Context) error {
// do dagger stuff
})
}
extension:
func main() {
dagger.Pipeline(Alpine{})
}
type Alpine struct{}
func (Alpine) Build(ctx dagger.Context, pkgs []string) (Container, error) {
...
}
Don't take the term Changed Session->Pipeline, probably more in line with current terminologySession as permanent, just a placeholder.
maybe your callback runs in your client... or in a frontend container... but in the end it's the same code wrapped differently
Yeah I am defaulting to the code just running wherever you invoke it, but I understand what you're saying. I need to think about it some more in terms of how it would actually be implemented
primal wyvern
I had this blur as well in my head at some point.
For me, it's more about the interface. The interface is the same. But one will call the remote one via graphql, and the other will resolve from a graphql query. But it's the same parameters both side.
You could cut the middleman (graphql) if you would execute directly on host.
So you call the backend lib directly, instead of calling the client.
Does it make sense?
Or am I off topic?
earnest notch
I had this blur as well in my head at some point.
For me, it's more about the in...
Yes it does I think. Like right now with the new code-first alpine implementation, it's just a main func, so it would be neat if something happened when I actually invoked it directly on the host, i.e. go run examples/alpine/main.go
One interesting possibility is that the API that is parsed from Alpine{} is turned into command line flags. i.e. I could do go run examples/alpine/main.go --pkgs curl,jq and the output would be the filesystem built.
I guess the question is whether:
A) The code implementing the alpine API is executed directly on the host or
B) The code implementing the alpine API is executed inside buildkit, the same way it would be when invoked as a dependency
For alpine specifically, it doesn't really matter since it has no dependencies on its external environment. But that's not always true for other extension/pipeline code
primal wyvern
I guess it's just a matter of layering.
For Go, I would just have the pure Go API, and possibly generate the graphql layer (server + client) automatically.
server + client would just be there for encoding/decoding structs+fields.
Then, for the containerized or not, it's again another layer. Do we run the graphql only on the containerized system, or we could also run it outside of it?
I guess it only make sense to use graphql when containerized (in the case of Go script calling Go extension).
But of course, for Go script calling any other language extension, graphql is still the glue, no matter what, and that extension itself could also run non containerized.
but the more I explain my understanding, the more it make sense to just have any graphql API in a container no matter what (without thinking about the API call that can get more complex)
earnest notch
Possible model:
- If you want to invoke your dagger code directly wherever you are, just do that. Use a
dagger.Pipelinecallback, usego run,node index.js, etc. - If you want to invoke your dagger code containerized+cached, use a client. Clients include both our generated code clients, a raw graphql client, or the
daggercli (i.e.dagger run examples/alpine/main.go)
Idk none of this 100% adds up to me yet, but there is absolutely something here.
Once we've gotten the basic code-first implementation merged maybe we should try updating demov2 with this sort of approach where scripts+extensions are all unified as pipelines. Or we could come up with a demov3 around this idea. It's just a bit too hazy for me until I start actually trying it out in practice.
strange bear
I think maybe we're talking about different things
I was looking for a way to simplify things and now we're talking about 2 different ways to invoke the same code
But I might be just missing the connection, my head hurts
primal wyvern
I guess I might be offtopic
strange bear
What I'm talking about is making it so easy to write custom graphql resolvers, that even the inline queries for your one-off script, you end up wrapping in a resolver then invoking that resolver from your script.
The benefit being there is only one way to develop a pipeline, whether you're writing it inline for your script, or sharing it in an extension
earnest notch
What I'm talking about is making it so easy to write custom graphql resolvers, t...
That's what I want too, I am doing a poor job of explaining it
strange bear
Ok it's probably me then š
earnest notch
The containerized/uncontainerized issue is something we need to contend with, but the goal would not be to explain any of that to users
primal wyvern
I mean, it seems similar, finally to what I was explaining
strange bear
"like buildkit gateway pattern" is my anchor here, I know we have a shared understanding of that at least š
primal wyvern
except, for you, you want to make it always use graphql no matter what, even though it would look like you use the direct code?
strange bear
Yes, I don't think we can avoid that (nor should we)
primal wyvern
got it
strange bear
graphql resolvers don't map one to one with function calls
so I think there would be lots of weird little issues if we tried
primal wyvern
for sure
strange bear
Also we may have telemetry, caching, and other features that depend on everything going through a graphql pipe to a containerized router
primal wyvern
But about the OP, isn't it about reducing as much as possible the GraphQL boilerplate so that you can just focus on your pipeline, and then people can use it from a script or as an extension?
strange bear
I'm talking about the situation where "people" is you
First you write pipelines as client queries in your script. Later, you refactor those pipelines to have an API, so that they can be shared and reused in an extension.
What if you could do both at once.
We already have a plan to reduce boilerplate in both cases:
- To reduce boilerplate in client queries: query builder
- To reduce boilerplate in server resolvers: code-first framework
What I'm asking here is: could those 2 improvements be combined in a single unified DX
So it's an additional step beyond removing boilerplate: taking advantage of removed boilerplate, to reduce code duplication
earnest notch
I can imagine ways in which we can make it very easy to take what used to be client code, copy it into server code wrapped with typed parameters+returns and then call that as a client. Same as refactoring a chunk of code out from a big function to its own separate function.
I feel like you are describing something more than that though. That the code you write as a client could literally be directly turned into an api somehow? I have some incredibly vague sense of something here, but I'm really not sure. If you have any vague sketches of what you think this would look like, that would help too. Otherwise I will keep mulling on it
strange bear
I just mean force client developers to always write server code
I do not mean allowing regular client code to magically become server code
More like: force client devs to do the work, and make it so easy that it's ok
- Before: your magefile can call the query builder directly, then ask the SDK to invoke the query
- After: your magefile registers a resolver which can call the query builder and invoke the queries; the magefile then asks the SDK to run that resolver
earnest notch
Ah okay, the point I'm confused by is "the magefile then asks the SDK to run that resolver". How does it ask the SDK to run the resolver? Before there were two concepts: query builder and resolvers. Now there's those two concepts still but also a third one that is something that lets you invoke a resolver but not via a query?
strange bear
I don't know. Might just be a bad idea or nonsensical
It might be what @primal wyvern said: SDK lets you call resolvers directly on the client (not via an actual graphql)
primal wyvern
Here's what I understood and I tried to explain: https://mermaid.ink/img/pako:eNptjk1vwyAMhv9K5HNSJaOpAoeeetxpu01Ik5d4K2oCDIjUNs1_n5tq0758QNg8j18maF1HoOAtoN9n9w_aZlyxDcanrCi2F_TmucW-v2QvYzSWYvyDLO47E3RMZKNx9oZ446ln5Qr9fvtP_-RvxJew6N_DIYeBwoCm449P14mGtKeBNCi-dhgOGrSdmcMxuceTbUGlMFIOo-8w0c4ghw6gXrGPPPVon5z70YOa4AiqkPWq5GqEvBNCyGadwwmUaOpVtakryUcpZLmZczgvG6r5A70QbT4)](https://mermaid.live/edit#pako:eNptjk1vwyAMhv9K5HNSJaOpAoeeetxpu01Ik5d4K2oCDIjUNs1_n5tq0758QNg8j18maF1HoOAtoN9n9w_aZlyxDcanrCi2F_TmucW-v2QvYzSWYvyDLO47E3RMZKNx9oZ446ln5Qr9fvtP_-RvxJew6N_DIYeBwoCm449P14mGtKeBNCi-dhgOGrSdmcMxuceTbUGlMFIOo-8w0c4ghw6gXrGPPPVon5z70YOa4AiqkPWq5GqEvBNCyGadwwmUaOpVtakryUcpZLmZczgvG6r5A70QbT4
strange bear
"not found"
primal wyvern
business is whatever a user wants to create/connect
and either you call it directly from your code (like calling the netlify API directly from Go, for example)
Or you wrap it in an extension behind a graphql resolver that your script calls or you wrap it even further in a pipeline that your script calls.
am I right that both the pipeline or the extension would be behind a graphql resolver?
atomic rock
@strange bear @earnest notch Merging the two threads together after giving it some thoughts
I donāt fully grasp the āxdx is merged with the client codeā yet, but, in that spirit, I think there might be ways to make them cohesive and ālook the sameā
Havenāt gotten the chance yet to try and play with both at the same time (except the tiny experiment from last night), but for instance the conversation about positional arguments on the other thread:
Maybe thereās a way so that positionals in XDX and codegen are the same, meaning a function signature in XDX matches the one in codegen
The trade off is itās going to be a lesser experience in both (since not optimized for the use case), but it would be same experience
Again thatās just an example, but the idea is to get them much closer together.
Thereās also the thing from last night where from XDX you can return a client codegenād type directly ā basically a way to get them to play nice together
Thereās probably a ton of other things we could do so they become āstreamlinedā and feel like just one SDK, rather than 2 parts
earnest notch
Thereās probably a ton of other things we could do so they become āstreamlinedā ...
The idea of being able to, in your code, accept those objects too (so accept a Container instead of a ContainerID) is pretty interesting too. Basically hide the rules about graphql input types from users writing code
atomic rock
Right now weāre in Phase One which is just getting each one working on their own (both are incredibly complex, @earnest notch pulled some heroic tricks to make xdx work). Phase Two is taking a step back and rethinking them as one (basically making them aware of each other, boost xdx with codegen and vice-versa)
atomic rock
The idea of being able to, in your code, accept those objects too (so accept a C...
Yep exactly
earnest notch
Yah I can't really think very clearly about any of this until it's concrete in front of my eyes. Once we and everyone else start to play with it it'll become obvious what we want to write intuitively and then we'll figure out how to make that happen.
atomic rock
Is this the right thread for SDKs in general?
I reworked the "client connection" part of the Go SDK: https://github.com/dagger/dagger/pull/3265
It's a conversation starter, not ready to merge as is. Among other things, it supports connecting via stdio
/cc @earnest notch @strange bear
strange bear
Canāt hurt to restart a āGo SDKā thread IMO
earnest notch
Code is reviewable for code-first xdx now too btw: https://github.com/dagger/dagger/pull/3191
Been working on updating docs intermittently throughout day, will update with that later tonight. Also going to do a pass where I comment the code a bit more since I realize the reflect/ast stuff is a bit nuts at times. But won't be making any significant changes to the code, so feel safe to take a look now
Also @atomic rock you will be happy to see that I took the opportunity to clean up some of the stuff around RemoteSchema and CompiledRemoteSchema (those types don't exist anymore) since I had to make changes to all that stuff anyways. There's still complexity but I managed to find a less awful way of modeling it now I think.
atomic rock
Code is reviewable for code-first xdx now too btw: https://github.com/dagger/dag...
I'm super excited about that š
I'm in a good place with codegen, will be able to show a demo tomorrow, PR ready tomorrow as well
*initial PR, at least
earnest notch
*initial PR, at least
Yeah there's a million follow ups for code-first too (short and long term), but it has covered enough cases that people shouldn't be too surprised using it (I think)
primal wyvern
btw, I still don't know what XDX means. Sombody cares to explain? š