#first thing that comes to mind is that

1 messages · Page 1 of 1 (latest)

wicked kestrel
#

(starting thread)

fossil pendant
#

nice

wicked kestrel
#

Thinking out loud: when an extension returns a result it just json serializes it (and write to /output/dagger.json)

fossil pendant
#

excerpt from generated code:

// An OCI-compatible container, also known as a docker container
type Container struct {
    q *querybuilder.Selection
}

// A unique identifier for this container
func (r *Container) ID(ctx context.Context) (ContainerID, error) {
    q := r.q.Select("id")

    var response ContainerID
    q = q.Bind(&response)
    return response, q.Execute(ctx)
}
wicked kestrel
#

So if we instrument the structs we generate to have custom MarshalJSON

fossil pendant
#

which is at odds with returning from code-first

wicked kestrel
#

that actually might be sort of clean-ish? It would all be taken care of by generated code

#

Seems logically consistent

#

idk it feels weird but maybe plausible

fossil pendant
#

Yeah, similar to the interface suggestion I was making (e.g. call .Execute() or something if it implements the generated inteface)

#

but very magic

#

I don't know how "generic" it is

#

like, for container, nobody really asked for an ID

#

we'd have to imply that you want the ID and just that

#

does it work for all generated types then?

wicked kestrel
# fossil pendant but very magic

Yes but it does actually make sense on a non-magical level too. The idea is that if you want to "serialize a query "that means you turn it into an ID, which lets you "resume" it later

wicked kestrel
#

Which might be okay?

#

idk I'm not totally sold but something along those lines (finding a way to cleanly define what it means to serialize an unfinished query) feels promising

fossil pendant
#

I think the end result is similar but it'd be more like the interface i mentioned (need a context and other crap to fire the query, which the json marshaller won't give)

#

but yeah, same result

gilded wraith
#

Sorry if I’m being naive but the problem is related to the fact that extensions may want to return callbacks rather than fully resolved scalars?

#

so that those callbacks can just eg. wrap to underlying core resolvers?

wicked kestrel
fossil pendant
#

just trying to see how we can combine "code-first schemas" and the "client sdk"

wicked kestrel
#

The interface approach makes sense too yeah, it's that or make any field named "ID" special

fossil pendant
#

it conjunction with codegen, basically all those types already DO implement the interface

#

so it goes like this: returning a custom type? we'll call ID() for you and return just that

wicked kestrel
#

Yep so valid return values from an extension handler are:

  • scalars
  • things that have .ID()
  • lists/structs/etc. that contain only those
gilded wraith
#

So when implementing an extension in Go, the Go SDK gives me a server-side Container that wraps the client-side Container ?

wicked kestrel
fossil pendant
gilded wraith
#

But from the point of view of my extension, me receiving a Container from a query and me returning a Container are different Go types in practice right?

fossil pendant
#

you can't receive non-scalars

#

only scalars or Input Objects (which you can't return)

#

so there's never an input that you can return

gilded wraith
#

Well that’s the thing. Maybe we want to create such a type anyway to facilitate middleware?

#

Presumably the query builder has such a type

#

don’t you generate a Container type in query builder @fossil pendant ?

#

so “receive from a query” is really “receive from the query builder” then

wicked kestrel
# fossil pendant what about non scalars for which we use a trivial resolver (e.g. what you did in...

Yeah so that would be a struct of scalars that are already filled in.

Maybe instead of using .ID() we could have a more particular special method like DaggerID()? Then we rely on codegen to add that (or users trying to do something very advanced outside of codegen/their own codegen).

So the slightly refined logic would be:

  1. If object has .DaggerID() method, call that and serialize it to /outputs/dagger.json
  2. Else, json.Marshal whatever you get and write that to outputs
    ?
fossil pendant
#

hmhmm. maybe. random things:

  • ID / DaggerID: could have DaggerID aliased to ID, but ID must be there as well (that's the client binding)
  • The problem is generalized, also happens from non-scalar-non-trivial-returns (e.g. if you were to return a NetlifySite from deploy and have a resolver for e.g. LogsURL)? Not that we need to promote them explicitly, but now because of this it can happen
fossil pendant
fossil pendant
fossil pendant
#

I have codegen generating this alongside ID, if it encounters one:

func (r *Container) Serialize(ctx context.Context) (any, error) {
    id, err := r.ID(ctx)
    if err != nil {
        return nil, err
    }
    return map[string]any{"id": id}, nil
}
#

altered alpine (manually, to simulate what the extension sdk would do):

func (r *alpine) build(ctx context.Context, pkgs []string) (any, error) {
    api := graph.New()

    // start with Alpine base
    image := api.Container("").From("alpine:3.15")

    // install each of the requested packages
    for _, pkg := range pkgs {
        image = image.Exec([]string{"apk", "add", "-U", "--no-cache", pkg}, graph.ExecOpts{})
    }
    return image.Serialize(ctx)
}
gilded wraith
#

(so not an input)

# 
func (r *alpine) build(ctx context.Context, pkgs []string) (dagger.Container, error) {
    api := graph.New()

    // start with Alpine base
    image := api.Container("").From("alpine:3.15")

    // install each of the requested packages
    for _, pkg := range pkgs {
        image = image.Exec([]string{"apk", "add", "-U", "--no-cache", pkg}, graph.ExecOpts{})
    }
    return image
}

Something like that 👆

fossil pendant
#

that's what i'm trying to do

gilded wraith
#

Ok thanks for confirming

fossil pendant
#

that's the problem, what build needs to return is the container ID

#

the container from the query doesn't contain any field, yet

wicked kestrel
# fossil pendant hmhmm. maybe. random things: - ID / DaggerID: could have DaggerID aliased to ID...

The problem is generalized, also happens from non-scalar-non-trivial-returns (e.g. if you were to return a NetlifySite from deploy and have a resolver for e.g. LogsURL)? Not that we need to promote them explicitly, but now because of this it can happen
Yeah that does make it less clear. This reminds me of the past conversations about maybe having some default behavior where if you don't select any scalars all the trivial ones get autoselected for you.

So with the NetlifySite case, you could have an extension return that type, no scalars selected, in which case the extension runtime sees that there is no .ID() or Serialize() or whatever, and then falls back to selecting all the trivial scalars?

Adds another layer of complexity, but on the plus side I suspect it would actually work perfectly with the way subresolvers work in terms of parent objects

wicked kestrel
# gilded wraith so “receive from a query” is really “receive from the query builder” then

Maybe I misunderstood what you were suggesting here, but my original interpretation was that basically instead of having resolvers like:
func Foo(id dagger.ContainerID) dagger.Container

we could just have func Foo(ctr dagger.Container) dagger.Container . Basically hide the whole ID thing from user extension code entirely (even though IDs would still be in low-level graphql API).

I do like that idea a lot, it would require a solution along the lines of what we're talking about here still I think, but it probably is a better DX if possible

gilded wraith
#

I remember @fossil pendant proposing to implement a query builder that would be lower level but leaves you free to implement conveniences like that on top, without being forced to

gilded wraith
wicked kestrel
#

I think there will also end up being similar things on the extension-side. I am supporting just providing a struct with fields+methods and deriving the schema from that, but it probably makes sense to give a lower-level interface too for explicitly registering handlers (haven't tried yet, just been thinking about it)

gilded wraith
#

I need to play with both of your branches tomorrow before I can weigh in here

#

I feel just informed enough to confuse you but not enough to actually help 😁

wicked kestrel
#

@fossil pendant where would this logic break down:

  1. If returned object is from codegen but not a scalar, then select all trivial scalars and write those as outputs
  2. Else, just json.Marshal whatever you get and write that as outputs

I think it covers:

  1. the case of anything with an ID
  2. the NetlifySite w/ non-trivial LogsURL resolver you mentioned above
  3. All the other simpler ones

Unless I'm missing something, which is becoming more likely the night goes on

fossil pendant
#

it wouldn't cover #1, no?

#

oh, by "select all trivial scalars" you mean that?

wicked kestrel
fossil pendant
#

yeah

#

maybe? a bit too tired to think it through 🙂 I'll get back to you tomorrow. First reaction is it's a tad funky to select all non trivial fields

#

like directory contents (listdir), we're lucky it now takes an optional path, but otherwise it'd fall under that too

#

which means that returning a directory forces evaluation of the entire thing + a costly listdir grpc call

#

also uhm, the way query builder works we'd be sending a query for each field with the full parents etc. Would need to change that somehow to be single query

wicked kestrel
fossil pendant
#

i'm thinking from codegen point of view, where every field has a function

#

the way I see it:

  • we do the serialize workaround we talked about with ID
  • we do what you were describing from the point of view of code first
  • we do what you were describing from the point of view of codegen (e.g. change codegen so we always get all trivial scalars)
    • but then every call takes ctx and returns an error
    • how do we "resume"?
#

I'll sleep on it 🙂

#

for 1), I just tried what we discussed and "simulated" the code first part (what we'd do when writing /outputs):

func (r *alpine) Build(ctx context.Context, pkgs []string) (*graph.Container, error) {
    api := graph.New()

    // start with Alpine base
    image := api.Container("").From("alpine:3.15")

    // install each of the requested packages
    for _, pkg := range pkgs {
        image = image.Exec([]string{"apk", "add", "-U", "--no-cache", pkg}, graph.ExecOpts{})
    }
    return image, nil
}

// FIXME: don't mind me
func (r *alpine) build(ctx context.Context, pkgs []string) (any, error) {
    var (
        ret any
        err error
    )
    ret, err = r.Build(ctx, pkgs)
    if err != nil {
        return nil, err
    }

    if s, ok := ret.(Serializer); ok {
        return s.Serialize(ctx)
    }

    return ret, nil
}
wicked kestrel
# fossil pendant for 1), I just tried what we discussed and "simulated" the code first part (what...

Yeah I think that's what we should do (the lowercase build implementation would be generalized and exist in the dagger.Serve code here-ish: https://github.com/dagger/dagger/blob/50d98787678c17c38d1eba17275995358b2156b1/sdk/go/dagger/server.go#L280-L288)

Then we can try out different approaches for the Serialize implementation (just find an ID, select all trivial scalars, etc.) to see what makes the most sense.

I'll try to re-explain the "select all trivial fields" again tomorrow (will also validate it to myself too then).

fossil pendant
#

sounds good 🙂

#

the cool part is the above actually works ... (well, I still get platform issues so the query fails, need to check why with Alex)

#

alpine went from 100 lines to 20-ish

gilded wraith
#

coming back for more confusion questions

#

since a chain of calls to the query builder maps to a chain of resolvers… is it possible for a router to do the reverse, eg. use a “queriable” as a resolver?

#

(not sure what the right term is for what the query builder returns)

fossil pendant
#

ok, got the codegen to work with LLB-style arguments. Not sure yet if it's the way to go (a bit magic), but it works (although the code is highly un-maintanable because of complex templates

# 
core := api.New()
dir := core.Directory()

contents, err := dir.
    WithNewFile("/hello.txt", api.WithDirectoryWithNewFileContents("world")).
    File("/hello.txt").
    Contents(ctx)

require.NoError(t, err)
require.Equal(t, "world", contents)

return nil
#

/cc @faint relic ^^

#

just started working now, need to clean up a bunch of crap