#Short term Go DX improvements

1 messages · Page 1 of 1 (latest)

wraith igloo
#

Creating thread...

#

I remember @lapis axle a few weeks back suggesting that we hold off on codegen for now, and add it as an add-on later when it's good enough to not make things worse. Seems like this is still relevant.

#

cc @merry trail who experienced the issue

#

and @molten hemlock as well

lapis axle
#
  • We have a long-term solution with the query builder approach, but still lots of work to get it to work properly (quoting @lapis axleuzzardi )

More like an experiment. I don't know if this is the best DX, just thought it's worth exploring

--> Is there a stopgap we could implement to improve the situation in the short term?

Query first or code first stopgap?

I was thinking of the "on the fly structs" approach that I think Hasura uses, and I believe we used in the Blocklayer client @lapis axleuzzardi ? It's not perfect but at least it's consistent, and it's kind of the least painful version of raw graphql queries in Go.

I don't recall this, need to brush up

wraith igloo
#

It would be code first

lapis axle
#

Random thoughts on stopgaps:

  • Code first: I think by supporting @collapse or whatever (or auto-collapsing 1 field structs), it'd be 10x better already
wraith igloo
#

Client constructs their query with a custom Go type. Same type is used for both query and response, which solves the "mismatch" problem we experience now

#

In the current method we get half an abstraction, which it turns out is worse than no abstraction

lapis axle
wraith igloo
#

Code first, but requires knowledge of the schema.

lapis axle
#

so a "schema first" in between, ish?

wraith igloo
#

Well, this is for client generation, so I don't know what "schema first" would mean in this context exactly

#

You don't need to write any raw graphql 🙂 But you do need to write Go types that match the structure of your query

lapis axle
#

but those Go types aren't auto-gen'd right? You have to "write the query" (in Go rather than GQL)

wraith igloo
#

I feel weird explaining this to you since I remember distinctly you explaining it to me last year 😄

white dirge
wraith igloo
#

@white dirge personally if the Go abstraction for arguments is too awkward, I would just send a raw gql query (but keep the same go types to receive and use the response)

lapis axle
#

So for instance in the query builder experiment:

  • There's a high-level "code first" code-gen thingie (alpine.Build()), similar to what we're doing today with codegen

  • There's a low-level "schema in Go" thingie, that's used by the high-level code-gen

#

(don't have a point here, just braindumping thoughts)

#

The low-level "schema in Go" is not really a schema, it's a way to programmatically build a query and bind return variables

# 
    var contents string
     root := Query().
         Select("core").
         Select("image").Arg("ref", "alpine").
         Select("file").Arg("path", "/etc/alpine-release").Bind(&contents)
#

(DX optimized for code-gen, not humans; can be improved)

wraith igloo
#

Does Arg take interface{} or string values?

lapis axle
#

interface{}

#

you can bind a struct for instance

wraith igloo
#

I see, yeah you get max flexibility then we could build whatever abstraction we wanted on top?

lapis axle
#

Yeah exactly. This is for instance the equivalent core.Image(), code-gen'ed by "hand" for now:

func (c *Core) Image(ref string) *Filesystem {
     c.q = c.q.Select("image").Arg("ref", ref)
     return &Filesystem{
         q: c.q,
     }
 }
wraith igloo
# 
    q := g.Netlify(token).Site(siteName)).Deployment(b.contents)
    res, err := q.Execute(ctx, dagger.IncludeField("URL", "Logs"))
#

At this layer I (try to) keep things simple by:

  • Forbidding individual selection of scalar fields
  • Only returning the innermost struct (flattening)
  • Assume scalar resolvers are cheap and don't fail
#

I'm assuming this will be the most robust design eventually, but in the meantime... I like Hasura's DX better than ours

#

So my suggestion is to swap that in as a replacement for our "pre-written query" generator (the one using operations file) to get short-term relief, while we perfect the awesomeness above

white dirge
#

I am in agreement that we should find a short term solution, but it's worth looking to see whether there is any way of just quickly updating the current client codegen to collapse fields

#

That would be strictly better than today, whereas Hasura is better in some ways but worse in others (it is almost as bad as writing a raw query if you need to provide args, which is probably many if not most of the queries users write)

wraith igloo
#

(on a call but will resume right after)

white dirge
#

Honestly, it's might be worth seeing whether we could quickly implement flatten for fields too. We can just use a fork of genqlient for now. Then in the most likely future we will just move away from genqlient. But in the unlikely event we want to stick with it we can merge the support upstream since they clearly want it anyways.

If adding support for that is too much effort (more than a few days) then I'm not sure it's worth it.

wraith igloo
#

(I'll just add that not having to write individual operations file in each extension would be an additional win - it's a major downside of the current method that we're still dragging around)

white dirge
# wraith igloo (I'll just add that not having to write individual operations file in each exten...

That's true too. The thing about Hasura is that looking through our schemas, the majority of our resolvers take args. So if we suggest users do that instead of using the current codegen clients then they will all be forced to use the odd struct tag format or just fallback to using the raw graphql client.

Also, technically there's nothing stopping go users from using either Hasura or the raw graphql client today. So the only "change" would be to update our docs w/ examples of using those.

wraith igloo
#

Yeah I didn't mean to emphasize Hasura specifically, it's more of a proxy for that particular pattern of querying GQL in Go. Maybe it's just what the raw client does actually?

#

Given our limited engineering budget for core DX until the next wave of contributors ramps up... We need to pick our priorities carefully.

  • My conclusion from last week is that extension development should be our top priority because it's a bottleneck for everything else. So we should focus as much scarse core engineering capacity on making extension DX (xdx?) 10x better.
  • That requires not taking on too much engineering challenges on the client side for now. Keeping things simple even if it means less bells and whistles, is the way to go IMO
#

One example of keeping it simple is that I'm dropping my idea of relying on cross-language collaboration by shared pre-written queries ("build your API without having to write an extension!")

#

I think we should also drop the current Go generator ("write an operations file in your extension and we'll generate a great client for you") instead of spending more effort to tweak it

#

If that means clients are stuck with the vanilla graphql client in the short term - assuming in Go that means creating custom response types, and possibly writing gql by hand - I think that's an acceptable tradeoff if that frees you @white dirge and @lapis axle to focus on XDX 🙂

sharp kindle
sharp kindle
lapis axle
sharp kindle
#

graphql is full of escape hatches.. docstring directives 😶‍🌫️

wraith igloo
#

Yes collapsing is a perfect word for it

white dirge
#

Yeah just referring to the fact that when an operation returns a single scalar, the return type of the codegen client should just be that scalar type, not the whole maze of types as seen in the response of the graphql query. I think they seem to be calling this "flatten" https://github.com/Khan/genqlient/issues/30

If it was really trivial to get that with a small change to genqlient then it would perhaps make sense to just do it as a temporary solution. But looking at their issue for it I don't think it's trivial at all so probably not worth it.

I think we should also drop the current Go generator ("write an operations file in your extension and we'll generate a great client for you") instead of spending more effort to tweak it
I'm thinking about it. The raw gql interface in go is a huge pain. However, the upcoming changes to the core API to that use more chaining will make the current issue way way way worse. So yeah we could just revert back to the raw client for now until we have the query builder approach ready.

My hesitation stems from the fact that most extensions call other extensions (or the core api), which means that the client binding DX is a subset of XDX. It's also going to be work in and of itself to update all our existing go examples to not use the codegen client and to document the raw client.

#

Still thinking about it though, there's no good answer until we have the better query builder type solution

#

I'll get back after a lunch break

wraith igloo
#

@white dirge they didn’t fully implement it in genqlient though. What they did implement seemed really different when I read the docs, but maybe I misunderstood

white dirge
# wraith igloo <@949034677610643507> they didn’t fully implement it in genqlient though. What t...

I'm just looking at that issue and the terminology they use there, not sure if their other docs are consistent with it. They seem to have implemented what we want but only for a very special case (fragments). I was hoping that there was a chance it would be easy to generalize what they did to fix our current problem but after a quick skim that seems doubtful. So that's probably a route we can rule out

wraith igloo
#

you’re right I had missed that part. Reading the issue clarified it

white dirge
#

Like I said before, I think the only good answer is to have a query builder type solution. But in the meantime, writing out the options explicitly to think through them:

A) Only support raw gql. Update examples and docs to use raw gql client (can also mention hasura and graphb as options).

Pros:

  1. rm operations.graphql
  2. Works with the new API proposal. The current genqlient output seems like it will probably be so hard to use w/ the new withX and other methods that it would be almost unusable.
  3. Easier transition to future query builder approach (turn your strings into method calls)

Cons:

  1. Raw gql queries in go are really annoying to get right. I often make typos and get extremely unobvious errors

B) Leave it as is except update the docs to mention the gotcha about following the right path in your output type.

Pros:

  1. Least amount of short-term work

Cons:

  1. Gotcha still exists, even if it's documented.
  2. Will probably not be usable w/ new API
#

After digesting some more, I think I am leaning towards A too. It causes me pain to write those raw graphql queries, but it's more like "tedious" pain than it is "surprise, gotcha" pain that users currently run into. It also is more of a step in the direction we want to go than it is to just maintain the current state.

sharp kindle
#

Cons:

  1. Raw gql queries in go are really annoying to get right. I often make typos and get extremely unobvious errors

I guess the playground can help there? specially after this 😉 https://github.com/dagger/cloak/pull/198

white dirge
#

It would be nice if there was a way to enforce strict unmarshalling, but not sure if that exists

wraith igloo
#

Maybe the playground could provide some convenience / relief along the way, like a plugin that generates the Go boilerplate for you from the query? 😇

white dirge
#

But that isn't to say the pull request isn't awesome, btw!

white dirge
wraith igloo
#

Right but that plugin would be an awesome companion to the query builder as well 🙂

#

But yeah you're right.

#

Core features before bonus features

white dirge
#

Oh totally, sorry I'm not trying to dismiss these other ideas at all, I like all of them. I am getting hyperfocused on the extreme immediate term

#

Just the instant pain relief

wraith igloo
#

It gives us options if, say, we're down the road of raw gql clients, starting to experience some recurring pain with these typos, and query builder is still some ways away. We have the option to bridge the usability gap with cloud

#

Here we go! merging 163. Just another stepping stone 🙂

white dirge
#

Would be insanely cool for our docs too

sharp kindle
# wraith igloo Maybe the playground could provide some convenience / relief along the way, like...

there's also an intermediate term which is to provide a go playground like DX where you can iterate on your queries with the native clients (raw, hasura, etc) and get immediate feedback to see if you're unmarshalling the response correctly. I have the feeling that the current feedback loop to test an extension is quite painful since it requires starting cloak, waiting for the extension to be re-loaded, open the playground and check if your query succeeded.

#

coming up with a go playground (and eventually other languages) like experience in the current playground should be something we could get pretty quick

#

we're currently prototyping this with @exotic jolt with the client-side javascript extension in graphiql

white dirge
white dirge
sharp kindle
sharp kindle
lapis axle
white dirge
#

The fact that the current operations+genqlient approach will likely be so bad that it's unusable with the new core API is probably the biggest factor for me

#

Happy to sync up more if needed, will hold off on doing anything until you get back and have more time to think. I have plenty more to do in the meantime 🙂

white dirge