Thought experiment - client API

[manic tapir]

Suppose that the core API included:

• Access to project dir (equivalent of core.#Source, kind of)
• Ability to read env var (client.env)
• Ability to execute commands on the client (client.command)

Then a copy-pasteable snippet could become a complete project template

I don’t know how this would work exactly. I was hoping that writing it down would make it more clear, but so far it’s not working 😅

[cinder oriole]

On a technical level this would be possible (it would actually be easy today because the client binary runs the cloak server; harder in the future when we split client+server but still possible).

I guess the most challenging part would be setting up the trust model we've discussed in the past around this.

Then a copy-pasteable snippet could become a complete project template
I suppose the way this could work is that the copy-paste snippet would be either a cloak CLI command or a script that uses our embedded SDK. Not quite as nice as a graphql snippet, but maybe good enough?

[manic tapir]

Yes this where I got stuck. If it’s random executable code then it’s not “copy pasteable” in the way that I mean (too host-specific). I don’t know how to solve that.

maybe cloak needs a builtin repl / scripting language ?

[cinder oriole]

This has crossed my mind a lot. Graphql queries are almost this except for the limitations with chaining where you can't e.g. build two objects via chaining and then plug those two objects into another chain (can elaborate if that sentence makes no sense). But maybe there is a way to do that. There is something called apollo link which seems to solve this (see option 3): https://www.apollographql.com/blog/apollo-client/using-apollo-link-to-handle-dependent-queries/

Apollo link is not part of the graphql spec, it's a custom thing on top, but that doesn't mean we couldn't use it or be inspired by it.

Another alternative is that the cloak CLI (aka bash sdk) is the repl. Need to revisit that and figure out how to make passing outputs to inputs not extremely ugly (as it was in the bash sdk prototype from before the graphql switch).

And really we also shouldn't rule out what you originally described in terms of extending the core API with the ability to directly request resources from the client (and handle the trust model issue). Then snippets wouldn't have to run on the host. You could, e.g., have a python snippet that references client resources but is actually run in buildkit. Something like

cloak do <<EOF
#!/cloak/python
import cloak
...

• This is a half-baked example, not sure what #!/cloak/python means but possibly something there, idk

[manic tapir]

Yes that sentence makes sense 🙂 It's the lack of composition in graphql that makes it almost enough for this, but not quite. Extending graphql could be one answer but it's scary to me, because if it deviates from the graphql mainstream, then we risk repeating the Cue mistake ("I don't want to learn this new and weird thing").

The downside of adopting a scripting/config language other than gql (eg. via a repl or builtin interpreter), is that it probably won't mesh well with gql: queries in strings, lots of boilerplate to invoke them and get the result... In the end end it might be quite ugly.

[cinder oriole]

It's possible, but what I'm imagining is that the snippets in a given scripting language would be almost indistinguishable from implementations of extensions themselves. So if the boilerplate is ugly, that means that writing extensions involves too much ugly boilerplate and that we need to make improvements there.

I'm starting to come around on the original idea here of being able to access client resources via the api. Obviously need a trust model, but I can see a lot of nice benefits if we enable users to just pass scripts written in arbitrary languages directly to cloak. This would almost be like a 3rd variation on some of what we were discussing yesterday:

1. I can use cloak by just executing existing extensions from the cloak cli or from an embedded client
2. I can use cloak by authoring new extensions and then executing them (via cli or embedded client)
3. new variation - I can use cloak by passing scripts to the cloak cli, which will run in containers but have access to the client api

3 is actually just a very slight variant of using an embedded client in 1. It's just that instead of having to import the cloak library and execute the engine callback directly on your host, you just provide a script with the engine callback and cloak executes it for you. Benefits of it are:

1. Can pass around snippets
2. No limitations of low-level gql queries
3. Language agnostic, choose whatever language you want. For this use case in particular we will be biased heavily towards languages like bash, python, cue(?)
4. Avoids host-dependencies

Sorry if I'm just restating your original idea, typing it out explicitly helps me think through it