#maintainers

1 messages · Page 1 of 1 (latest)

tepid nova
#

@cosmic cove 👋

wet mason
#

[@civic yacht: moving the discussion here]

Struggling with graphql-go. Apparetly there's no way to "extract" (print) the schema. The only stuff I found in there is introspection, but that returns the data in a different format than gql

civic yacht
wet mason
#

Yeah, been looking into that, but it takes an AST and the "schema" itself (graphql.NewSchema) doesn't have any AST representation

civic yacht
#

Ah okay, yeah not sure off the top of my head then. Wouldn't be surprised if it exists buried somewhere in the library. What's the need to print it after its loaded though?

wet mason
#

There's no loading with graphql-go, it's code first

civic yacht
#

Oh right, that's why I was using that graphql-go-tools repo before (the other one named that, not the wundergraph one)

wet mason
#

Yeah, I've been trying to use graphql-go directly

#

graphql-go is already half maintained, graphql-go-tools wraps it and has like 15 stars and 3 commits for the year, was trying to go "mainstream" if we can

civic yacht
#

Haha I know, none of those are a good foundation, we encountered that bug with parsing the doc strings the other day too

#

I'm thinking of what other options there are (besides finding a replacement for graphql-go, which is what we need to do in the longer term)

#

Like I said before, in the very short term it's fine to just use gqlgen for core and submit the whole query since we won't really have the same caching problems that we have with non-core packages... It's hard to say if that's better or worse than continuing to use graphql-go-tools for now.

For the slightly longer term, I'm almost done updating all the existing actions to use the new resolver-per-execop model. I'm curious what the code generated for the core schema will look like. The code I wrote last week handled subresolvers, so I think it wouldn't be that far off from already working with core now. I just need to try it out and see what it looks like.

wet mason
#

Submitting the whole query only works so long as it exclusively uses core types, unless I'm missing something?

#

I guess the only way it would work is if core uses federation, but that's another headache

civic yacht
#

Yes, it would have to be that the resolver for anything under core just submits the part of the query that involves core, so yeah that actually would require a little bit of query manipulation, pretty ugly.

Let me just double check I understand the problem:

  1. The core schema needs to be obtainable as a string so that it can be used for client stub generation
  2. The core schema needs to be loaded into graphql-go , which doesn't inherently support loading schemas from strings (you have to use the shitty graphql-go-tools lib)
  3. graphql-go doesn't have any way of converting a code-first schema back into a string

If so, then I suppose the other very short term possibility is to just maintain the schema in a file and in code.

#

Which is obviously terrible, but it would unblock us and not rely on graphql-go-tools. My main concern right now is that I just want to see the resolution of FS as returned from a user action working under the new one-resolver-per-execop model

#

If we can see that working, then we can be confident that it will work and then go find a better solution than maintaining the schema in 2 places.

wet mason
#

Yes, it would have to be that the resolver for anything under core just submits the part of the query that involves core, so yeah that actually would require a little bit of query manipulation, pretty ugly.

Not only that, but a federation or stitching equivalent ...

yarn { build(...) { exec(...) } } --> we can't just send an exec() out of the blue, we're lacking context. We'd need to send filesystem(id: XXX) { exec(...) } to "resume" the query (what stitching does by hand and federation automatically)

  1. The core schema needs to be obtainable as a string so that it can be used for client stub generation

Come to think of it, we could do without for client stubs.

  1. The core schema needs to be loaded into graphql-go , which doesn't inherently support loading schemas from strings (you have to use the shitty graphql-go-tools lib)

yep

  1. graphql-go doesn't have any way of converting a code-first schema back into a string

yep

civic yacht
#

Come to think of it, we could do without for client stubs.
Yeah, I mean just to unblock our testing of seeing this all work e2e, we can temporarily skip that, in which case we wouldn't ever need the core schema as a string right?

#

We probably need to temporarily skip client stub gen for core anyways since we don't know how the chaining is going to work there yet

wet mason
#

Even long term, stub generators are capable of generating from introspection, they don't need the actual schema

civic yacht
#

True true, using the schema as a string was just convenient for now

wet mason
#

yeah

#

I just don't see a clean long term path, yet, for this gqlgen-go stuff

#

*graphql-go, but gqlgen-go is actually a good code name 🙂

civic yacht
#

Also apparently there is a tool that converts introspection json to graphql schemas: https://github.com/potatosalad/graphql-introspection-json-to-sdl
Can try it online here: https://codesandbox.io/s/graphql-introspection-sdl-svlx2

No clue if it works but another possibility if ever needed.

I just don't see a clean long term path, yet, for this graphql-go stuff
Yep it's not awesome. A couple of the options are:

  1. Fix graphql-go upstream or fork it if strictly needed
  2. Find a better server written in Go that supports dynamic loading of schemas (i.e. not gqlgen)
  3. Use the typescript Apollo implementation. Pros: High-quality, supports dynamic stuff we need. Cons: not in go, which is fine cause we can run it in buildkit but complicates architecture.
  4. (Utter last resort, only feasible in the long term) Make our own go server??
wet mason
#

Yeah agree with your assessment

#
  1. Find a way to add dynamic entries to gqlgen. I don't think it's likely, but one can dream
civic yacht
#

My current thinking is that we probably should go with 1. for the immediate term. graphql-go isn't great but the bugs are just obscure enough that they aren't complete dealbreakers. Avoiding graphql-go-tools helps.

2. can be an ongoing research effort, there are plenty of others out there worth a try at least. 3 and 4 are only if we reach the "long-term" and still have had no luck

#

Yeah 5. isn't worth ruling out

#

Or 5B: fork gqlgen

wet mason
#
  1. graphql-go-tools (the other one, wundergraph), as a "broker" of some sort. Not likely either, but not impossible
civic yacht
#

graphql-go-tools
Actually one question, is graphql-go-tools a layer on top of graphql-go that adds federation router support? Or is it a complete separate implementation? I originally thought it was the latter, but want to double check

wet mason
#

completely separate

civic yacht
#

Cool okay, that makes it slightly more intriguing then

wet mason
#

it's not really meant to resolve on its own, their examples dispatch to gqlgen servers

civic yacht
#

I see, yeah still worth a look if we become desperate

wet mason
#

haha. "in 3, 2, ..." 🙂

#

ok i'll try tomorrow 1., lower my expectations a little bit and see if i can find something acceptable short-ish term

#

I was trying to find a long-term solution but that's just not happening right away

civic yacht
#

Haha, I mean I think we can get by with graphql-go for a little bit here. So more like `5, 4, ..."

But the consolation for me is that the fallback option of using apollo server is honestly not that bad. It's annoying to setup since it's not go, but if you get over that hump it's a perfectly good option. So that's not bad as far as worst-case scenarios go.

civic yacht
wet mason
#

Well we'd probably still need to "federate" from there to a core server in Go, because of LLB and anything-but-Go is not realistic

civic yacht
wet mason
civic yacht
wet mason
#

(specifically, TO core)

#

since core actually does need "federation" (because we're daisy chaining actions into filesystem sub-solvers)

#

Anyway. I'll re-try tomorrow from a clean slate, I'm too deep in the rabbit hole now

civic yacht
wet mason
#

It’s a hard come back from federation, I thought we were really close to have solved all of this … except for the nasty cache

civic yacht
# wet mason It’s a hard come back from federation, I thought we were really close to have so...

Caching is hard, as they say... But yeah it sucks cause it was so close to what we needed.

Consolation: while the code I'm writing that handles the resolver-per-execop model is ugly at the moment (at least in Go, it's actually totally fine in TS), I'm starting to see how it can become non-ugly and we could still make it easy to add new SDKs in the future even if it's not as "off-the-shelf" as federation. Can show more tomorrow

wet mason
#

Nice!

tepid nova
#

Would you guys be ok to catch ne up on the latest exciting graphql implementation excitement tomorrow after the demo? I could use a mental checkpoint 😅

civic yacht
civic yacht
#

(for tomorrow) @wet mason I took some shortcuts to save time, but you can now run the following query from my branch:

cloak -c examples/alpine/dagger.yaml query <<'EOF'
{alpine{build(pkgs:[]){file(path:"/etc/alpine-release")}}}
EOF

where alpine build returns the chainable Filesystem type . Changing the path you read from the output of alpine build doesn't invalidate the cache for the parent resolver. Branch here: https://github.com/dagger/cloak/tree/cache-per-resolver

Fortunately all the shortcuts I took to get that query working are conceptually straightforward to cleanup+generalize (i.e. right now I just return the base alpine image because I didn't want to construct the chained query of installing arbitrary numbers of packages, I need to update the new go codegen stuff to automatically use Filesystem instead of FS, need better deserialization of the Filesystem type in the server when returned by user actions, etc.).

Think the basic concept is at least proved now though!

tepid nova
#

DX question : it still an ongoing discussion how to invoke dependencies from a graphql query, and how it interacts with chaining? ie. the “alpine” from snippet above

civic yacht
# tepid nova DX question : it still an ongoing discussion how to invoke dependencies from a g...

Yes absolutely, the current effort is just meant to give our server the basic ability to have queries that include resolvers from different schemas (i.e. alpine is one package but the other resolvers are on Filesystem from the core package) in addition to a few related problems around how resolver execution is cached.

The DX around how all that is actually presented to users is very open to discussion; at this exact moment we are just focusing on the low-level features needed for these use cases. Will be easier to explore more once that's in place.

tepid nova
#

Agree 💯 that’s how I remembered it, just wanted to make sure before making random DX suggestions 🙂

tepid nova
#

another topic: maybe “custom SDKs” are actually plugins. With SDKs being core

#

  • Use the CLI to run automations, interactively or from a shell script, makefile or CI script

  • The CLI can load plugins to access additional actions, artifacts and services. Examples of plugins: alpine, docker/compose, github/actions, yarn. Some of these require dynamic actions (eg. load a docker-compose project from a directory, expose its services and artifacts. Load a package.json, expose its scripts as actions; etc)

  • Use the SDKs to 1) run Dagger automations in your tool or 2) develop plugins

tepid nova
#

Something worth investing effort into: not only user docs but also design docs

civic yacht
# tepid nova - Use the CLI to run automations, interactively or from a shell script, makefi...

Yeah I think the definition of SDK is quite broad and also tied up in some of the ideas around package management. I think it's something like:

  1. An action package implementation consists of a set of pretty much arbitrary artifacts. This may be source code or, like you said, stuff like docker compose yam, package.json, makefiles, k8s yaml (?), etc.
  2. When an action package is imported, an SDK is specified, which is a way of converting those arbitrary artifacts into a set of actions invokable via dagger. It will be common for a package author to include a specification of the SDK that should be used to package their artifacts and make them invokable, but it's not strictly required. It's also possible for the package user to specify the SDK when they "import" the package.

I need to think about the terminology a bit more here though, what we are currently calling an SDK is not exactly the way I'm using it above. Either way, I think I'm in agreement with your general sentiment

civic yacht
# tepid nova Something worth investing effort into: not only *user docs* but also *design doc...

Yep I started updating the original (pre-graphql) proposal in notion with a new overview of the architecture (much more succinct than before). Was planning on chipping away at this a bit each day throughout the week. But maybe it actually makes sense to just put that in the cloak Github repo rather than keeping it in notion, so it's available to users too. I can port it over pretty easily.

wet mason
civic yacht
#

Obviously we can adjust the terminology as time goes on pretty easily

tepid nova
#

Welcome @orchid cosmos 🙂

#

You should have received an invite to join the repo

#

Looking forward to building cool stuff on cloak with you!

orchid cosmos
#

👋

tepid nova
#

@wet mason @civic yacht my list of candidate terms from our discussion:

  • action (unit of code that dagger can run)
  • runtime (special code that helps dagger run actions from different sources)
  • sdk (everything a developer needs to develop for Dagger in their favorite language)
  • provisioner (special code to install dagger on a particular infrastructure)
  • plugin (a way to extend dagger with additional actions, runtimes and provisioners)
#

(i’m actually warming up to “runtime” @wet mason 🙂 )

wet mason
#

I guess to go back to the lambda analogy, you could have the "go sdk" but the "go-1.18 runtime"?

#

re what I was mentioning earlier, the current Dockerfile for alpine is this:

# syntax = docker/dockerfile:1

FROM golang:1.18.2-alpine AS build
WORKDIR /src
RUN apk add --no-cache file git
ENV GOMODCACHE /root/.cache/gocache
RUN --mount=target=. --mount=target=/root/.cache,type=cache \
    CGO_ENABLED=0 go build -o /out -ldflags '-s -d -w' .

FROM scratch
COPY --link --from=build /out /entrypoint
#
# the following 2 lines should go away once the SDK supports introspection
#
COPY --link ./examples/alpine/schema.graphql /schema.graphql
COPY --link ./examples/alpine/operations.graphql /operations.graphql
ENTRYPOINT ["/entrypoint"]
#

it's a "generic" go dockerfile, we don't need anything special in the build phase

#

the interaction between the cloak server and the SDK is done at runtime (AddMount() of the payload request)

civic yacht
wet mason
# civic yacht I think what we currently call the "server sdk" is actually exactly the same ide...

I think what we call the "server sdk" is also what they call the SDK (e.g. github.com/aws/aws-lambda-go/lambda). "Libraries, samples, and tools to help Go developers develop AWS Lambda functions."

And what they define as runtime I think it's just bootstrapping that'd be covered by the OCI spec (they support running outside of containers so they had to define their own thing).

Here's their example:

#!/bin/sh
cd $LAMBDA_TASK_ROOT
./node-v11.1.0-linux-x64/bin/node runtime.js

which is basically a WORKDIR + ENTRYPOINT in OCI

tepid nova
#

@wet mason you’re baking in the assumption that the ideal DX in every language will always be to write a complete runnable program and main function manually. I don’t think we know that for sure. We shouldn’t make strong assumptions on how much boilerplate each sdk may need to manage; certainly we shouldn’t assume it will always be zero

#

a runtime is “basically a workdir + entrypoint” on top of a vanilla dockerfile only in a zero boilerplate situation, which may not be universal

civic yacht
#

Here's their example:
That's what they call a bootstrap, the runtime is the code inside runtime.js :
Your runtime code is responsible for completing some initialization tasks. Then it processes invocation events in a loop until it's terminated.
So I think it would be more like WORKDIR+ENTRYPOINT is a bootstrap and the command invoked by the entrypoint is the runtime, which corresponds to our use case reasonably well, though I don't know if we need to borrow the "bootstrap" terminology, that seems kind of confusing

wet mason
#

@civic yacht Found another one: https://github.com/nautilus/gateway

This one doesn't do proper apollo federation, just manual merging. And it's using the same underlying AST library as gqlgen

wet mason
#

@civic yacht Quick update on graphql-go:

  1. Have a better grasp of how the AST parser works, how graphql-go works, etc
  2. Started to write some code to convert AST (what we get from the action) into our own graphql.Schema (what graphql-go wants)
  3. It's pretty powerful, we can do everything programmatically (since we're rebuilding the schema), can hook into everything (e.g. I was able to support extend type, etc)
  4. We have fine grained control on how to handle resolvers.
  5. It's complex. Like, even for straightforward things such as figuring out the type of a field it involves recursion ("String" is actually wrapped in an ast.Named wrapped in a ast.NonNull)
  6. I've realized that deep down, that's what graphql-go-tools does. Basically a bit weird since we're parsing the schema to figure out a resolver map, and its parsing the schema again to figure out where to put those resolvers
  7. I'm looking to see if maybe we can re-use parts of tools, but manipulate graphql objects directly rather than their abstractions (e.g. just use them to convert AST into graphql, then deal with that directly)
civic yacht
wet mason
#

[String!]! is a NonNull->List->NonNull->String, and I'm forgetting some layers.

There's directives, interfaces, enums, ....

There's dependencies (e.g. to generate type Foo { bar: Bar! }, you must first handle Bar)

And other complexities

Since this work is purely for graphql-go, and it's purely if we do ast->graphql (e.g. irrelevant if we use introspection queries), it's likely we'll throw it away. So I'm going to try and reuse bits and pieces of -tools if possible

civic yacht
#

Yeah it seems like 100 years ago but I had to deal some of that ast parsing with the previous lazy implementation: https://github.com/dagger/cloak/blob/main/api/lazy.go

(Side note, I think we can safely delete all that code now, I don't think there's any chance we are going back to that approach)

It's tedious but ultimately pretty logical.

So I'm going to try and reuse bits and pieces of -tools if possible
Yeah no sense in re-writing the exact same code. The only huge issue I ran into was that bug when we added docstrings to the schema and it would randomly just output a completely empty typemap

#

Trying to remember the exact problem call there

wet mason
civic yacht
tepid nova
#

how easy is it to generate a graphql schema programmatically? Is there a simple go library to do that ?

civic yacht
#

The code we currently have for doing that is a mess though because I didn't know what I was doing when we embarked on this graphql stuff, so that's what Andrea was talking about iterating on above

tepid nova
#

Got it thanks

#

I’m exploring the idea of “drop-in runtimes” that can auto-detect and run all pre-existing automations directly from dagger: docker-compose projects, makefiles, npm script, dockerfile targets, etc

#

that would be a game changer for ease of adoption. Basically zero-code onboarding

#

I’m assuming these runtimes will need to implement a special hook (itself running on dagger of course) that takes a source repo as input, and outputs a graphql schema exposing all the actions, artifacts and services that it found.

#

In that case there’s no sdk to speak of. Since there is no development necessary. So “runtime” makes perfect sense.

#

Presumably the Go & Typescript SDKs would ship with a runtime of their own, implementing the same hooks. Perhaps their implementation will be very simple (if there is no boilerplate to generate) or become more advanced (if code and/or schema generation is involved)

#

This matches common use of the terms “runtime” and “sdk”, as in “the JDK vX ships with the JRE vY”

civic yacht
#

I’m exploring the idea of “drop-in runtimes” that can auto-detect and run all pre-existing automations directly from dagger: docker-compose projects, makefiles, npm script, dockerfile targets, etc
Yep this would be insanely powerful. You could essentially import anything into your code and call it with typed, generated clients. Pretty much endless possibilities, but will mention that this could be an approach to support backcompat w/ Europa (you can import a dagger.#Plan from a cue file or something). That's a tangent though

#

It's also not just that you can import anything and call it, you can do so in a way that takes advantage of buildkit's caching and other features, which makes it not just a convenient interface to existing tools but potentially an actual enhancement of them in terms of performance too

tepid nova
#

Yes all of that 👍👍👍

#

Also! It’s the missing first step to onboarding. before writing any code to call these actions, you can invoke them straight from the cli. From there: shell script; then as your tooling matures: new code using SDK.

#

This makes the learning curve very gradual which is the opposite of today…

#

So it seems like you’re saying I can get this for christmas? 😁

civic yacht
# tepid nova Also! It’s the missing first step to onboarding. *before* writing any code to ca...

Yeah there's pretty much a universe of possibilities just from the CLI before you write anything; any makefile, any npm script, anything with an API more or less. In combination w/ future magic caching it gets really crazy (shared public caches backing all of this, making it instantaneous if anyone else has run the same thing before).

I think the hardest part of this will often be writing the part of the runtime that derives the API of whatever it's running. Like I know it's possible to figure out the "api" of a Makefile, but that sounds like a lot of work. Once you've done that though, you'd need to generate a graphql schema for it (tedious but straightforward) and write the part that invokes the Makefile based on the input from cloak (the protocol we discussed before), which is probably straightforward in this case.

So I cautiously think having the underlying mechanisms in place in cloak should be possible by Christmas 🙂

#

Whether we could figure out how to robustly parse a Makefile by that point is a different question, but I'm sure other formats are much more amenable to the type of parsing we'd need to do to derive their API

tepid nova
#

I meant the concept of christmas but the date works too actually 😁

civic yacht
tepid nova
tepid nova
#

Welcome @bronze hollow 🙂

bronze hollow
#

😀

empty waveBOT
tepid nova
#

Well, it worked 🙂

#

With all the ideas and exciting discussions flying around, I thought now would be a good time to move those to a more permanent form: github discussions etc.

empty waveBOT
#
shykes
[cloak] New discussion #17: Zero\-code adoption using drop\-in runtimes

I really, really love the idea of "drop-in runtimes" that can auto-detect and run a project's pre-existing automations directly from dagger. For example:

  • docker-compose projects
  • makefiles
  • npm scripts
  • dockerfile targets
  • etc.

This would enable zero-code onboarding, meaning that a new user could get value from dagger without having to write a single line of code. That would be a game changer for ease of adoption.

I’m assuming these runtimes will need to implement a special...

tepid nova
#

As discussed this morning @civic yacht @wet mason @hasty basin @rancid turret @cosmic cove 👆🏻

civic yacht
tepid nova
wet mason
#

They do IoT stuff and so in their examples they end up federating gRPC services over a single graphql API. So the "federation transport" is completely swappable. The code is pretty nice, much much nicer than the other framework we saw

wet mason
#

@civic yacht Another thread: in "one resolver call at a time", how does the action know which resolver it needs to call? What happens with nested resolvers?

tepid nova
#

In the alpine action, what is generated from what? Code scaffolding from graphql schema?

civic yacht
#

As opposed to other languages like python that have some really nice code-first graphql tooling already

empty waveBOT
#
aluzzardi
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

Definitely interesting.

/cc @marcosnils -- we were briefly discussing no/low code adoption yesterday, this approach could be it (actions are available to run before you write a single line of code).

I think the gap to make this possible is smaller than it appears. I don't think we would need to generate schemas on the fly, GraphQL is flexible enough that a static schema works. For instance for make:

# List the targets
make {
  targets {
    name
  }
}

# Run `mak...
cosmic cove
#

Welcome @round granite ! 👋

round granite
#

thanks! 👋

empty waveBOT
#
sipsma
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

One downside of a static schema is that the generated client stubs for them will be less useful

I was going to say the same thing, but I think this doesn't have to be a binary choice. We could support runtimes supplying a static schema (in cases where that makes sense, or in this Makefile example, maybe that's just what's practical for a v1 implementation even if the more typed version would be preferable). But we could also support them providing dynamically generated schemas too. Both ...

empty waveBOT
#
sipsma
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

Addendum to previous comment: Actually if we have a totally generic untyped API then we probably wouldn't need a runtime at all; it could just fit in the current action model today. I could already just go add a make action written in Go or TS that accepts as input a filesystem and a target to run. So maybe viewing this as a custom runtime only makes sense in the case where we are dynamically generating a schema derived from the makefile? Not sure yet, worth thinking through this distinctio...

#
sipsma
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

One quick note to self and @aluzzardi about implementation details: this goal seems like it will require that we support subresolvers in user actions (i.e. fields not directly under Query can also be explicit, custom written resolvers). We weren't sure if we wanted to support those initially just because it's more to document for developers, but this all seems like a compelling enough use case that we should. Luckily it won't be a huge refactor or anything like that, just need to add a few ...

empty waveBOT
#
aluzzardi
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

One downside of a static schema is that the generated client stubs for them will be less useful (because the schema contains less information about what actions are available). So a "build" action implemented as a yarn script would not be a first-class citizen compared to one implemented in a native SDK: no auto-complete from the IDE, no built-time type checking etc.

That's the typical GraphQL pattern, to have static schemas (e.g. you'd query { users(name: "aluzzardi") { email } } rath...

tepid nova
#

FYI I cleaned up the README a little bit

#

(I merged my own PR, but will only do that for README)

empty waveBOT
#
sipsma
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

For bindings, doing things like for _, target := range make.Targets() { make.Run(target) } becomes complicated (unless you use reflection to figure out what's available).

I think the way this would work is that:

  1. You are writing an action that relies on some Makefile, so you add a dependency (to cloak.yaml or whatever replaces it) that points to the fs containing the Makefile and specifies the make runtime
  2. You then run cloak generate which introspects the makefile and dynamica...
#
shykes
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

That's the typical GraphQL pattern, to have static schemas (e.g. you'd query { users(name: "aluzzardi") { email } } rather than { aluzzardi { email } }.

But if I implement a build action, I want to write func Build(), not func Action(name string) { if name == "build" { .. } }, right? And I want my action to be invoked by client with foo.Build(), rather than foo.Do("build")? What I'm saying is, if we want that pattern for actions written in Go, we would want the same pattern...

#
shykes
[dagger/cloak] New comment on discussion #17: Zero\-code adoption using drop\-in runtimes

Maybe we are not talking about the same thing with the term "dynamic schema". I'm talking about having a middleware that produces a schema for cloak to consume. Whether that schema is dynamically generated is up to the middleware. Once generated and loaded by cloak, it is "static" and therefore would require no particular graphql magic, and it would not complicate the usage patterns.

tepid nova
#

Are these github notifications too verbose? We could create a separate #cloak-activity if so

civic yacht
empty waveBOT
tepid nova
#

OK I'll take care of it

empty waveBOT
tepid nova
#

@civic yacht @wet mason I'm around if you're feeling up for a live conversation. I have some DX thoughts that I want to share, but they're too vague to write down productively...

#

Don't want to break your flow though

#

The best title I can think of is "API-centric vs. CLI-centric"

#

And the evergreen "terminology"

civic yacht
tepid nova
#

I can work on a PR for some of the small cosmetic changes to the v1 demo that we talked about

#

dagger.yaml -> cloak.yaml etc

#

assuming you haven't gotten around to that yet

civic yacht
tepid nova
#

OK, I just created my very first PR from VS Code... apologies in advance if I messed it up

#

I haven't changed dev setup in... Well I prefer not to say 😅

wet mason
civic yacht
wet mason
#

re the dockerfile field: I think @civic yacht had a few ideas about that (I don't)

the background of the problem is: docker build . doesn't work on our current actions because they're doing "relative imports" (because they're in the same repo as the SDK)

so instead we do a pseudo docker build . -f examples/netlify/Dockerfile from the repo root, hence the local: <repo root> + dockerfile: <that file>

#

if the actions were on a different repo we wouldn't have that problem, but that's how it is now

#

(the dockerfile field is optional, it's just that we need to pass that option for our specific repo setup)

#

if we vendored the sdk inside the action we'd get rid of that problem, but it'd make development cumbersome

tepid nova
#

Yeah I’m thinking a very simple cosmetic change that keeps the functionality intact:

  • Rename the config field dockerfile to source
  • Remove the trailing /Dockerfile from that field in all yaml files
  • Change the go implementation to append /Dockerfile to the path after loading it
#

so eg. dockerfile: examples/netlify/Dockerfile becomes source: examples/netlify

#

The desired effect is simply that people don’t see the word “dockerfile” at that stage of the demo, to avoid distracting them with an implementation detail and questions of how much of an implementation detail it actually is.

civic yacht
# tepid nova Yeah I’m thinking a very simple cosmetic change that keeps the functionality int...

My original idea was that we could cheat by abusing the fact that our examples all exist in a single go module, which we could have used to set the docker build context as the place where the go.mod is. I have a memory of there being some simple go library call that let you get the nearest go.mod of parent directories, but I can't find it on googling so I'm probably just crazy, nevermind.

Either way, the idea would have been a dumb hack, if we are just trying to avoid dockerfile this a much simpler solution. Can also rename local to context in this case. It will still look weird but that's fine for now.

#

We do also already have a delightful hack for how we build the shim binary: we embed the source code of it into the cloak binary and then build it with buildkit when starting up. Technically we could do that with all our "universe" packages too, but not worth it yet

tepid nova
#

Yeah local is verbose & also leaks implementation detail into the demo, but less distracting than the word “dockerfile” in this context. If there’s an easy way to hide local that would be great, just lower priority IMO

tepid nova
civic yacht
tepid nova
#

sounds good

#

Thanks for another great demo today and @round granite for the great feedback

civic yacht
#

Agreed, really appreciate the feedback from everyone so far!

tepid nova
#

What's the difference between source and core in the graphql API? I got a little confused trying to write valid queries in the interactive sandbox.

#

For example there is core { image } and source { image } with different interfaces, I wasn't sure which to use

civic yacht
tepid nova
#

btw I was able to build and use cloak without problem 👍

civic yacht
#

source will be gone, only core will exist

tepid nova
#

OK got it

#

For the purpose of demo, may I suggest renaming source to core2 to clarify to the audience what's going there

#

(only if it's an easy change to make)

civic yacht
#

we are hoping that my change will be in tonight, so then Andrea and I can practice tomorrow and if it all is smooth, we won't have to deal with that at all anymore

tepid nova
#

I gave an impromptu demo to Tibor (buildkit dev) and that was a point that confused him (I didn't have a good answer so we were confused together)

tepid nova
wet mason
#

@civic yacht Do you think we'll keep aggregating schemas using the original schema file (e.g. read that file from the image) or using the introspection format? (e.g. exec the action and request the schema back)

Asking because it affects quite heavily the merging part

wet mason
#

but tomorrow we'll create a fresh demo tag once we've run through it

wet mason
tepid nova
#

yeah but there's no core in llb

#

so in the context of "choose between core and source" he did not catch the reference (and tbh I forgot about it)

wet mason
#

oh i see

civic yacht
#

Erik Sipsma3294 Do you think we ll keep

tepid nova
civic yacht
tepid nova
#

In the meantime I'll start replying to your comment

wet mason
#

I have time now or later

civic yacht
wet mason
#

@civic yacht Yep sure. Feel free to merge

#

I'm working from a different directory so no conflicts

#

@civic yacht Yesterday I finally found a good enough solution for the API, happy to chat about that

wet mason
#

@civic yacht So it's working out pretty well with the new design, i converted bits and pieces of core (as a standalone thing, still need to integrate back). I'm making heavy use of extend but now I'm wondering/hoping the code generators are going to be happy with that

wet mason
#

[=== note for later ===] @civic yacht We should think about multi platform sooner rather than later, before it throws wrenches in the API

#

(like it did in the CUE runtime)

civic yacht
# wet mason [=== note for later ===] <@949034677610643507> We should think about multi platf...

Agreed. Also cache configuration, which turned out to be hard to make configurable: https://github.com/dagger/dagger/pull/2519

Multi-platform and cache configuration also have a similarity in that they are both like "ambient" configuration, where you want to apply some global configuration that then potentially impacts every action in DAG. Not saying they need to be solved in the same way necessarily, but might be something there, idk

tepid nova
#

@orchid cosmos may have suggestions on this 🙂

wet mason
#

Let me know if you have any comments. Did a lot of cleanup (I think the most extensive is exec.schema.go)

#

for Monday Erik Sipsma3294 Almost done

cosmic cove
#

welcome @proven rock ! You will have repo access shortly 🙂

proven rock
#

Hey everyone! Thanks for inviting me, and thanks again for the chat earlier 🙂

civic yacht
#

@wet mason started by just seeing what gqlgen does for the implementation stubs when give extend type query. (starting thread)

wet mason
#

[for tomorrow] @civic yacht Just pushed a router branch, it's kinda working: https://github.com/dagger/cloak/commit/76b2fb36ff8b1776415aff6818de05a071ab3916

  • The "old" API server is completely shutdown, relying on the new router
  • Alpine is getting loaded, using extend type Query and I can see the schema in graphiql
  • Using the new filesystem { loadExtension } pattern
  • [todo] alpine is not working, requires sub-resolvers
  • [todo] engine is a patchwork, I need to clean that up
  • [todo] config is only half ported. Missing operations, missing grabbing the "core schema". All codegen is broken

The API is pretty neat, solved a bunch of problems:

  • remoteschema.Load() returns a router.ExecutableSchema
  • router.Add() allows to extend the API dynamically, atomically
  • Extension loading is not a special case: core/extension.schema.go, only a few lines of code
tepid nova
#

Thought experiment of the day: what if we pulled in the client API after all?

proven rock
#

Quick one: You mentioned during the demo that inner containers interfacing with the API have a socket mounted. How can I run just the service serving that socket (i.e. a kind of cloak serve), or would that be the API hosted by cloak dev?

proven rock
#

I've shared an initial thoughts doc with those of you who were in the meeting yesterday about TVL/Nix/Cloak integration. Anyone else here curious about it, ping me your Google account email and I'll invite you.

Also curious regarding our confidentiality discussion if it'd be fine for me to share this with 1-2 other people in TVL (who won't share it further).

civic yacht
# proven rock Quick one: You mentioned during the demo that inner containers interfacing with ...

Yeah so cloak dev is one option that will make the API available at localhost:8080. We also are working on what we call the "embedded" use case where you just import a cloak library which enables you to start up the cloak engine and make queries to it from code executing uncontainerized on your host machine. Right now only Go is supported for this, but in the long term we intend to support all of the languages we support authoring actions in.

civic yacht
cosmic cove
civic yacht
#

for tomorrow Erik Sipsma3294 Just

cosmic cove
#

welcome @warm tundra ! Thanks again for your time today 🙏

wet mason
#

@civic yacht Semi available this afternoon -- I just pushed secrets support (clean) and in-container router unix socket (needs cleanup), I was able to get alpine fully working (by luck -- even though we don't have sub-resolvers, I guess the right solver got called anyway). So it's in a good enough state to fix sub-resolvers etc

proven rock
#

Just got through reading all the comments on the doc, thanks for the input. I think it would be worthwhile to write up something about the ideas and goals of caching specifically in Dagger/Cloak vs. Nix, as I think we have some confusion about different terms here (which makes it hard to talk about stuff like how to mount a /nix/store into a Dagger build step, as we have different notions of why one might want to do that).

I'm happy to kick that off in the next few days with:

  1. my view of how this works in Nix (specifically, cache keys, distribution, "importance" of the cache)
  2. my view of how this works in Dagger/Buildkit (I'm likely missing a lot of details here, so input would be needed).

Does that sound useful?

#

(Also, of course, does anything like that already exist on your end?)

tepid nova
civic yacht
# proven rock Just got through reading all the comments on the doc, thanks for the input. I th...

Yeah agree a discussion post would be great, I'm more than happy to help fill in details about dagger/buildkit caching and see how we can get it to mesh with Nix. Unfortunately there is not really any documentation from buildkit on how its caching model works as a whole (just docs for a few selected parts) and there is thus not a ton yet from dagger either, but fortunately I've done enough work on buildkit that I either know the answer or know where to look in its codebase for the answer

tepid nova
#

Thank you for sharing your thoughts so quickly and thoroughly @proven rock it is much appreciated

tepid nova
#

@wet mason @civic yacht ok I may be warming up to the idea of keeping an extension build op in the core, in addition to an extension run. But I would really like the build entrypoint to target cloak itself (eg. not a dockerfile) and to support sharing by copy-paste (because it’s an amazing way to grow the community)

proven rock
cosmic cove
#

@everyone We are starting a weekly virtual meet-up for all Early Access Cloak users. Please add a 👍 to the time that works best for you. We are hoping to expand the time options soon, but will need to stick to PST mornings for now.

#

Option #1 - Wednesdays at 9 am PST

#

Option #2 - Thursdays at 10 am PST

tepid nova
#

Something I don't understand in the current extension loading flow @civic yacht @wet mason is how we handle transitive dependencies:

  • does each extensions have its own cloak.yaml?
  • If so, are all cloak.yaml files loaded recursively for each extension in a project's dependency tree?

Right now it's ambiguous because the only cloak.yaml I've seen with dependencies is for todoapp, which we only invoke directly in the demo. If another extension imported todoapp, would it work, with the right dependencies loaded in the right place?

civic yacht
#

Something I don t understand in the

cosmic cove
#

Welcome @copper snow ! Thanks again for your time today!

copper snow
#

Thanks for inviting me. I'm honoured and really excited about Cloak.

copper snow
#

Do you mind if I test cloak in public repositories? That would mean leaving cloak specific config in public repos, probably on branches though.

tepid nova
#

Not a big deal, it might confuse people, or maybe peak their interest? 🙂

copper snow
#

I'd most probably leave those tests on branches for now.

tepid nova
#

Also keep in mind that it's virtually certain that all our interfaces will break. Including name and format of config files, directory layout, graphql API, name and syntax of command-line tools...

copper snow
#

Yeah, I'm aware. I just want to play with it in a couple existing projects and I don't know what level of publicity is okay for you. I'm not going to use it for anything mission critical.

tepid nova
#

That's perfectly fine, thanks for checking

#

Worst case, someone asks "what's cloak?" and we can show them a demo 🙂

#

I have questions on the relationship between "projects" and "extensions". This is related to the topic of transitive dependencies.

wet mason
#

@civic yacht pushed some basic docstrings, api seems to be happy still

civic yacht
tepid nova
#

Continuing DX discussion : maybe 1) a project can optionally define top-level actions and types; each action has its own schema, code, build config etc; 2) a project can always be imported as an extension to another project. If you import an extension which doesn’t define any actions or types , it will simply add nothing to the importing project’s scope (except perhaps a top-level query container with nothing under it)

#

So, in its strictly technical definition, every project is an extension; in the common meaning, an extension is a project whose purpose is to implement types and actions to extend other projects

#

OR we define an intermediate format for an extension bundle, and then the extension is the artifact produced by one project and imported by another

#

That artifact could be a docker image; or a generated directory in the project dir; or it could be a copy-pasteable query specifying how to produce either of those

tepid nova
#

Related: it might be very useful for each extension to have access to the root directory of the project it is extending.

#

(similar to core.#Source it it returned the plan directory instead of package directory)

#

(time to open a bunch of issues 🙂

tepid nova
tepid nova
#

Does gql support maps? The equivalent of this in Cue: foo: [string]: ...

#

Or do I need to use a list eg. foo: [ {name: string, ...}]

copper snow
#

AFAIK gql doesn't have a map type.

The way you describe projects and extensions above: it reminds me of GitHub Actions: a repo itself could contain an action that you use in a workflow (defined in another repo).

The delivery format is either JS in the referenced repo or a container image.

Personally, I like the simplicity of GitHub Actions from both a consumer and action writer perspective and Universe (and the implied CUE dependency mgmt) resembles the same experience, so from a DX perspective I would aim for the same from a high level perspective.

civic yacht
# tepid nova Does gql support maps? The equivalent of this in Cue: `foo: [string]: ...`

Like Mark said, there is no builtin map type. The options I know of are:

  1. Return list of key,value pairs like you said
  2. Model as a field that takes args, e.g. add a field like foo(key: String) String. This is the approach we use to enable obtaining mount outputs after an exec: https://github.com/dagger/cloak/blob/78c76e1821fe2e845f670af9745570af80e4f5a3/core/exec.schema.go#L53

Most of the arguments against adding a map type don't really apply to our use case, which is unfortunate: https://github.com/graphql/graphql-spec/issues/101#issuecomment-170170967

But I think the alternatives are good enough, plus in higher-level sdks we can add sugar to hide this (i.e. ability to turn a map object into that resolver pattern of foo(key: String) String)

wet mason
#

@civic yacht I noticed the "field heuristic" is gone now -- doesn't that mean we're invoking the extension even for trivial resolvers?

tepid nova
#

Is there any way we can get rid of cloak generate in the development workflow? Is it standard enough that we can get all our developers to embrace it; and a good enough pattern that we will never want to get rid of it?

#

Personally I hate generated scaffolding, it’s like a reminder that my platform is not good enough (or I wouldn’t have to commit boilerplate in my repo). But maybe it’s just me

#

I guess the main benefit is that the full generated boilerplate is what your language tooling sees, so filenames & line numbers will always match, you can always look at the code etc

#

of course the code you look at may be inscrutable, but at least you can look at it

#

But it makes me sad that there’s no better way

#

because that boilerplate is fugly

civic yacht
#

The problems it's currently solving are:

  1. I don't want to write schema.graphql and then have to write corresponding function signatures (input args and output type) when I go implement the actions
  2. Some generation of boilerplate that helps the runtime call the correct function when invoked
  3. Generation of clients for invoking dependencies

I would say that 2 is just an artifact of being a prototype; it should eventually be possible to, e.g., just provide some go source files to the SDK and it will take care of generating that boilerplate internally before building it into an executable extension (it would never need to be committed to your source code repo).

1. is an inherent property of a schema-first SDK. I like code-first approaches much more and it might make sense for us to emphasize use of code-first SDKs, but I don't know if we can ever make that an absolute requirement? I think it probably has to be up to the SDK (medium-low conviction here)

3 is probably a requirement unless we figure out the route where we have some cloud service that integrates w/ each languages package manager and generates stubs on the fly (as discussed a while back now)

#

So overall answer is that cloak generate could, in time, be reduced in scope and probably morphed into something that looks different (maybe it's just a cloak do call to an sdk extension rather than its own special cloak subcommand), but some of the features it currently provides will probably need to continue existing. If that makes sense?

civic yacht
# tepid nova because that boilerplate is fugly

Also I agree, my plan today is to go fixup a lot of the little worts that have popped up in terms of DX or that are easier to address now that we've made adjustments elsewhere in the implementation. One of them is going to be hiding that boilerplate more, it grew much larger with a change yesterday, I can do a couple simple things to get it back to looking like it did before.

tepid nova
#

Ok I see. So that “part 2” is our lever for picking low-hanging fruits. Once we’ve pulled that lever we can re-assess and see if the DX is good enough that it’s not an issue anymore.

Right now these parts 1,2 and 3 are indistinguishable for me so I’m missing that dimension

civic yacht
# tepid nova Ok I see. So that “part 2” is our lever for picking low-hanging fruits. Once we’...

Yep exactly. I think when we switch to the model we discussed this morning where SDKs become extensions it will become trivial to address this, so that'll be a good time to fully fix it. What I'm doing today as a short term will separate it out from user action implementation code and hide it, which is more just a small step in the right direction.

And yeah 1 and 3 are currently all mushed into cloak generate so it's hard to tell the difference between them. It might make sense to split the concept of a stubber out into these two subproblems so they can in theory be addressed separately: https://github.com/dagger/cloak/blob/main/docs/concepts.md#stubber

tepid nova
#

In my mind "stubber" would become an implementation detail of the SDK. Ie. part of its generate hook probably?

civic yacht
tepid nova
#

Ah that reminds me of one element of yesterday's DX conversation that didn't make it in the config docs draft... Needing generated client stubs for my project (to write custom client code) without necessarily having implemented actions of my own

wet mason
tepid nova
civic yacht
tepid nova
#

I'm exploring ways to get more mileage out of pure graphql queries. Any reactions to this sequence of queries?

    universe(version: "cloak") {
        netlify { install }
        yarn { install}
    }
}

{
    core {
        getenv(key: "GITHUB_TOKEN") {
            save(key: "$token")
        }
    }
}

    core {
        git(remote: "https://github.com/dagger/todoapp") {
            save(key: "$src")
        }
    }
}

{
    yarn {
        build(source: "$src") {
            save(key: "$build")
        }
    }
}

{
    netlify {
        deploy(token: "$token", contents: "$build")
    }
}
civic yacht
#

I m exploring ways to get more mileage

tawny flicker
#

I have a question regarding buildkit and multiplatform. Is it possible to have a multiplatform buildkit cluster? Like cluster made up of amd64 and mac m1 machines, and build would get distributed on the right arch with a specific build arch metadata?

civic yacht
#

@wet mason (continuing from DM) here's the change I'm imagining:

  1. Change loadExtension to expect a Filesystem with a cloak.yaml and relevant code dirs. Essentially, a project dir as described here though simplified to start (e.g. don't need to handle multiple codedirs in first iteration): https://github.com/dagger/cloak/pull/60/files
  2. Change Extension to have a subresolver called install (sibling to existing name,schema,operations). If install is resolved, it will compile the actual code using the sdk specified and then stitch the schema in. Otherwise if only schema and/or operations are selected, only those will be returned without trying to actually compile anything.
    • For the first iteration, we will cheat on this sdk part by only supporting a dockerfile sdk still (pretty much the current state of things). Just to keep this initial change minimal. Can generalize in follow ups.

This will enable us to solve the DX problem because the current cloak generate tooling can just skip selecting install and only get the schema+operations, so codegen works even if there are compile errors. It also consists of changes we'd need to make anyways to get to the ideas around projects+sdks-as-extensions, so no throwaway work.

I think the biggest lift here is that cloak.yaml parsing will migrate from cmd/cloak to be inside the cloak server. I think this will be mostly rearranging code rather than writing that much new code, so might not be too bad?

Let me know if you think this makes sense too.

wet mason
#

so codegen works even if there are compile errors

Out of curiosity, why is it important that codegen works with compile errors?

civic yacht
# wet mason > so codegen works even if there are compile errors Out of curiosity, why is it...

It leads to insanely frustrating situations where you are in the middle of changing your extension code, then also you want change something about your schema or add/update a dependency, but oh turns out you can't do that because you have a syntax error, left a var in go unused, didn't annotate a type is TS, etc.

This happens all the time when I am updating actions. I typically end up just deleting all my code so I have an empty main func in go (or just export _ = {} in TS), run codegen, undo my changes to go back to what I had before. I personally get frustrated with this all the time, for external users it would be even worse I think.

Also in general, there is no reason that your implementation code should have to be compilable in order for codegen to run; codegen only depends on schemas. The only reason this behavior exists today is just it was simpler to implement in the prototype when we were starting out. So given that and the fact that the fix to this moves us a step in the direction we want to move anyways in terms of where cloak.yaml is parsed and being able to model sdks as extensions, it felt like a good next thing to tackle.

wet mason
#

Yep I get that -- I meant like, why would we loadExtension the extension we're working with, rather than just reading the schema that's sitting right there?

#

I get loadExtension for dependencies (which should compile -- and if not I don't think it's important?)

#

but for your own code that you're actively developing, isn't it strange to (pseudo) loadExtension(".") { schema }?

#

(or perhaps that's tied to the sdk-as-extension? in which case I see why it'd make sense)

civic yacht
# wet mason (or perhaps that's tied to the sdk-as-extension? in which case I see why it'd ma...

Yes this exactly. Agree that with the change I described above alone we'll be in a sort of in-between state where it's odd that you have to load the local extension into cloak and then it just gets passed back to the cloak generate command running on the host (though only odd in terms of our internal code, the user won't be affected yet).

The idea is that this is just the first step though; one of the next steps would be that everything cloak generate is doing migrates to be implemented in an extension (aka the "stubber" part of the sdk-as-extension model). That's just a ton of changes to make all at once, so felt like this was a good compromise as a first step in that we move towards the long-term model and squash that DX bug at the same time, if that makes sense

tepid nova
#

Finished calls for the day, going to catch up on all things cloak FYI 🙂

wet mason
#

(or anything else -- the only point I was making is we probably won't be calling loadExtension anymore with the sdk-extension, I think?)

tepid nova
#

@wet mason @twin crow the return of rcli 😅

wet mason
#

@civic yacht it's kinda working but very very wonky

alpine := core.Image("alpine")
res, err := alpine.
    Exec("apk", "add", "curl").FS().
    Exec("curl", "https://dagger.io").
    Stdout(ctx)

QUERY: query{core{image(ref:"alpine"){exec(input:{args:["apk","add","curl"]}){fs{exec(input:{args:["curl","https://dagger.io"]}){stdout}}}}}}
#

In terms of DX in Go it's a game changer though, if we can get it to work properly:

func alpine(packages ...string) *Filesystem {
    fs := core.Image("alpine")
    for _, pkg := range packages {
        fs = fs.Exec("apk", "add", pkg).FS()
    }
    return fs
}

alpine("curl", "jq", "bash").Exec("ls -l").Stdout(ctx)

QUERY: query{core{image(ref:"alpine"){exec(input:{args:["apk","add","curl"]}){fs{exec(input:{args:["apk","add","jq"]}){fs{exec(input:{args:["apk","add","bash"]}){fs{exec(input:{args:["ls -l"]}){stdout}}}}}}}}}}
#

(Plus, by switching to evaluate:false, all of the above would be a single LLB solve)

civic yacht
# wet mason In terms of DX in Go it's a game changer though, if we can get it to work proper...

Yeah I mean it looks awesome and is way better than the autogenerated clients. I just can't tell yet whether it will be confusing or not that you're building queries rather than sync executing. Like if you stopped at one of the Exec the result won't be that the exec happened. I'm sure that'll trip someone up but the question is more how many it will trip up, can we alleviate that through docs, is that tradeoff worth it.

I'm leaning towards that tradeoff being worth it because it's so much better than the current state (in go at least), but obviously not something we can say for sure until we test it out with users.

wet mason
#

Yeah, 100% agree

civic yacht
#

And also, like we were saying before, I think the choice we're making here is probably about our "default go sdk". Once sdks are extensions there's no reason someone with different opinions can't make their own

wet mason
#

To help somewhat navigate this, takes a ctx & returns an error = sync. Otherwise, async

#

So you need to learn the "trick" once

civic yacht
#

Right exactly, that helps a lot. I'm sure there's more we could do on that front too, maybe with the names of the methods?

wet mason
#

But yeah, it is less obvious

civic yacht
# wet mason (Plus, by switching to evaluate:false, all of the above would be a single LLB so...

Agree this is a good thing and applies to core, but will add that it breaks down a little bit for non-core actions. Like if you from an extension outside of yarn something like: core.Git(..).Yarn("build").Exec(...).Exec(...).Stdout(ctx), then even though the query itself is built lazily and submitted only once you reach stdout, during execution once the remoteSchema resolver for yarn is reached, you have to synchronously evaluate it because you have to call ReadFile to get the output of the execop for the yarn extension

#

That's not a big deal, we should take what performance optimizations we can get but be okay with areas where it's slightly less optimal, but just worth keeping in mind since I almost forgot about it

wet mason
#

Right now I took the approach of "scalars are sync". Kinda make sense when you look at graphql, you CAN'T select a type, you need to select a scalar field. So basically Exec etc is async, Stdout() Stderr() those are sync

It's a bit tedious though because scalars are the most interesting bit so you'd have to error check everyone of them

The alternative would be to make "types are sync" and auto-select all scalar sub-fields

The annoying part though is that maybe you don't want to select a sub-field

wet mason
#

like the 2 execs post-yarn are a single solve

#

but maybe that's EVEN more magic

civic yacht
wet mason
#

Yeah

civic yacht
#

On the other hand, err is always annoying in go, so at least we are just in line with the language

tepid nova
#

How could we take advantage of the many existing graphql APIs for cloud services, to speed up development of corresponding cloak extensions? Coukd we wrap & extend the upstream gql schemas?

Netlify, Vercel, Github, etc

#

This becomes easier if we allow more flexibility in the structure of extension’s gql schema (no constraint to match precise action layout)

tepid nova
#

FYI Netlify Graph (formerly OneGraph) has a next.js stubber in case if it's of interest:

unknown.png
copper snow
# tepid nova How could we take advantage of the many existing graphql APIs for cloud services...

One potential solution for integrating generated SDKs is generating code that wraps calls to those SDKs.

For example I worked on generating an AWS SDK wrapper library for Temporal: https://github.com/temporalio/temporal-aws-sdk-go

You could use the same mechanism to generate the schema and any necessary code that calls into the SDK.

Using existing APIs directly could work, although you would still have to implement authentication (usually already provided by SDKs).

GitHub
GitHub - temporalio/temporal-aws-sdk-go: Temporal activities and wo...

Temporal activities and workflow stubs that wrap AWS Go SDK - GitHub - temporalio/temporal-aws-sdk-go: Temporal activities and workflow stubs that wrap AWS Go SDK

tepid nova
#

Thanks for the reading material @copper snow

copper snow
#

I wonder how "project" and "context directory" would map to common types of software projects (eg. App with single deliverable, monorepo with multiple deliverables, etc)

cosmic cove
tepid nova
#

@civic yacht @twin crow when you have a minute, can you write down the feedback you gave me in the PR comments? (question about how much graphql we should show users; and question about maybe generalizing further with an "entrypoint sdk") Thanks!

tepid nova
cosmic cove
#

<@&1003717314862129174> Hi everyone! Just a reminder that we are having the first Cloak Early Access Community call this week on Thursday at 10 am PST. We look forward to seeing all of you!

tepid nova
#

@twin crow ok I think I got it 🙂 will update my PR tonight

#

(we did a dogfooding session @civic yacht and it helped unblock a few things)

civic yacht
tepid nova
tepid nova
#

@cosmic cove let me know if you find this more clear than the previous version

cosmic cove
tepid nova
#

FYI <@&1003717314862129174> we have a instance of the sandbox running on a dev machine, you can access it here: https://playwithcloak.infralabs.io

Please don't share, it's not setup for production use. It may break at any time as we push breaking changes (like it did earlier today 🙂

#

But it does work, and it's quite addictive!

#

Trying the "copy curl" feature:

curl 'https://playwithcloak.infralabs.io/' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' -H 'DNT: 1' -H 'Origin: https://playwithcloak.infralabs.io' --data-binary '{"query":"# Write your query or mutation here\nquery {\n  core {\n\t\tgit(remote:\"https://github.com/dagger/dagger\") {\n      file(path:\"README.md\")\n    }\n  }\n}"}' --compressed
tepid nova
cosmic cove
tepid nova
tepid nova
#

@civic yacht @twin crow @cosmic cove here are the raw files from our DX session:

tepid nova
tepid nova
# tepid nova cloak.yaml / dagger.yaml: ```yaml workflows: develop: description: Run a ...

develop.sh:

#!/usr/bin/env/bash

## 1. What is this script's runtime environment?
## 2. How do I install custom tools?
##      a. no customization
##      b. sdk-specific
##      c. sdk-agnostic
## 3. How do I access operator's host system? env variables, files etc
##      a. run workflow in "introspection mode", returns list of needed privileges
##      b. list needed privileges in project file
## 4. How do I make Dagger API queries?
## 5. How do I load local directories into the API?
##      a. from a workflow: `host { readdir }`, authorized by workflow privileges
##      b. from a custom client running on the host (embedding use case): run "anonymous workflow" as a callback in your code.
##          needed privileges must be declared, but in a struct rather than yaml 
## 6. How do I pass data between API queries?
##      a. avoid: prefer chaining
##      b. query scalars -> bash variable -> query arguments
## 7. How do long-running services work?
## 8. Can I use client stubs for extensions, and if so, where do they come from?

# Option 1: raw query + chaining
dagger query <<'EOF'
{
    host {
        workdir {
            yarn {
                install {
                    yarn {
                        run(script: "start")
                    }
                }
            }
        }
    }
}
EOF

# Option 2: client stubs + fancy stitching
dagger do -p host workdir |
dagger do -p yarn install --src "$workdir" |
dagger do -p yarn run --map id src
#

@civic yacht I’m wondering why it’s not a widely adopted practice to just walk the whole query tree and generate stubs for everything down to scalar fields

#

I think I understand why: it defeats the purpose of carefully querying what you need and nothing more

#

but our situation is different since the size of data to query is not an issue. We want to traverse the graph to chain operations

#

So why not expose the whole graph in client bindings?

civic yacht
tepid nova
#

You can even generate the full struct type for each leaf, so what if it’s “inefficient” in our case it’s a rounding error

tepid nova
#

just get all the fields, who cares

civic yacht
# tepid nova just get all the fields, who cares

This we actually care about, getting a field isn't necessarily free and the cost depends on the implementation backing the query. For instance if you return a type like:

{
  cheap String
  expensive String
}

There can be resolvers attached to those even though they are both scalars. And if you always select all of them you can't choose to skip expensive. Also, because resolvers can have side-effects, it may often make sense to not select them

tepid nova
#

Right but maybe that tradeoff is fine in the case of dagger

#

because strings won’t actually be expensive

#

and if they are you can drop to raw gql?

#

or make it a convention that expensive fields shoukd take at least one argument?

#

we may have wiggle room there if you’re looking for ways to make the implementation work

#

it doesn’t have to work in 100% of queries, just 80%

#

Separately from that: this path would remove the need for hand crafted operations in the extension correct?

#

in bash sdk it could map to a tree of sub commands with subcmd-specific flags

civic yacht
# tepid nova or make it a convention that expensive fields shoukd take at least one argument?

That's possible but it would be a departure from "standard graphql" (acceptable but nice to avoid when possible). I guess I don't know what we get from this.

Separately from that: this path would remove the need for hand crafted operations in the extension correct?
Are you imagining that we would use this approach to automatically generate query operations? If so, then yes this is one path to removing a need for handwriting query operations, but I don't think it's the only path. I think what Andrea was experimenting with would create the same end effect but in a way that doesn't require us to impose special rules about which fields should have expensive operations or not.

civic yacht
#

My understanding of your suggestion is that we could autogenerate a query operation for a given schema by walking the schema and selecting every scalar field we find. We'd have to skip fields that take args and deal with the chaining case where you could walk in an infinite loop, but probably possible. Is that correct?

I think the general idea of the approach Andrea was trying was that we can generate client stubs that operate essentially as "query builders", so then you can construct the operations from your actual code, including fields that take args, skipping expensive ones or ones with side-effects that you don't want, etc. It's sort of like a client-side+code-first approach of defining query operations (whereas today it has to be defined extension-side+schema-first in operations.graphql).

tepid nova
#

Yes that’s right, sorry I was typing while walking in the street.

To recap:

  1. It would be desirable for SDKs to generate client bindings directly from a dependency’s schema, without requiring that the dependency also provide hand-crafted operations

  2. Andrea is exploring a way to achieve that in the Go SDK. There are still some quirks to work out, but we can assume it’s possible.

  3. The same thing seems possible in bash sdk: instead of generating tree of Go types, we generate a tree of CLI subcommands to be invoked by the script. Again, no need for hand crafted operations.

  4. If it helps iron out the quirks, we have the possibility (I think) to use heuristics that may not work for all graphql APIs, but work for us because of specificities of the cloak API (querying all scalars is cheaper because of smaller datasets, super-short network path, no https overhead; expensive scalars could have a convention to make them easier to spot). But maybe we don’t need this or it’s not actually practical.

  5. Therefore I will assume in docs/next that extension-supplied operations don’t exist (ie no operations.graphql). If someone thinks that’s not desirable or practical. please bring it up in docs/next !

civic yacht
# tepid nova Yes that’s right, sorry I was typing while walking in the street. To recap: 1....

Yes agree on all points, only slight divergence is that the heuristics of 4. that I was imagining we could use are a bit simpler (and thus handle slightly less cases), but that's not meaningful; I agree on the main point that heuristics are one possible route and that it could turn out we don't need and/or can't have them.

Also I'd add 6., which is that I think it's logically provable that for every possible hand-crafted query operation, it's possible to instead create an equivalent field in the schema that takes the same args and internally is implemented to perform the same query as the operation. When combined with relatively straightforward heuristics, this may be another route by which we can fully eliminate the need for ever requiring handwritten operations.graphql. E.g. an operation like this:

query Foo(a: String) {
  bar {
    baz(a) {
      someString
    }
  }
}

could always be represented as a field in the schema like:

extend type Query {
  foo(a: String) String
}

and then implemented as (using pseudocode):

func(a string) string {
  return bar.Baz(a).SomeString
}

Not saying this is a better approach, just an additional option on the table.

  1. Therefore I will assume in docs/next that extension-supplied operations don’t exist (ie no operations.graphql). If someone thinks that’s not desirable or practical. please bring it up in docs/next !
    Yeah, let's operate on this goal (hence next) until proven otherwise. It may take a while to achieve the goal depending on how we prioritize it (which is to say, I probably can't delete all the operations.graphql files in the next week), but it's the right goal to set for now I think.
tepid nova
twin crow
#

I played with the project file definition later today so I am sharing some thoughts that are connected to previous discussion. It's not super detailed, but I can write down more details for each part if there is an interest, otherwise just ignore! https://github.com/dagger/cloak/pull/79

cosmic cove
cosmic cove
#

Welcome @astral zealot @blazing prism @vital plaza ! As mentioned earlier this week, we have our community call tomorrow 👆

tepid nova
#

@civic yacht I'm getting test errors in my core/git PR, which makes sense because I'm changing the API. Do you know where are the tests are, so I can change them?

#

It looks like it's running stuff from examples, but I don't know which parts. All of them?

#

I'll start by changing all examples, and see if that fixes it 🙂

civic yacht
civic yacht
tepid nova
#

That's fine, the whole point of this PR is for me to get practice with the codebase

#

So that I can be more helpful (and also make better DX suggestions informed by knowing how things actually work)

#

@civic yacht to run those tests do you just cd core/integration && go test or is there a makefile somewhere?

civic yacht
tepid nova
#

@civic yacht more of a taste question. Which form do you prefer:

Option 1.

{
  git {
    remote(url: "https://github.com/dagger/dagger") {
      fetch(ref: "main") {
        file(path: "README.md")
      }
    }
  }
}

Option 2:

{
  git {
    remote(url: "https://github.com/dagger/dagger") {
      ref(name: "main") {
        fetch {
          file(path: "README.md")
        }
      }
    }
  }
}
#

Option 1 is what I have in the PR, and I like how it sets us up to query more things about a remote, like perhaps a list of its available branches, etc. But if that's true then might as well go all the way and apply the same pattern to refs as well? Hence option 2.

#

Pushing might look like this:

query push(source: Filesystem!, remote: String!, ref: String!) {
  git {
    remote(url: $remote) {
      ref(name: $ref) {
        push(source: $source)
      }
    }
  }
}
civic yacht
tepid nova
#

Exactly, the whole point is maximum flexibility, might as well lean into it

tepid nova
#

@civic yacht how do you feel about this:

    extend type Query {
        "Built-in containers capabilities"
        containers: Containers!
    }

    "Built-in containers (OCI) capabilities"
    type Containers {
        "Reference a remote container repository"
        repository(address: String!): ContainerRepository!
    }

    "A remote OCI container repository"
    type ContainerRepository {
        "Lookup a tag in the repository"
        tag(name: String!): ContainerTag
    }

    "A tag in a remote OCI container repository"
    type ContainerTag {
        "Current checksum value for this tag"
        checksum: String!

        "Pull the image at the given tag"
        pull: ContainerImage!
    }

    "An OCI-compatible container image"
    type ContainerImage {
        "Root filesystem of the image"
        fs: Filesystem!
    
        "Raw image configuration (json-encoded)"
        rawConfig: String!
    }
#

Example query:

containers {
  repository(address: "index.docker.io/alpine") {
    tag(name: "latest") {
      pull {
        fs {
          file(path: "/etc/motd")
        }
      }
    }
#

It brings the question of core.#Exec vs docker.#Run and that whole chunk of problems. I'm not thrilled with the current situation in Europa, but not sure what's the best way to improve on it

#

Maybe there should be a Container type separate from Filesystem, and each could be extended separately

#

Exec and the chaining mechanism could be moved to a Container type. We could add the various dockerfile-like metadata operations, so you could do

pull {
  exec(...) {
     exec(...) {
       set(user: "foo", workdir: "/root", env: { DEBUG: "1" }) {
            exec(...) {
              fs
            }
       }
     }
   }
}
#

fs { container { fs { container } } } 🙂

#

maybe more accurate: fs { newContainer { fs { newContainer } } }

civic yacht
#

I think I agree that there needs to be a separate type that is a Filesystem+Config combination; ContainerImage is probably the right name for that type.

Exec and the chaining mechanism could be moved to a Container type. We could add the various dockerfile-like metadata operations, so you could do
Yeah I was just wondering about that too. I'm not sure if by "move" you meant that Filesystem no longer had exec, or if they both would have exec now.

I think it would be fine for both of them to have their own exec fields. ContainerImage's exec would be almost the same thing as Filesystem's (and would use it underneath the hood), the only differences being that the config defaults are applied and it returns a ContainerImage type

tepid nova
#

Yeah I was thinking remove it from Filesystem since you can simply do newContainer { exec } instead

#

but that's a small cosmetic detail

#

Ok I'll attack this as my next PR, more plumbing to move around, more practice 🙂

civic yacht
# tepid nova Yeah I was thinking remove it from `Filesystem` since you can simply do `newCont...

True, yeah it'll be easier to tell what feels better after trying it in practice.

In general I also had some hesitation with the extremely generic containers namespace, which conceptually could encapsulate an enormous number of types+apis, but in retrospect maybe that's not even a bad thing. User extensions don't have to use it (e.g. a k8s extension can just make its own top level k8s namespace) but maybe it would be interesting and more powerful if they could extend under it (e.g. a k8s extension could extend ContainerImage with functionality to deploy an image as a pod or something). The k8s example might not be the best, but in general I guess it's fine to have highly generic top level namespaces since it's possible to extend functionality within them

tepid nova
#

Yeah, my thinking so far is: top-level namespace is a good fit for mounting an external, global namespace

#

So git is our gateway to all git remotes everywhere (by url)

#

containers is for all container registries everywhere

#

etc

#

normally, every extension should be possible with either 1) a new top-level namespace for globally addressable things, and 2) extending a core type for everything else

#

Off the top of my head, for kubernetes:

top-level: interact with a cluster

query {
  kubernetes {
    context(config:Filesystem!) {
      cluster(name: String!) {
        // direct query to cluster goes here
        apply(...)
      }
    }
  }
}

Extend filesystem for loading yaml files etc

extend type Filesystem {
  kubernetes {
    validate: ...
    kustomize: ...
    // not sure what would be useful here
  }
}
tepid nova
#

@civic yacht the p.Source trick you showed me only works for one level up, right? If I want to access the grandparent or further up, I need to roll up the fields so that they are all available in p.Source correct?

#

Context: containers { repository { tag { pull } } }

I need to access fields from both tag and repository to execute pull

civic yacht
tepid nova
#

Am I right that it's OK to implement resolvers for the whole hierarchy of types in a single schema implementation, as long as there's only one path to the same type? Then as soon as there's more than one way to get the same type, that's the moment to break out that type into its own schema implementation

#

For example in my PR, I can bundle the resolvers for Containers, ContainerRepository, ContainerTag in the same go struct, because they are tightly coupled. But I have to break out Container into its own separate Go type, because I could get that type from the top-level or from Filesystem { newContainer } etc.

#

Is that right?

civic yacht
# tepid nova Is that right?

I think so? I guess my brain is collapsing when I try to imagine this right now, will be easier to say when looking at the code. But that sounds generally right, that's what we did with Filesystem for example

tepid nova
#

In the context of the current code: this is why Exec is implemented in a separate file, rather than the same file as core { pull }

#

Or Filesystem for that matter

#

Basically I'm trying to understand the rule for when to break out my GraphQL types in different go files

civic yacht
#

I think it's more just whatever organization feels best; we just wanted to avoid the previous iteration where every type + resolver was in the same file and it became a huge blobby mess. There isn't any hard rule where you have to split out to different go files or even different go types. The pattern is just to split things up where it makes sense (i.e. Exec is a lot of code that is specific to exec, so split that out to its own file and structs). The fact that all the schemas get merged together via extend makes it easy+nice to do this, but it's not strictly required technically speaking, if that makes sense.

tepid nova
#

Roger that. Thanks!

tepid nova
#

Based on @twin crow ‘s feedback I think we can simplify the workflow implementation for the v2 demo. Will write it down tonight or tomorrow.

#

cc @civic yacht @cosmic cove 👆 it should only remove implementation work compared to the flow we wrote today

twin crow
civic yacht
# tepid nova cc <@949034677610643507> <@954113233889951754> 👆 it should only remove implemen...

Awesome, I've got my workflow implementation POC at the point where dockerfile boilerplate is gone (just have to set an sdk value) and that DaggerServer boilerplate present in ts extensions is gone from the workflow file: https://github.com/sipsma/cloak/blob/workflow/examples/todoapp/app/cloak/index.ts

Was just gonna go clean up the local-dir+secret handling, at which point it should start to resemble what we were envisioning I think. Still a bunch of little details that are off, some P1s from discussion earlier, etc.

Will update my code based on the new v2 (v2.1?) you post later

tepid nova
#

TLDR is that there are still competing ideas for making the first workflow as easy to write as possible. I don’t yet know which one we should choose. Meanwhile we can show a great demo without choosing, by showing only the fundamentals. This will be less work to ship and will help us figure out which convenience is actually best in practice, by experiencing the pain of its absence

tepid nova
#

possible convenience features:

  1. Workflows are client code written by you, and run in containers by the Dagger CLI. Dagger SDK also helps you with client stubs. This is currently documented in docs/next.

  2. Workflows are just regular clients that you run yourself. Dagger SDK helps you with client stubs.

  3. Workflows are queries configured in the project file, and made directly by the Dagger CLI. Everything that doesn’t fit in a query should be implemented as an extension. This is @twin crow ‘s proposal as I understand it

  4. Workflows are special kinds of extensions. As proposed by @civic yacht

These features could be potentially mixed and matched

tepid nova
civic yacht
# tepid nova Behold! https://github.com/dagger/cloak/blob/0d78700dc5a97e74ef8163efc7ba1be7fd1...

I gotta run to cook dinner and stuff, but saw the mermaid diagram briefly, looks great! Will look more later or tomorrow

Just pushed support for local-dir+secret inputs to the workflow poc. It's slightly different than what we discussed earlier, can discuss reasons why later (also adjustable of course): https://github.com/sipsma/cloak/blob/workflow/examples/todoapp/app/cloak/index.ts

But now you can run cloak -p examples/todoapp/app/cloak.yaml do deploy from the root of the repo in that branch and that workflow file executes, seems to work so far

tepid nova
#

The more detail I try to add to the architecture diagram, the more questions I have 😅

#

For example, cloak.yaml is both a development config (how to generate code for, and possibly build my workflow); but also a runtime config (which extensions to load when running my workflow). Not sure if it's a bad thing but made me raise my eyebrows

tepid nova
#

Maybe it could be purely a development config? This would be possible if we moved even more work to the SDK, at the code generation phase. For example, instead of relying on the cloak CLI to bootstrap the engine and load extensions; maybe the SDK could generate native code that does it? So after running cloak generate, the workflow would be 100% self-contained, native code. No runtime parsing of cloak.yaml required.

#

I'm also wondering if perhaps having a cloak.yaml per workflow, rather than per project, would be simpler

#

All things which we can discuss on tomorrow's community call 🙂

cosmic cove
round granite
#

is anyone around who can help me troubleshoot real quick?

#

I am trying to set up a new example action from the docs in the readme and encountering some errors

civic yacht
round granite
#

no prob

cosmic cove
#

<@&1003717314862129174> community call starting now. See you there!

tawny flicker
#

Have we explored the idea of using plain Makefiles as the way to interact with dagger/cloak?
Of course, there are some limitations (no clear schema for each target, so we don't know easily/by default what are the input, what are the outputs)?

#

If we could run makefiles cloak make, but each target being an execution in a container in buildkit, I think it would help people jump into cloak already.

#

But, it would also bring a lot of bugs for Makefile that wouldn't work in our use case by default (I guess).

#

Each Makefile target would be the image of the target at the end of the run (under the hood), and wouldn't run again if it's cached

cosmic cove
#

Thanks to everyone who joined the Community Call today. Lots of exciting stuff! It was great to see @round granite 's experience as a new user in real-time and to dig into @copper snow's specific use case and feature requirements more.

civic yacht
# tawny flicker Each Makefile target would be the image of the target at the end of the run (und...

Yeah I think there's a lot of interesting possibilities in terms of integration with existing tools like Make, some previous discussion about the various options here: https://github.com/dagger/cloak/discussions/17#discussioncomment-3327549

There are some tricky parts of this like you said. Make essentially uses your mutable local filesystem as its cache, so the most naive way of translating that to buildkit execops (just run each makefile target invocation as an execop on top of the base layer) wouldn't quite work as the filesystem changes would not be "additive". Could address that using cache mounts (hacky but simple) or probably via some fancier merge+diff logic (robust but harder), but I think that would be one of the main challenges. Either way, this would be a problem that is solved in the "Makefile SDK", using current terminology.

You're right though that if we solve those problems it would be awesome to just point dagger at a Makefile, dagger autodetects that it should use the "Make runtime" and then executes targets with all the nice caching and other benefits dagger provides (especially in a hypothetical future where dagger provides magical distributed caching)

tepid nova
#

@tawny flicker probably this can be done very naturally via the bash sdk

#

Run make, which runs a shell script which happens to be a cloak workflow

tawny flicker
tepid nova
#

Currently it looks like this:

.PHONY: web
web: web_redirects # Run the website locally
    yarn --cwd "./website" install
    yarn --cwd "./website" start
#

Instead we might change it to look like this:

.PHONY: web
web:
    ./website/dev.sh
#

And that shell script might look like:

#!/usr/bin/env bash

src=$(dagger query 'host { localdir { id } }')
install=$(dagger query <<-'EOF'
query install($src: Filesystem!) {
    yarn {
      install(source: $src) {
        ...
      }
    }
}
EOF

# etc.
#

That dev.sh script is a dagger workflow

#

Developed using the bash SDK

#

But it's invoked via the usual Makefile

#

(the contents of the shell script is an approximation, since we don't have a bash SDK yet, but it's close enough)

#

This also makes the assumption that SDKs can generate code smart enough to make the workflow runnable standalone (here ./website/dev.sh).

If that's not the case, then running a workflow might not be runnable directly, and might instead require loading into the CLI, eg. dagger do ./website/dev.sh or dagger do -p ./website dev

#

That's the topic I mentioned I'd like to discuss @civic yacht : how smart can we, and should we, make code generation in the SDKs 🙂

#

does that make sense @tawny flicker ?

tawny flicker
#

It does make sense, and it is #1 in my list.
Though for me, it's less interesting than #2 which would automagically run the Makefile without work on the user's part.
But clearly, that #2 is way more complicated to implement (as Erik confirmed)

tepid nova
#

@tawny flicker a make API extension would be pretty cool too 🙂

cosmic cove
#

Welcome @tight socket !

tepid nova
#

Looking forward to seeing what you build @tight socket

tepid nova
#

FYI @civic yacht I just refactored the structure of the docs, to unify it all and make it easier to navigate as a whole. I preserved everything you wrote but moved it around to fit the file hierarchy. Can you take a look and let me know if it makes sense to you?

https://github.com/dagger/cloak/pull/88

#

Oh you already commented 😉

civic yacht
tepid nova
#

OK, note that I moved most of your concepts.md + some of the README into a "writing extensions" doc

#

Some of it may be redundant or internally inconsistent, but at least now that will be easier to spot and fix

#

I also gave up on next, I think it's just easier if we make it all one set of docs, and iterate

#

I can walk the line between "real" and "aspirational", I'm used to that 😉

civic yacht
#

We just have to be barely consistent enough to get going

tepid nova
#

I'm taking over the swag department

civic yacht
#

Truly inspirational

#

almost a whole life philosophy

ancient kettle
#

We have an eventually consistent company.

tepid nova
#

I'm writing the demo v2 workflow in typescript and man, I have many questions 😅

tepid nova
#

2nd question: in the yarn package, why query the mount ID for /src in and reuse it across yarn install and yarn build ?

#

Since it's a regular read-only mount, and we have the ID of the source already

#

Is it to force an actual solve? Like you need to get something out of the yarn install, or buildkit will skip resolving it?

civic yacht
# tepid nova <@949034677610643507> I'm looking at the workflow POC in your branch, trying to ...

It is set because the cloak.yaml specifies workdir: true, which results in the FSID for that localdir to be passed as input to the workflow. That's the high-level, let me know if you were looking for more of the low-level details.

2nd question: in the yarn package, why query the mount ID for /src in and reuse it across yarn install and yarn build ?
This is because we don't want the original FSID we mounted in, we want the FSID of that mount plus the changes that were layered on top of it as part of the execop. The yarn install we run makes changes to that /src mount, so we want those changes to be present when we mount /src in the next yarn run ... too. Basically, it's just the equivalent of the .GetMount method in the go LLB library if you are familiar with that.

tepid nova
#

But aren't those changes discarded by buildkit?

#

I guess I was not familiar 😉

#

That's impossible to do in a dockerfile, correct?

#

(and in Dagger 0.2)

civic yacht
# tepid nova But aren't those changes discarded by buildkit?

No actually, you can get the output of a mount after an execop that includes the changes, it's a super helpful feature.

That's impossible to do in a dockerfile, correct?
(and in Dagger 0.2)
Yes and yes (wanted to get that feature in early here so it wasn't lost)

tepid nova
#

Well that's a 🤯 moment for me

civic yacht
#

Haha yeah buildkit is full of all this magic that nobody knows about since it's not really documented

tepid nova
#

It's a very common stumbling point for Dagger users today btw ("why are my changes lost?")

civic yacht
tepid nova
#

OK awesome. As for question 1 (workdir): I have to write the host-native workflow version of that. I'm thinking of a query like { host { workdir { id } } } wdyt?

#

I'm thinking we cordon off all privileged host access in there, and then we can figure out how to add access-control and perhaps mocking / dependency injection 🙂

#

For example we could support end users overriding the workdir with export CLOAK_WORKDIR=/another/path and that would carry over to the embedded engine. Similar pattern to LD_PRELOAD

#

Also how a web sandbox would work. Could plug in a git remote instead of the actual workdir of the web server process

# 
// Load app source code from working directory
const source = await client.request(gql`
  {
    host {
      workdir {
        id
      }
    }
  }
`)
.then((result: any) => result.host.workdir.id)
#

It buys us some time before we have to deal with more host access, like arbitrary paths on the host etc

#

All I need for the demo is 1) workdir and 2) env variable. If you agree with the above, we have a way to do both

#

(I also thought of a possible way to make the engine fully ephemeral much sooner than initially thought. But will leave that for later 🙂

civic yacht
#

Yes that's easily doable. I didn't add the host namespace yet, it's just under core, but that's obviously trivial to change.

tepid nova
#

What is it called today?

#

Oh shit, wait there is one more host I/O feature we're forgetting

#

write access to the workdir...

#

Which my standalone build workflow will need, or it's useless

civic yacht
# tepid nova *write* access to the workdir...

Right now we just have {core{clientdir(id: "foobar"){id}}}, where foobar is the human-readable name of the localdir you pass as input configuration to the engine. workdir would essentially just be underneath the hood a const id provided to clientdir (can expand if this makes no sense, either way what you described is doable easily)

There was support for local exports a few weeks ago, I think it got lost in all the refactoring, but it will be straightforward to re-add

tepid nova
#

Re: how workdir works under the hood: yes makes sense. Basically the same way docker build context is just a const id too 🙂

#

In fact if we still had a Dockerfile-like model where you drop a file at the root of your project, then I would probably propose calling ours context too... But in the current DX, with each workflow being a full-blown software project in its own directory, it would make less sense. Better to call it workdir and make it easy to customize (ie it can default to actual workdir of the client process, but can be changed to any local directory)

tawny flicker
tepid nova
#

Next naive question: what would be the native way to run my typescript workflow?

#

I guess possibly wrap it in my client's yarn? 🙂 yarn run build -> runs my typescript build workflow which invokes yarn in a container?

#

Or run it with node?

civic yacht
# tepid nova I guess possibly wrap it in my client's yarn? 🙂 `yarn run build` -> runs my typ...

This is where my lack of JS/TS knowledge will start to show. I think you can do yarn run build and then you'll end up with a directory called dist where you'll have js files that you can execute with node dist/index.js or whatever. However, I haven't tried that from the host directly before, that's just what we do in the current Dockerfile (or in the TS sdk that replaces Dockerfile in my workflow branch)

civic yacht
tepid nova
#

zero change to dev workflow, would be pretty cool

#

except yarn install would be redundant

#

or I guess not for local dev

#

I really love this for drop-in adoption of dagger

#

no disruption to your dev workflow

#

that’s the real benefit of embedding

civic yacht
tepid nova
#

yes

#

pretty killer IMO

civic yacht
#

Cool yeah, makes perfect sense

tepid nova
#

makes more sense for a yarn script that is a little customized already

#

If yarn has a middleware system, there could eventually be a middleware that just transparently does this for all your scripts

#

Depends on the tool’s abilities. Whatever they are, we can embed cloak in them

civic yacht
#

Right, it's much easier to adopt dagger in baby steps rather than just port everything over immediately. I personally already run everything possible in containers, so I was coming at this from a highly biased perspective before, but it makes perfect sense that most normal humans probably aren't starting there

tepid nova
#

same for make etc

civic yacht
tepid nova
#

@twin crow @civic yacht could dagger bootstrap its buildkit on a mac with colima to remove the docker4mac dependency?

#

zero runtime dependency in all hosts would be amazing

#

(and by “bootstrap buildkit” I mean bootstrap its own daemon embedding buildkit)

#

the difference would be that our daemon would bundle both a buildkit daemon and client, to avoid having to reimplement localdir streaming etc in the cloak api

#

in that case clustering would require nodes to talk to each other with a lower level API (not the regular graphql api) so that they can stream their local dirs to each other etc

#

that clustering api could just be the raw buildkit grpc

civic yacht
#

I don't know very much about the implementation details of lima/colima yet, but in theory sure I can't imagine why. We just need a way to run daggerd in a linux VM. Docker desktop is at the end of the day just a way to run containers in a linux VM; as far as I know colima is just another way of accomplishing that with a different stack.

the difference would be that our daemon would bundle both a buildkit daemon and client, to avoid having to reimplement localdir streaming etc in the cloak api
I guess you'd need your localdirs from your mac mounted into the VM then, but yeah that's supported afaik

twin crow
#

instead of shipping buildkitd, he ships the whole stack (vm + buildkitd)

#

and I believe Olivier's will be on that channel very soon 🙂

tepid nova
#

ha ha perfect 🙂

cosmic cove
#

yeah, he mentioned a work account, so just wanted to make sure that I added the right one.

#

Welcome @untold shoal ! We were talking about you above 😉

tepid nova
untold shoal
#

Hey Cloak peeps
re: colima

Yeah, I’m building my own stuff on top of lima: buildkitd & containerd running directly inside a Debian VM.
It is trivial to do.
In my use case, I do not need Docker at all (neither desktop, cli, nor daemon) (nerdctl and the buildkitd socket is enough for what I need)

A VM definition in lima is literally 10 lines of yaml, and the only trick is to mount and expose the (buildkitd) socket on the host (gotcha with permissions on the sockets that are wrong by default (TM), but that is a detail).
Then having that VM run as a macOS service is again trivial (just barf a launchctl plist).

I am also wrapping that up with ghosttunnel and step-ca for mTLS and cert management.

Lmk if you are going with any of the above ^, happy to chip in / share / align.

#

here is my Lima def - it is very dumb (and also is still problematic with perms on the socket if systemd bounces off buildkitd, but hey, that should get you started):

# 
images:
  - location: "https://cloud.debian.org/images/cloud/bullseye/20220711-1073/debian-11-generic-amd64-20220711-1073.qcow2"
    arch: "x86_64"
    digest: "sha512:6f6896d5740efb095d30e5118aeced30eda596dd5cddc28929767dc568f1833d84621a7b71d0aa403fcae3aef9c566ce7403b5a59dcaa74f6d3ea0e171ec38b0"
  - location: "https://cloud.debian.org/images/cloud/bullseye/20220711-1073/debian-11-generic-arm64-20220711-1073.qcow2"
    arch: "aarch64"
    digest: "sha512:ce07048a3760e22d96244f399fa242930eafb5b7a8fa0d0014e6616e86d0c38e544b2206154c6cc6eb1b3924b74d337725791b2305f59527c87a2fbd8a8e6bf1"

containerd:
  system: true
  user: false

portForwards:
  - guestSocket: "/run/buildkit/buildkitd.sock"
    hostSocket: "{{.Dir}}/sock/buildkitd.sock"

provision:
 - mode: system
   script: |
     #!/bin/bash
     set -eux -o pipefail
     export DEBIAN_FRONTEND=noninteractive
     # Install git
     apt-get update
     apt-get install -y git
     # Bounce it
     systemctl restart buildkit
     # Enable multi architecture support
     # XXX parameterize and own this
     nerdctl run --privileged --rm "tonistiigi/binfmt" --install all

 - mode: user
   script: |
     #!/bin/bash
     set -eux -o pipefail
     # Fix perms
     sudo chown -R root:$(whoami) /run/buildkit

# XXX parameterize this and allow people to control it? or stick with bk limits
cpus: 4
memory: 16GiB
disk: 100GiB
#

(you should be able to run bk rootless as well ^ - it just does not cut it for me but it is equally easy to do)

#

(and to close this topic: I have no experience with colima and not sure what the value add is - since it is just using lima under the hood... cut the middleman...)

untold shoal
cosmic cove
untold shoal
#

After first cloak demo this morning, Sam suggested that I share any feedback here broadly, so, here it is, in the hope that this is helpful:

———————————————————————————
a. TL;DR I like cloak - I can see myself using it as a base block for my own product in the future - pending my own schedule / uncertainties, I will likely start poking/evaluating in late September.

———————————————————————————
b. we have clearly been working on the same stuff independently, with very different technical angles
Also, for myself, this is not the end game - it is a base building block / requirement for my product.
My focus is on observability, automated testing acceleration leveraging ultra-cool-time-travel-debuggers.
In that context, I would gladly just trash all my homegrown stuff and replace it with cloak, if cloak provides what I need.

———————————————————————————
c. For context - as I shared during the demo - here is what I have built so far:

  • a base “runner”, installable on mac with no runtime dependencies whatsoever (lima+buildkitd+containerd+step+ghost)
  • on the client-side, a javascript (web and node) component usable by developers that exposes a SDK to write plans consisting of actions (actions themselves can be written in JS, seamlessly, inside the plan)
  • the javascript implementation is able to load and reuse plans and actions written in other languages (eg: go)
  • pretty sure it is obvious in this part of the world but yeah: plan <=> pb.Definition and action <=> pb.Op
#

In term of technical implementation, I basically just rewrote most of buildkit/client/llb and buildkit/solver/pb in typescript, allowing me to manipulate directly protobuf LLB messages and marshal them into my SDK, and conversely serialize plans & actions into protobuf - to be fed to buildkitd.
Porting LLB has been a mixed experience:
On the positive side:

  • it is definitely doable and did not take too long (about 10 days total since Andrea tried to discourage me 😄 - mostly because I was not too familiar with client/llb yet - next time I have to port it, probably to C# or kotlin, it will likely be much faster)
  • I am pretty sure it could be written once and transpiled to a bunch of other languages - that part of buildkit really does nothing but manipulate a graph and serialize / deserialize protobuf (with some metadata fiddling with caps and attrs for good measure)
    On the negative side:
  • I do not quite like LLB higher level abstractions for that client side - I think a more straightforward SDK could be built on top of the base protobuf definitions without all the dancing around

The way I see cloak, you guys did bet on graphql as your base API layer and exchange format - while I did bet on protobuf for exactly the same purpose (also: transport does not matter).
I am not sure what you guys are thinking about this - but I would personally prefer this (graphql | protobuf) to be an implementation detail and not the API that most users are consuming.

The upside of my approach as far as I am concerned is that I have the ability to code any new feature very fast and purely in typescript.
If buildkitd has the primitives, I can just add it to my SDK without having to go upstream, since I am basically just talking “LLB over protobuf” on the wire.

#

———————————————————————————
This is actually the #1 most important thing for me if I were to adopt cloak.
eg: the ideal scenario for me would be that your graphql API is a 1-1 passthrough / mapping of LLB, and that would be done once, now.
I do not need (or want) higher-level SDKs (mainly because I will build some myself) - they can evolve, change, add new features, hide others, that is fine - but what I need is the most direct access possible to the primitives.
Concrete example: I just do not want to have to wait for things like:
https://github.com/dagger/dagger/issues/2756
I absolutely understand and respect why it is taking time.
But then from my perspective, that means Dagger API is acting as a gatekeeper introducing its own layer of complexity and bugs - I cannot implement my stuff directly, I have to first go “upstream” and expose it.
It is much more powerful for me to just have the LLB primitives exposed in a passthrough fashion (eg: ProxyEnv is an ExecOp param, that’s it - dont care if that’s getting through graphql, protobuf, or an avian carrier).
Then Dagger value could be in additional value not provided by buildkit (below).

#

———————————————————————————
The #2 feature for me is the ability to parallelize running different actions over a fleet of runners (and overall fleet management of course).
I do not have that right now myself but I am working on it.
The scenario is very much: customer X has a pipeline that currently takes 7 hours, distributed over 10 nodes (so, that is really <70 hours of compute).
I need a way to design a plan that says: run that collection of tasks over a bunch of different buildkitd - some node on developers laptop in the office, some other nodes in a kube cluster on AWS.
If I do not have that one feature ^, I simply cannot make a compelling argument for my customer to give Jenkins the kick in the buttocks that it deserves.

I am definitely going to build that (and obviously fleet management - I alluded to mTLS and PKI above ^^^).
I absolutely need cloak to either provide that ability as a feature, or allow me to build it.
If there is a way to share efforts on building this, I am listening.

#

———————————————————————————
#3 I need detailed access to data about the build.
Pretty much what you get from buildctl —trace
I do not need “tracing” per se - just basic execution information about actions (were they cached, how long did they run, exit code, stdout / stderr).
I could theoretically do that myself by wrapping out ExecOp calls made by the user inside my own scripting, and time the actions, but we all know how that shit ends (eg: not pretty):
https://github.com/dagger/dagger/issues/80#issuecomment-768580198

———————————————————————————
Finally, #4, there are a number of other things that are doable right now piping to buildctl, that I definitely need:

  • import/export cache control
  • multi-arch build management
  • control over entitlements (eg: host networking)

Sorry for the long writeup.
Hit me whenever if you want to chat about something ^.

cosmic cove
tepid nova
#

@untold shoal thanks for the writeup! If you don't plan on using the Cloak SDKs, because you plan on creating your own SDKs at approximately the same level of abstraction, then that might be a sign that you should not use Cloak. Perhaps your project and Cloak are just "cousin projects" with similar goals but different strategies for achieving them. And that's ok! We can still help each other out and probably reuse each other's code in the future. But IMO using Cloak without using the Cloak SDKs will be a frustrating experience for you.

tepid nova
#

@hasty basin @obsidian rover @dense dust FYI here's the current status of cloak docs:

  • We had a bunch of docs growing organically in a few different places: README, docs/, docs/next. We just moved then all into docs/ instead
  • These docs are not fully consistent. Some started from internal architecture, others from UX, the two are mixed together at the moment
  • Meanwhile demos/ holds the "official" demos.
    • Demo V1 is the one we've showed everyone in this channel. Now frozen except for incremental improvements and maintenance
    • Demo V2 is work in progress, and also drives short-term dev priorities, ie "demo-driven development" 🙂
#

The most exciting part about demo v2 (I think) is that it describes how to adopt cloak in an existing project, and get value quickly very little disruption. That would be a major improvement over the current dagger.

wet mason
#

👋

cosmic cove
tepid nova
#

@civic yacht specifically in the context of todoapp, how should I invoke the build workflow?

#

react-scripts build

#

Since in JS/TS world everyone seems comfortable with custom command-line entrypoints to their code (yarn, node, react-scripts), perhaps our TS SDK should provide one as well?

#

sed -i package.json s/react-scripts/cloak-scripts/ ? 🤔

#

Feels weird to me, but so does most JS best practices 😉

#

I can't tell if it's "usual weird" or "unusual weird"

civic yacht
# tepid nova <@949034677610643507> specifically in the context of todoapp, how should I invok...

If your workflow is in workflows/build and its own package.json and tsconfig.json are in there too, then the low-level commands to run it would be yarn --cwd workflows/build build to transpile ts->js and node workflows/dist/index.js to actually run it.

I agree that we should wrap that in a package.json script. So one possibility is that if package.json is autogenerated by cloak generate, then we can just inline those commands as targets.

#

I wonder if there is already a way to programatically alter an existing package.json to add a new script. If so then that would be a route for the use case where the user wants to re-use an existing package.json rather than have their own.

tepid nova
#

I have to firewall the topic of "project-centric vs workflow-centric" for now, or my brain will melt... For now I'm just trying to fill the FIXME in my current version of demo (workflow-centric) and hope that it stays relevant when comparing a project-centric and workflow-centric variation of it

#

Reading your explanation, I am realizing that node cannot run index.ts directly so the transpile step must always be called first correct?

civic yacht
tepid nova
#

So assuming ts-node is not an option, how would I typically run the transpile step? Is that standardized or fragmented across a million js build tools as well?

civic yacht
#

So tsc is the standard tool AFAIK

tepid nova
#

OK that's the stuff that would go in the workflow's package.json right?

civic yacht
tepid nova
#

Yeah let's assume that for my own sanity 😉 I'm trying to not think about what happens if todoapp happens to be written in typescript

civic yacht
tepid nova
#

OK, so to recap:

  1. I write my workflow's TS code (index.ts)
  2. I write or generate my workflow's package.json, which includes a standard-ish build rule that calls tsc to transpile
  3. To build my workflow, I call yarn --cwd ./projects/workflows/build run build (or run tsc manually)
  4. Now I can run my workflow directly with node ./projects/workflows/build/index.js (or is it dist/index.js?)
  5. That node comand can be wrapped as a npm/yarn script by modifying todoapp's own build script to node ./projects/workflows/build/index.js ?
civic yacht
#

Yep that looks basically right to me.

  1. Now I can run my workflow directly with node ./projects/workflows/build/index.js (or is it dist/index.js?)
    It would be dist/index.js usually, though dist is configurable in tsconfig.json (can also just put index.js right next to index.ts if you want)
  2. That node comand can be wrapped as a npm/yarn script by modifying todoapp's own build script to node ./projects/workflows/build/index.js ?
    Yeah, it might also make sense to also put the tsc call in there too, just so you can avoid having to think about the fact that you first need to transpile your build tool before invoking it to build your code (😵‍💫 )
tepid nova
#

Yeah that's the part that hurts my head

#

I guess it doesn't help that we're using "build" as the reference workflow here...

#

It adds a little dash of recursion that is

#

A little je ne sais quoi

civic yacht
civic yacht
#

I'm not a TS/JS dev though, so we need to run this by others sometime too

tepid nova
#

For the sake of simplifying the cognitive burden, would it make sense to distribute a small npm package called tsdagger that is basically a simplified entrypoint to 1) transpile index.ts and 2) run it on the fly with node? Basically a specialized tsnode

#

That way I can just change my todoapp package.json to "build": "tsdagger ./workflows/build" ?

civic yacht
#

Oh I see, you are thinking about the fact that you have to manually write that all out

#

In that case, possibly? Or maybe the todoapp could just be "build": "yarn --cwd ./workflows/build run" and then have the package.json in ./workflows/build have a script called run that does the transpile+node exec?

tepid nova
#

Yeah that works too but it's a yarn script that itself recursively depends on, and calls, yarn

#

which again makes my head hurt

#

I'm going to let my brain cool off... I just pushed cleaned up notes to a PR. Left this particular part as a FIXME for now 🙂

#

Maybe looking at the Go workflow in more detail will help get some perspective and unblock us on the TS part

civic yacht
tepid nova
#

Btw it was implied in the demo flow, but I'm thinking for now we skip all things "universe" and just stick to git targets for everything

#

we can add the convenience of a managed namespace later

#

(git or local dir)

scarlet gate
#

👋

tepid nova
scarlet gate
cosmic cove
#

Welcome @north jay ! Thanks for your time today. We look forward to getting your feedback.

wet mason
#

@tepid nova @civic yacht Just caught up with the thread

  • All this confusion is mostly because we're not familiar with the typescript toolchain, but it's a pretty "normal" workflow for TS devs (as in, there must be an easy generally accepted solution)

  • If this were in JS (and btw it 100% could be) just to remove the TS toolchain unknowns, the generally accepted way would be to point to a .js file within scripts (e.g. "deploy": "./workflows/deploy.js"). That's what we've been doing in the past for instance to invoke custom scripts to e.g. generate the universe reference documentation. The following is an example from the npm docs:

{
  "scripts" : {
    "install" : "scripts/install.js",
    "postinstall" : "scripts/install.js",
    "uninstall" : "scripts/uninstall.js"
  }
}

  • Alternatively, if the dagger SDK installs a binary in node_modules, it could be "deploy": "ourbinary something something") (e.g. that's how react and docusaurus do it)

  • Invoking yarn from package.json is weird. Never saw something like that on the wild (with my limited experience). First of all, perhaps the user is using npm rather than yarn, the 2 are interchangeable. Also, having multiple levels of yarn is a PITA (as we've seen in cloak's own repo) -- there's "yarn workspaces" but it's uncommon and hard to set up (usually can be found in repositories exporting a bunch of different npm packages from one repo). In terms of DX it's also so so. Beside package.json/tsconfig, there's also a bunch of other config (e.g. eslint). The IDE will be confused and you'd end up with different linting configuration etc

wet mason
civic yacht
# wet mason <@488409085998530571> <@949034677610643507> Just caught up with the thread - Al...
  • All this confusion is mostly because we're not familiar with the typescript toolchain
    💯💯💯
    Invoking yarn from package.json is weird. Never saw something like that on the wild (with my limited experience). First of all, perhaps the user is using npm rather than yarn, the 2 are interchangeable
    Yeah that makes perfect sense. I was just thinking "put commands I was running on my host into the script" but you're right that it's not remotely universal.
  • If this were in JS (and btw it 100% could be) just to remove the TS toolchain unknowns, the generally accepted way would be to point to a .js file within scripts (e.g. "deploy": "./workflows/deploy.js"). That's what we've been doing in the past for instance to invoke custom scripts to e.g. generate the universe reference documentation. The following is an example from the npm docs:
    Awesome, do you know if those .js files have something like #!/usr/bin/env node at the top? Or is there magic logic where if the script points to .js file its invoked with node? Either way, sounds like a better approach
civic yacht
# wet mason Yeah, from a quick search this seems to be the way. Perhaps we could use some he...

Agree, my limited understanding of the problem is that

  1. I added a dependency on execa to the dagger sdk, which is ESM only: https://github.com/sindresorhus/execa/issues/489
  2. I fixed it by making the dagger sdk esm only, which then forced me to make the workflow script that imports the dagger sdk as a library esm only
  3. But it turns out ts-node doesn't support that: https://github.com/TypeStrong/ts-node/issues/935

I don't really understand the underlying issues here at all. I gleaned that ESM is supposed to be the "future-way" that replaces commonjs, but that the rollout has been taking a long time and not going smoothly, causing issues like this?

I also suspect it may have been possible to using the async import() to import execa, which probably would have allowed me to keep everything commonjs, but I'm not sure if that has other implications and not sure if it would totally fix the issues with ts-node since execa would still be in the dependency tree?

#

^ mood when trying to understand and explain all of this

wet mason
#

Haha totally 🙂

mellow bolt
#

It's late for me, but I can think about it tomorrow

civic yacht
wet mason
#
  • Alternatively, if the dagger SDK installs a binary in node_modules, it could be "deploy": "ourbinary something something") (e.g. that's how react and docusaurus do it)

There's still this avenue if we need it

Basically package.json has node_modules/.bin in PATH, and the SDK can provide a binary. So we can provide some wrapper if needed

civic yacht
dense dust
#

The conflict between ESM and CommonJS is wonderful because it makes you rethink your own existence.

#

The transition from CJS is really slow because they expected mantainers to update their libraries to ESM fast, so it's a pain now.

#

The newest libraries like Vite, Vitest, Vue and such, tend to use ESM, so we should aim for that IMO instead of CJS.

civic yacht
dense dust
#

Yes, making a library ESM and CommonJS compatible is the best way to go, but I'm not sure how hard it is.

cosmic cove
wild zephyr
# tepid nova OK, so to recap: 1. I write my workflow's TS code (index.ts) 2. I write or gene...

we could also have a start/run target in our package.json which compiles and runs the app directly: i.e

{
  "name": "todo-app",
  "version": "1.0.0",
  "description": "",
  "main": "dist/app.js",
  "scripts": {
    "start": "tsc && node dist/app.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "devDependencies": {
    "typescript": "^4.5.2"
  }
}

this is very common in the JS/TS ecosystem

scarlet gate
#

Hey,
today I started with cloak. I tried the step in the getting started cloak -p examples/yarn/cloak.yaml do --local-dir source=examples/todoapp/app --set name=build

When I execute this I get this.

cmd/cloak/generate.go:5:2: package embed is not in GOROOT (/usr/local/go/src/embed)
../../../../go/pkg/mod/github.com/99designs/gqlgen@v0.17.14/codegen/templates/templates.go:7:2: package io/fs is not in GOROOT (/usr/local/go/src/io/fs)
#

How can I resolve this?

#

Which go version do I need to build cloak? After upgrading to version 1.19 I get another error: # runtime/cgo _cgo_export.c:3:10: fatal error: 'stdlib.h' file not found

wet mason
#

@scarlet gate Do you get this when compiling cloak? Or is cloak already compiled and it's when you're running it?

wet mason
scarlet gate
#

@wet mason I already updated to version 1.19. And here I get this one # runtime/cgo _cgo_export.c:3:10: fatal error: 'stdlib.h' file not found

I will try to update my x-code stuff.

I will document this in an easy 'Prerequesit' and 'Troubleshoot' section.

wet mason
#

Which platform are you running on? If you want, until we figure out what’s wrong, I can just build a binary and send it to you?

#

The error looks like a Go toolchain error (broken install)

#

From what I’m seeing it’s a problem with Go and macOS upgrades

scarlet gate
wet mason
#

@scarlet gate x86 or ARM?

scarlet gate
wet mason
#

ok, coming up

#

sent!

cosmic cove
wet mason
#

@civic yacht Hey, been pairing with @stray heron on a workflow on top of #95 (to get the workflow goodies) -- getting this error: could not find : stat : no such file or directory

First it was at cloak generate, got around it by using the main version of cloak for generation

But now happening as well just by running the workflow

#

(
also: had to add a small panic fix:

diff --git a/engine/engine.go b/engine/engine.go
index feb7d7e..d167843 100644
--- a/engine/engine.go
+++ b/engine/engine.go
@@ -90,6 +90,9 @@ func Start(ctx context.Context, startOpts *Config, fn StartCallback) error {
                        socketProviders,
                },
        }
+       if startOpts.LocalDirs == nil {
+               startOpts.LocalDirs = make(map[string]string)
+       }
        startOpts.LocalDirs[workdirID] = startOpts.Workdir
        solveOpts.LocalDirs = startOpts.LocalDirs

)

civic yacht
wet mason
#

I know the branch is WIP but @stray heron was getting into writing a workflow and thought it'd be better to try and do that on top of the WIP PR 🙂

#

@civic yacht no extensions so far

#

(e.g. we avoided doing that, just plain old core)

civic yacht
#

Oh for sure, the JS workflow actually runs successfully right now for me (though you will need to change package.json to not use the git+ssh dep for dagger sdk for now)

civic yacht
# civic yacht Oh for sure, the JS workflow actually runs successfully right now for me (though...

Actually this is a good thing to chat about quickly if you have time @wet mason https://github.com/sipsma/cloak/blob/220d40b2d98bc5ed8d3f44ea727463e88ab410db/examples/todoapp/app/package.json#L39-L39

Basically, until we have an actual npm package our choices are to either keep using local deps (tons of ../../../) or to use the git+ssh to pull from private repo. The problem is that package.json doesn't let you specify a subdir of the git repo. So then the choices are either to rearrange our dagger/cloak repo to have an actual package.json in the root that builds the sdk or to create something like dagger/cloak-nodejs-sdk and move the sdk out to its own repo.

#

I would have liked to avoid moving the sdk to its own repo so early on, but it is probably the direction we want to go long term anyways, so maybe that's the right choice for now? Any thoughts?

stray heron
civic yacht
tepid nova
#

Now would be a great time to start working on a generated API reference doc for cloak... Anyone interested in that challenge?

#

@wet mason you mentioned you started looking at existing tooling in the graphql ecosystem, did you find something that you like?

wet mason
#

Also, alone it’s not very very useful just as a “blurb” of reference docs

It gets more useful with sample queries and a bit of editorial

Similar to godoc.org — alone it’s not better than getting the same information straight from vscode, it becomes useful with the “examples” section

I believe one of them (spectaql I think) had a section were you could shove example queries along with some markdown

dense dust
#

Taking a look 👀

cosmic cove
#

@dense dust if you are interested in grabbing it, do you mind creating an issue too and tagging multi-lang?

tepid nova
#

@wet mason @civic yacht I'm trying my first demov2 run through, from main, I'm guessing I need to make to change my demo script so that it actually runs. Can you help me figure out what changes?

#
  • Should I move cloak.yaml to the root of todoapp (project-centric) assuming that's how the current implementation works?
#
  • How should I invoke the build workflow from todoapp? How many package.json files, what should be their contents etc
civic yacht
# tepid nova <@707661669819613324> <@949034677610643507> I'm trying my first demov2 run throu...

main doesn't have most of the changes needed, I've only merged a few standalone parts into main so far. You'll need changes present in the PR here: https://github.com/dagger/cloak/pull/95

I am still working on it obviously, but if you want to run through it I can either post some updated docs with a few of the commands needed (some flags have changed) or we can just work through it together, whatever you prefer

tepid nova
#

There seems to be consensus that in the case of a TS workflow run inside a JS project, we should combine package.json

civic yacht
#

In the sense that workflows are meant to enable gradual adoption of dagger into an existing project and thus should be part of your existing toolchain. But also just in the low-level technical details it makes things simpler overall

tepid nova
#

That sounds good to me. As long as we're not forcing everyone to do it. It will not always be practical.

#

There's a reason only JS devs use JS tooling.

civic yacht
tepid nova
#

ie. There will be workflows which cannot be mixed with the app to that point, and we want the experience of gradual adoption for those workflows to be painless too

#

So, with that PR 95, what's an example of cloak.yaml that would work right now?

civic yacht
#

You will need to have an ssh agent setup w/ SSH_AUTH_SOCKET btw (to pull deps from private git repos, which is what allows us to get rid of all the local ../../../ deps)

wet mason
#

Happy to chat — heading to dinner now

tepid nova
#

Generally I agree that frictionless adoption is the goal (as long as it's frictionless for everyone as discussed above). We have fully sandboxed extensions as our safety net: even if we screw up some aspects of the workflow DX, the extension ecosystem will preserve the integrity of the platform.

#

Better to screw up as little as possible though 😉

#

@civic yacht awesome thank you!

#

btw @civic yacht @wet mason looking at the CLOAK_CONFIG=... hack, makes me think that maybe having cloak generate embed the contents of cloak.yaml into the code wouldn't be so crazy 🙂

civic yacht
# tepid nova btw <@949034677610643507> <@707661669819613324> looking at the `CLOAK_CONFIG=......

I went down that route initially. It's plausible but it creates unavoidable boilerplate because there has to be user-written code that takes care of the embedding (can't be farmed out to the sdk library). I also don't know how plausible it is in every language (it is in js/ts and go, but not sure in general).

Also, on a more immediate level, it creates a headache in go because you embed the file, but buildkit's localdirs (which is how cloak.yaml is loaded) require an actual directory. Buildkit should be updated to accept a generic fs.FS but that's significant upstream changes. Only other possibilities in meantime are hacks.

The other possibility is to just switch to the project-centric approach. Actually in the current state of that PR it would literally just be a matter of copying the cloak.yamls to be in todoapp's root and then having the engine default to reading cloak.yaml from the workdir. We don't need to restart that whole thread now though, probably best for a live discussion later

civic yacht
# tepid nova <@949034677610643507> awesome thank you!

Np, another thing you'll probably run into:

If you are writing a go workflow, you'll need to make a go.mod of course. I went with putting it in todoapp's root because it surprisingly simplified things (can go into details if desired). But no matter it's location, you will end up putting a dependency on github.com/dagger/cloak in order to pull the Engine lib. This is a private git repo though, and go makes it insanely annoying to do that. You will have to

  1. (temporarily) have this replace since you need updates from the commit in the PR: https://github.com/sipsma/cloak/blob/5d91c69e976368ca957c969fd6d3511445a6e22a/examples/todoapp/app/go.mod#L8-L8
  2. export GOPRIVATE=github.com/sipsma/cloak
  3. Update your .gitconfig (probably in ~) with this:

[url "ssh://git@github.com/sipsma/cloak"]
insteadOf = https://github.com/sipsma/cloak

#

(once the PR is merged all the sipsma->dagger of course)

tepid nova
#

Yeah figuring out project-centric or workflow-centric will be super important, as soon as we have a working demov2 we can resume that 🙂 To be clear I am open to either, I'm just scared of getting it wrong and regretting it later.

civic yacht
tepid nova
#

I'm trying to find a way not to live type that CLOAK_CONFIG= during the demo, but don't see any

#

Oh one more noob question. I wrote my index.ts and ran cloak -p todoapp/workflows/build/cloak.yaml generate. It didn't fail but it didn't generate anything

# 
$ cloak -p  ./todoapp/workflows/build/cloak.yaml generate
#1 local://.workdir
#1 transferring .workdir: 8.33kB done
#1 DONE 0.0s

#2 copy / /
#2 CACHED
civic yacht
tepid nova
#

No I don't need the stubs. Still will run the command since as a noob cloak user I don't know exactly what that command does or if anything will work without it 🙂

#

Also I saw index.mjs in your example package.json and assumed that would be generated too?

#

Basically I wrote index.ts and cloak.yaml and don't know what to do next

civic yacht
# tepid nova Also I saw `index.mjs` in your example package.json and assumed that would be ge...

No that's because I just avoided some of the ts related issues by starting with js. You can still import our ts sdk and use it, so it worked for now. The .mjs is related to es modules (if you try to make it be just .js you get some wonderful errors).

If you want to use ts you'll have to add a tsconfig.json to todoapp, run yarn add --dev typescript, etc. Given that todoapp is just js and not ts, I'm thinking it may make sense to just go with js for it in the demo

#

Given that the workflow is mostly just graphql queries, you don't miss the type checking all that much

tepid nova
#

OK, in practical terms do I just rename my index.ts to index.js and then call it with node from todoapp's package.json?

#

Sorry mean .mjs

#

Do I remove the ```
gql`

civic yacht
# tepid nova Sorry mean `.mjs`

Yeah use .mjs for now. Once I have a free moment to sync up with Julian+JF hopefully I can figure out how to get that working with .js.

Do I remove the
No actually, that somehow works still. I thought the template strings were ts specific but I must have misunderstood something because they magically work still. Another question for our js+ts experts 🙂

tepid nova
#

I am embracing the magic

civic yacht
tepid nova
#

Note on the workdir plumbing in package.json. Do I understand correctly that yarn sets workdir to the project dir, so scripts can always rely on $(pwd) returning a meaningful path, instead of wherever the user was sitting when they ran yarn?

#

I'm guessing npm does the same

#

As a non-JS developer I am sad that I now have to install yarn on my machine...

civic yacht
#

So it works out how we want it

#

(based on my test I conducted 5 seconds ago)

civic yacht
tepid nova
#

Yes for sure... Also figuring out user-friendly invocation of extensions straight from the CLI, so that I can skip the workflow part altogether (or make it a shell script)

# 
$ sudo apt install yarnpkg
[...]
The following NEW packages will be installed:
  javascript-common libc-ares2 libicu66 libjs-inherits
  libjs-is-typedarray libjs-psl libjs-source-map libjs-sprintf-js
  libnode64 node-ajv node-ansi-escapes node-ansi-regex node-ansi-styles
  node-anymatch node-argparse node-array-find-index node-asap node-asn1
  node-assert-plus node-asynckit node-aws-sign2 node-aws4
  node-babel-runtime node-balanced-match node-bcrypt-pbkdf node-big.js
  node-bl node-brace-expansion node-braces node-builtin-modules
  node-bytes node-camelcase node-caseless node-chalk node-chownr
  node-ci-info node-cli-cursor node-cli-table node-cli-width node-clone
  node-color-convert node-color-name node-colors node-combined-stream
  node-commander node-concat-map node-core-util-is
  node-currently-unhandled node-dashdash node-death node-debug
  node-deep-equal node-defaults node-delayed-stream node-detect-indent
  node-duplexify node-ecc-jsbn node-emoji node-emojis-list
  node-end-of-stream node-escape-string-regexp node-esprima
  node-exit-hook node-extend node-external-editor node-extsprintf
  node-fast-deep-equal node-fast-levenshtein node-fill-range
  node-forever-agent node-form-data node-fs.realpath node-getpass
  node-glob node-graceful-fs node-har-schema node-har-validator
  node-has-flag node-http-signature node-iconv-lite node-imports-loader
  node-inflight node-inherits node-ini node-inquirer node-invariant
  node-ip-regex node-is-buffer node-is-builtin-module node-is-number
  node-is-promise node-is-typedarray node-isarray node-isstream
  node-js-tokens node-js-yaml node-jsbn node-jschardet node-json-schema
  node-json-schema-traverse node-json-stable-stringify
  node-json-stringify-safe node-json5 node-jsonify node-jsprim
  node-kind-of node-loader-utils node-lodash node-lodash-packages
  node-loose-envify node-loud-rejection node-micromatch node-mime
  node-mime-types node-minimatch node-mkdirp node-ms node-mute-stream
  node-node-uuid node-normalize-path node-oauth-sign node-object-assign
  node-object-path node-once node-path-is-absolute node-path-root
  node-path-root-regex node-performance-now node-process-nextick-args
  node-proper-lockfile node-psl node-puka node-pump node-pumpify
  node-punycode node-qs node-read node-readable-stream
  node-regenerator-runtime node-repeat-string node-request
  node-request-capture-har node-resolve node-restore-cursor node-retry
  node-rimraf node-run-async node-rx node-safe-buffer node-semver
  node-signal-exit node-source-map node-spdx-correct
  node-spdx-exceptions node-spdx-expression-parse node-spdx-license-ids
  node-sprintf-js node-sshpk node-ssri node-stream-shift
  node-strict-uri-encode node-string-decoder node-string-width
  node-strip-ansi node-strip-bom node-supports-color node-tar-stream
  node-through2 node-tmp node-to-regex-range node-tough-cookie
  node-tunnel-agent node-tweetnacl node-uri-js node-util-deprecate
  node-uuid node-validate-npm-package-license node-verror
  node-wcwidth.js node-wrappy node-xtend node-yallist node-yn nodejs
  nodejs-doc yarnpkg
[...]
4 upgraded, 186 newly installed, 0 to remove and 1462 not upgraded.
Need to get 20.0 MB of archives.
After this operation, 101 MB of additional disk space will be used.
Do you want to continue? [Y/n]
#

😭

civic yacht
civic yacht
#

It's something else

#

at least on debian it is

#

I might be wrong though, let me double check

#

Nevermind, I'm misremembering, ignore me

tepid nova
#

There's a custom apt repo

#

Getting git auth issues on yarn install...

$ yarn
yarn install v1.22.19
[1/4] Resolving packages...
error Command failed.
Exit code: 128
Command: git
Arguments: ls-remote --tags --heads git@github.com:sispma/cloak.git
Directory: /home/shykes/dagger/demov2/todoapp
Output:
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
#

I have ssh agent running

civic yacht
#

sispma -> sipsma

tepid nova
#

arg

#

Thanks it worked

#

Is there any situation where running yarn install on the host, then again in a containerized pipeline, may cause trouble? For example if host and container runtime are not the same platform?

#

Another one:

$ yarn install
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
error execa@6.1.0: The engine "node" is incompatible with this module. Expected version "^12.20.0 || ^14.13.1 || >=16.0.0". Got "10.19.0"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Looks like that's just my node version on the host that is too old or broken?

civic yacht
tepid nova
#

So far the hardest part of preparing this demo is switching to a primitive setup with dev tools installed directly on the host

civic yacht
tepid nova
#

cc @dense dust

#

Still stuck on installing nodejs on ubuntu..

#

Problem is I'm on a 18.04 host with node 10.19 installed. It's a physical host with barely anything installed, because I use containers like a non-crazy person

dense dust
#

I'll be home in 30. Usually nvm is used to manage Node versions

#

It's a cli tool I'm using currently and allows using different Node.js version in different directories

civic yacht
#

I was going to post my dev env's dockerfile that installs from a separate repo, but that sounds even better

tepid nova
#

Thanks, I've used nvm before and was hoping to avoid repeating that experience. Will give it a try now

#

The heart of the issue is that ubuntu 18.04 is very old

mellow bolt
dense dust
#

I'm running 20.04 with no issues tho

tepid nova
#

18.04 is LTS, I'm running it on a physical host, no plan to upgrade anytime soon

#

OK I got it to work with nvm... I will clean up the mess later

dense dust
#

But don't know any specific solution

tepid nova
#

I guess whatever the best practices are, we can implement them in a cloak extension (yarn? npm? node?) then I can uninstall all the crap and just run cloak do yarn build or something

# 
file:///home/shykes/dagger/demov2/todoapp/workflows/build/index.mjs:4
new Engine().run(async (client: GraphQLClient) => {
                        ^^^^^^

SyntaxError: missing ) after argument list

@civic yacht I'm guessing there's typescript-only typing information in there?

civic yacht
dense dust
#

I guess npm + yarn would be the most used. Right now pnpm (performant npm is gaining the most traction)

#

Basically yarn appeared to replace npnm and pnpm to replace yarn lol

civic yacht
tepid nova
# 
$ yarn run build
yarn run v1.22.19
$ CLOAK_CONFIG=./workflows/build/cloak.yaml node workflows/build/index.mjs
file:///home/shykes/dagger/demov2/todoapp/node_modules/@dagger.io/dagger/dist/engine.js:11
        if (!this.config.Workdir) {
                         ^

TypeError: Cannot read properties of undefined (reading 'Workdir')
    at Engine.run (file:///home/shykes/dagger/demov2/todoapp/node_modules/@dagger.io/dagger/dist/engine.js:11:26)
    at file:///home/shykes/dagger/demov2/todoapp/workflows/build/index.mjs:4:14
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:528:24)
    at async loadESM (node:internal/process/esm_loader:91:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12)

Node.js v18.8.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
# 
// Build todoapp, the hard way
import { gql, Engine } from "@dagger.io/dagger";

new Engine().run(async (client) => {
  // 1. Load app source code from working directory

  // 2. Install yarn in a container

  // 3. Run 'yarn install' in a container

  // 4. Run 'yarn run build' in a container

  // 5. write the result back to workdir
});
#

(empty implementation for now, following @hasty basin suggestion of demo flow)

mellow bolt
#

you should use if (!this?.config?.Workdir). Optional chaining

#

with question marks

civic yacht
mellow bolt
#

to avoid those errors

tepid nova
tepid nova
civic yacht
dense dust
tepid nova
#

Success! I have successfully run a worfklow that does nothing 🙂

$ yarn run build
yarn run v1.22.19
$ CLOAK_CONFIG=./workflows/build/cloak.yaml node workflows/build/index.mjs
==> dev server listening on http://localhost:8080#1 local://.workdir
#1 transferring .workdir: 7.19kB done
#1 DONE 0.0s

#2 copy / /
#2 CACHED
Done in 0.75s.
civic yacht
mellow bolt
#

@civic yacht do you have some time tomorrow to show me TS issues ?

civic yacht
mellow bolt
#

8 is perfect

tepid nova
civic yacht
civic yacht
mellow bolt
tepid nova
#

Thanks for your help @dense dust @mellow bolt

mellow bolt
#

I'll have a closer look to node sdk and play with cloak 🙂

dense dust
dense dust
mellow bolt
#

of course

#

I send you an invite

tepid nova
#

I tried to add my first query (interactively crafting queries with the sandbox is always a delight). Getting a new error when trying to run it:

$ yarn run build
yarn run v1.22.19
$ CLOAK_CONFIG=./workflows/build/cloak.yaml node workflows/build/index.mjs
==> dev server listening on http://localhost:8080#1 local://.workdir
#1 transferring .workdir: 7.19kB done
#1 DONE 0.0s

#2 copy / /
#2 CACHED

/home/shykes/dagger/demov2/todoapp/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1491
                        reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
                               ^
FetchError: request to http://localhost:8080/ failed, reason: read ECONNRESET
    at ClientRequest.<anonymous> (/home/shykes/dagger/demov2/todoapp/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:513:28)
    at Socket.socketErrorListener (node:_http_client:494:9)
    at Socket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNRESET',
  code: 'ECONNRESET'
}

Node.js v18.8.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ancient kettle
mellow bolt
tepid nova
#

Just checked, nope

civic yacht
tepid nova
# 
// Build todoapp, the hard way
import { gql, Engine } from "@dagger.io/dagger";

new Engine({
        ConfigPath: process.env.CLOAK_CONFIG
}).run(async (client) => {
  // 1. Load app source code from working directory

  // 2. Install yarn in a container
  const image = await client.request(gql`
        {
          core {
            image(ref:"index.docker.io/alpine") {
              exec(input: {
                args:["apk", "add", "yarn"]
              }) {
                stdout
                fs {
                  exec(input:{args:["apk", "add", "git"]}) {
                    stdout
                    fs {
                      id
                    }
                  }
                }
              }
            }
          }
        }
`).Then((result) => result.core.image.exec.fs.exec.fs)

  // 3. Run 'yarn install' in a container

  // 4. Run 'yarn run build' in a container

  // 5. write the result back to workdir
});
mellow bolt
#

not sure it will change anything but you .Then should be lowercase

tepid nova
civic yacht
tepid nova
mellow bolt
#

\o/

tepid nova
#

Looks like we're missing some logging / error reporting when the engine fails to initialize

civic yacht
#

Yeah my best guess is that .Then causes an error, which causes a promise rejection, but because we have a finally where the server gets shutdown wrapping that promise, we somehow end up with that error instead of the original .Then one

#

Or something

wet mason
dense dust
#

Yep, await is sugar syntax, widely replaced the Promise syntax lately

#

It makes handling Promises easier

#

You would assign the value of the awaited function to a variable instead of chaining the Promise with then in case it's fulfilled.

dense dust
# tepid nova I pushed it to a branch of todoapp if you want ot reproduce: https://github.com/...

When I run yarn run build I get this. I pulled the cloak repo and followed the instructions in the Getting Started doc, so I guess I have the latest version

x yarn run build
yarn run v1.22.19
warning ../../../package.json: No license field
$ CLOAK_CONFIG=./workflows/build/cloak.yaml node workflows/build/index.mjs
Error: unknown flag: --workdir
Usage:
  cloak dev [flags]

Flags:
  -h, --help                help for dev
  -l, --local-dir strings   local directory to import
      --port int            dev server port (default 8080)

Global Flags:
  -c, --context string   project context (default ".")
  -p, --project string   project config file (default "./cloak.yaml")

Error: unknown flag: --workdir
hasty basin
#

gq is fun!

npm install -g graphqurl
cloak dev
gq http://localhost:8080 -q 'query { core { git(remote: "https://github.com/dagger/dagger") { file(path: "README.md") }}}'
civic yacht
dense dust
#

I think something with the try/catch block would look like this

#

God I love this tool daggerfire

#

Because basically if you prefer using the Promise syntax you can too

#

And like this you would catch the error higher up the stack instead of using the server catch as a fallback

unknown.png
#

So from a monitoring standpoint I guess you could call any service from that catch block

#

In case any action fails

#

So many use cases come to mind 🤯

tepid nova
#

Nice @dense dust I was just wondering how to handle an error on file (to check if eg. a package.json exists in your repo)

civic yacht
# dense dust In case any action fails

Yeah error checking+handling is something mentioned by users in many if not most of our demos! It's just much easier to think about when you get to use normal language constructs for it

dense dust
#

Yep, my question would be how to decouple different queries, for example, because that way we could write specific error handling for each action

dense dust
# dense dust

What I mean is, here I'm declaring a query chain inside const image. Can I chain that const to another one?

#

Because if I'm able to do that, I can chain different gql queries and throw different errors for each one

#

Pseudocode would be something like

# 
try {
  const query1 = await gql...
    if (query1.error) throw new Error("query1")
  const query2 = await query1.gql...
    if (query2.error) throw new Error("query1")
} catch(err) {
    if (err.msg === "query1") {
      ...
    }
    if (err.msg === "query2") {
      ...
    }
}
#

I mean it would be like introducing steps inside the workflow, so each step can have different handling in case it fails, by chaining results of different queries

#

I guess that each time a query ends, so does the container, so it's impossible maybe

tepid nova
#

you can get the filesystem id output by one rquery, and pass it as input to another query

#

take a look at the demo v2 scenario it has an example of that

dense dust
#

Oh I see it, thanks!

tepid nova
#

Note it probably doesn't build as-is (I'm running through the demo from scratch and haven't reconciled the result back into that doc yet)

#

but it's a few typos away from running 🙂

#

Current status: sitting outside in a mall in Honolulu, setting up my newly acquired macbook pro 🙂

#

I am cutting ubuntu 18.04 from the equation

#

Kind of killed my productivity for today, but I'll get a day back on the flight home

#

(Note to self: hardcode digests so that buildkit doesn't try to fetch latest from the plane..)

civic yacht
tepid nova
#

It seems I have swapped my nodejs versioning issues with rosetta compiler issues 😭

civic yacht
tepid nova
tepid nova
civic yacht
tepid nova
#

I think you mentioned there is a proposal for that on the docker side?

tepid nova
#

Oh wait

#

Not what I meant, but yeah I agree

#

maybe the year after next for that

#

boot2cloak

#

btw Steeve (soon to join this channel) wrote boot2docker

#

Which was the first step towards docker for mac

civic yacht
#

We can compete with google's fuschia thing

I think you mentioned there is a proposal for that on the docker side?
Yeah there's a buildkit PR open to support pinning sources. I have not looked at the details yet, but glad it's coming soon hopefully

tepid nova
#

Meanwhile : has anyone by chance dealt with an apple silicon mac refusing to run intel binaries? (rosetta "magic" doesn't work as it usually does)

civic yacht
#

I run an arm64 linux vm (using an app called UTM) and just do all my dev in there, so I guess I've avoided x86 entirely

tepid nova
#

Fixed it with softwareupdate --install-rosetta 🙂

#

I needed it for Authy and Lastpass which are in the critical path to bootstrap everything else

#

(including my github access)

dense dust
#

I'm hitting a wall here I think

#

I'm trying to chain with an ID, and something seems to fail in the server, maybe I'm doing something wrong

civic yacht
# dense dust

Ah I think you are just missing " quotes around the value of id: on line 32

#

However, the whole server panicking doesn't seem like a great reaction to that...

#

I'll see if it's any easy fix

dense dust
#

Oh you were right

tepid nova
#

@dense dust you mentioned the npm -> yarn -> npm yoyo... Do you think we should switch our demo to user pnpm instead of yarn?

#

I guess running npm has the benefit of being familiar and (perhaps) also bleeding edge

dense dust
#

I think it depends on the public you are aiming for

#

Latest libraries usually have an example with pnpm in their docs

tepid nova
#

The goal is to follow whatever is accepted as best practice by at least 50% of the JS community 🙂

dense dust
#

But more established ones may not

#

Ok, so npm should be the one with the most market share

tepid nova
#

(doesn't mean 50%+ actually do it that way. Just that they know they should)

dense dust
#

I think there should be a way to know that

#

Let me check

tepid nova
#

That way it's both 1) acceptable to the 1% most advanced users, and 2) familiar to everyone else

#

back to nodejs issues... with homebrew this time

# 
gyp info it worked if it ends with ok
gyp info using node-gyp@9.0.0
gyp info using node@18.7.0 | darwin | arm64
gyp info find Python using Python version 3.8.9 found at \"/Library/Developer/CommandLineTools/usr/bin/python3\"
gyp info spawn /Library/Developer/CommandLineTools/usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/opt/homebrew/Cellar/node/18.7.0/libexec/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shykes/dev/dagger/todoapp/node_modules/fsevents/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/opt/homebrew/Cellar/node/18.7.0/libexec/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shykes/Library/Caches/node-gyp/18.7.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/shykes/Library/Caches/node-gyp/18.7.0',
gyp info spawn args   '-Dnode_gyp_dir=/opt/homebrew/Cellar/node/18.7.0/libexec/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/shykes/Library/Caches/node-gyp/18.7.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/shykes/dev/dagger/todoapp/node_modules/fsevents',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  SOLINK_MODULE(target) Release/.node
  CXX(target) Release/obj.target/fse/fsevents.o
In file included from ../fsevents.cc:6:
../../nan/nan.h:2536:8: warning: 'SetAccessor' is deprecated: Do signature check in accessor [-Wdeprecated-declarations]
  tpl->SetAccessor(
       ^
/Users/shykes/Library/Caches/node-gyp/18.7.0/include/node/v8-template.h:837:3: note: 'SetAccessor' has been explicitly marked deprecated here
  V8_DEPRECATED(\"Do signature check in accessor\")
  ^
/Users/shykes/Library/Caches/node-gyp/18.7.0/include/node/v8config.h:460:35: note: expanded from macro 'V8_DEPRECATED'
# define V8_DEPRECATED(message) [[deprecated(message)]]
                                  ^
In file included from ../fsevents.cc:6:
In file included from ../../nan/nan.h:2884:
../../nan/nan_typedarray_contents.h:34:43: error: no member named 'GetContents' in 'v8::ArrayBuffer'
      data   = static_cast<char*>(buffer->GetContents().Data()) + byte_offset;
                                  ~~~~~~~~^
1 warning and 1 error generated.
make: *** [Release/obj.target/fse/fsevents.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/opt/homebrew/Cellar/node/18.7.0/libexec/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Darwin 21.4.0
gyp ERR! command \"/opt/homebrew/Cellar/node/18.7.0/bin/node\" \"/opt/homebrew/Cellar/node/18.7.0/libexec/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js\" \"rebuild\"
gyp ERR! cwd /Users/shykes/dev/dagger/todoapp/node_modules/fsevents
gyp ERR! node -v v18.7.0
✨  Done in 6.38s.
#

This is with both yarn and node installed straight from homebrew

#

we're going to need SDK-specific sub-channels pretty soon 🙂

dense dust
tepid nova
#

Looks like removing yarn.lock and node_modules is solving it

dense dust
#

Thousands of JS questions in Stack Overflow often end with a comment from the author: "nevermind, I just deleted node_modules and reinstalled them"

dense dust
#

Here is a 2020 survey showing npm as a clear winner in the Utilities section. I don't know why the 2021 version does not have this section, but I don't think it changed much. I think at this point pnpm didn't exist either. I would use npm and announce the capability of using any package manager

scarlet gate
#

Hey,
I had to update my x-code
When I now execute this command cloak -p examples/yarn/cloak.yaml do --local-dir source=examples/todoapp/app --set name=build then I get the following message back:

exit status 1```

Did I miss something?
tepid nova
#

@scarlet gate looks like it can’t find a working buildkit

civic yacht
#

Yeah I was just confirming that's the error you get when dagger-buildkitd isn't running, it is. One sec @scarlet gate I will send you a command to run to start buildkit and then update the docs w/ it too

civic yacht
tepid nova
#

I’m familiar with the error since I ran into it earlier today 🙂

tepid nova
#

Does cloak currently support BUILDKIT_HOST?

civic yacht
# tepid nova Hence this issue 👆

I'll just go take care of that now.

Does cloak currently support BUILDKIT_HOST?
No. I was just thinking I'll also go copy-paste the code from dagger into cloak for setting this up automatically, which should include BUILDKIT_HOST but also starting dagger-buildkitd in docker if not already there

scarlet gate
tepid nova
civic yacht
#

(will fix the linting+test errors in PR tomorrow)

tepid nova
#

It would pretty amazing if the Dagger for VS Code extension could somehow auto-complete graphql queries as you type them in your code...

wet mason
#

Came across that when looking into the ecosystem tools. There’s a link somewhere in this channel

#

Caveat is it needs a cloak dev running (need a http endpoint)

#

Although our TS SDK (or a separate cloak eslint plugin) could get rid of that requirement

#

The joys of riding on top of an existing ecosystem 🙂

#

2 ecosystems in this case. The TS stuff for graphql is just first class

#

Current demo is still mostly built on top of the SDK we came up with for the very first prototype, at that point we didn’t have time for any niceties. There’s massive room for improvement

tepid nova
#

Another cosmetic bonus which could add a "wow" effect: since graphql queries are trees (ie. a limited subset of DAGs) and have a simple syntax to represent the path in the tree, we could group console output by graphql path, and get a nice hierarchical view of what's going on "for free"

#

Unlike in CUE where a given action could have any number of paths in the same CUE dag (because of references), and countless subtleties on how to format that path

#

especially with extensions, queries stay nice and sweet - and so does the console output

#

Also since graphql queries support comments, and those are received by the server (I guess?) you could optionally use comments in your script or extension to make the output nicer

#

Fun fact @wet mason: if you write a JS cloak workflow for a JS app, and you manage both with the same package.json, the workflow can brick itself by messing with the app's node_modules (because it is also it's own node_modules)

#

ask me how I know 😂

#

Also: the arch sharing in node_modules between host and containers may be a real issue

mellow bolt
tepid nova
#

I just ran into a tricky aspect of containerizing your existing automation scripts in place... Now seems obvious in retrospect but it took me a while to realize what was going on:

If you modify your build yarn script to:
* Build a container
* Mount your app source in that container
* Run yarn run build in the container...
* You are now stuck in a recursive loop 🙂

#

Thank god I didn't have cloak installed in that container or who knows how deep it would have gone

#

We're going to need an elegant solution to this.. duplicating each script in package.json is not going to scale

#

Will be the same problem in Makefiles, etc

tepid nova
#

For now I deal with it by running the contents of the npm script directly, rather than running npm/yarn inside the container.

In the case of todoapp's build script, this:

"build": "react-scripts build"

becomes this:

args: ["./node_modules/.bin/react-scripts", "build"]
#

This makes me think we could actually re-think the JS SDK entirely around the concept of package.json hooks

#

(I mean re-think the DX, not how it works under the hood)

wet mason
#

@tepid nova Yeah, very true, bunch of edge cases

Replacing the build itself with cloak is a pretty novel idea (compared to adding something “new”, like a deploy which is a net new rather than a replacement for an existing script)

I think there’s ton of potential in this, but I don’t know yet in practice if it’ll stick, time will tell

#

For instance, to be worth it, it must be at least as fast as yarn build without cloak. In theory it can be the case, but in practice in cue dagger we never managed to achieve this (reality of the overhead of having a VM on mac, paying host/vm/buildkit transfer costs), etc

tepid nova
#

Even if the script is new, you'll still (probably) want to invoke it like all your other scripts

wet mason
#

Oh yes totally agree with that

#

I just meant using cloak to create yarn deploy vs using cloak to replace yarn build

tepid nova
#

The choice of build specifically is another issue, I agree it's not necessarily the best place to start. Probably a complicated custom script is a better first target

wet mason
tepid nova
#

But the problem of "modify script Foo to run script Foo in a container" applies to all equally. Even, I think, if you're creating Foo with cloak (because it needs to fit in the same pattern as the other scripts)

#

In other words it's the same problem with build and deploy

#

Problem: my solution no longer works with a yarn API extension (since I'm no longer actually running yarn inside the container). Maybe a less naive version of the extension could still work (by reproducing the "inner" execution environment of yarn: open package.json, lookup the script, add node_modules/.bin to PATH, and run

#

Resolving at least some of those unknowns is in the critical path for demov2 FYI @wet mason ... if you have any further thoughts I'm interested 🙂

wet mason
#

Hmmm … I think even that wouldn’t do it since we are basically “moving” the script entry from package.json to the workflow (like, even opening package.json and looking up the script will still result in us invoking cloak, right?)

We’re basically lift&shifting the contents of the script from package.json into workflow.ts

#

(correct me if I misunderstood)

#

So I think that’s fine, we should just make it easier

tepid nova
#

Yes that's how we currently do it

#

But it's cumbersome

#

To write & to run

#

And it calls into question the role of "yarn" as an API extension

wet mason
#

First thought that comes to mind is to make it easier. Still use the yarn extension, but don’t use the script endpoint, use another endpoint that sets up ENV correctly etc

tepid nova
wet mason
#

So you take the contents of package.json verbatim (react-script whatnot) and just shove it into yarn { something(<paste here>) }

tepid nova
#

but it's not really yarn running it - should we maybe call the extension something else?

wet mason
#

Not glorious but at least it’s explicit about what’s going on and a simple copy paste

wet mason
tepid nova
#

yarn-ish? 🙂

wet mason
#

But there is some sort of “yarn environment thingie”

#

I mean, react-script doesn’t work from the terminal but it works in script

tepid nova
#

it's really tied to npm and more specifically the package.json spec

wet mason
#

Because yarn gives you some env guarantees (dev dependencies binaries are available)

#

There must be a name for that

tepid nova
#

the package.json spec 🙂

wet mason
#

Yeah, again this is my huge lack of knowledge about the ecosystem 🙂

tepid nova
#

Same, I've been reading this (and a few others) today to get the demo to work

#

(needed to figure out where react-scripts binary was stored to execute it directly)

wet mason
#

But yeah there’s this concept of “yarn environment”, somewhere underneath (well package.json actually)

#

There must be a good name for it and good rules

tepid nova
#

Note that this problem is not JS-specific. we'll have the exact same problem when eg. suggesting modifying your Makefile to run some rules in containers via cloak

wet mason
#

And basically what I said before: we could clone the behavior and add another action for it

#

So at least it somewhat makes sense to people familiar with it, and it’s a copy paste

#

So that’s the first idea

tepid nova
#

Once we do that though, might as well have the extension open package.json, and you can reference the script by name instead of passing inline contents right?

wet mason
#

Second idea it’s something more complex where our SDK adds a dev binary (cloak-script or whatever) and we use it to do some magic

#

But I’d need to dive deeper to figure out what the magic can actually be

tepid nova
#

(but then you need a wrapper tool to invoke that extension)

wet mason
tepid nova
#

In this scenario, we wouldn't I suppose

wet mason
#

Like we replaced the contents by node build.ts

#

The original react-script line is gone

#

I guess

tepid nova
#

We could move node workflows/build.ts to a separate key in the package.json, eg. {"cloak": { "workflows": {"build": "node workflows/build.ts"}}}

#

(it's legal)

wet mason
#

Oh I see

#

But yarn build will still invoke the script

tepid nova
#

And at that point... maybe that key is implicit and we just have the wrapper tool look workflows/<name of script> directly

wet mason
#

We’d have to do the opposite, have cloak in script and move the original command in package.json

#

Hm

#

Wait

#

What is the end goal? Just re-run whatever was in the script, as is?

#

No modifications?

#

I mean — I guess there’s like “deploy” that’s new, but for build do we only need to rerun as is?

tepid nova
#

In summary, two possible approaches:

  1. Modify original package.json scripts. Run with unmodified native tool.
  2. Preserve original package.json, add annotations. Run with thin wrapper tool.
wet mason
#

No customization?

tepid nova
#

(we moved to a thread, sorry for the noise everyone)

scarlet gate
#

How can I create the gen folder?
Was it cloak generate?

tawny flicker
#

yes

#

btw, I'm doing the tutorial for the extension.
If you cloak generate without specifying an SDK, it generates some graphql files in gen/core
If I specify an sdk (Go in my case), it generates a generated.go + a models.go + main.go, but nothing else.

#

But I guess it's because there is something missing:

cloak generate --output-dir . --context . -p example/foo/cloak.yml --sdk go
#1 local://.projectContext
#1 transferring .projectContext: 18.52kB done
#1 DONE 0.0s

#2 copy / /
#2 DONE 0.1s
2022/08/25 15:34:17 /home/dolanor/src/cloak/sdk/go/dagger/client.go:16 adding resolver method for Filesystem.loadExtension, nothing matched
Error: failed to solve: error generating code: validation failed: packages.Load: -: no Go files in /home/dolanor/src/cloak/example/foo
#

PEBKAC

#

I output in the root cloak folder, not the example/extension one

cosmic cove
bronze hollow
# 
{
          host {
            workdir {
              read {
                id
              }
            }
          }
        }

Should this work? 😅

wild zephyr
#

It doesn't do too much magic. It only changes the path to allow using "scripts" installed by modules in the node_modules

mellow bolt
civic yacht
bronze hollow