maintainers

[bronze hollow]

An extension is a GraphQL schema and a document, with some resolvers; right?

[civic yacht]

We have a community call in 50 minutes where we'd be happy to help, but also happy to help async here too 🙂

An extension is a GraphQL schema and a document, with some resolvers; right?
Yes, that's right

Some more high-level details here: https://github.com/dagger/cloak/blob/main/docs/writing_extensions.md

[bronze hollow]

I'm struggling to generate with graphql-codegen

⚠ Generate outputs
  ❯ Generate ./gen/pulumi.ts
    ✖
      Failed to load schema from schema.graphql:
      Unknown type: "FSID".
      Error: Unknown type: "FSID".

I have your fork and I'm using:

      - add:
          content: "import { FSID, SecretID } from '@dagger.io/dagger/dist'"
    config:
      scalars:
        FSID: FSID
        SecretID: SecretID

but it isn't happy with that

I've tried without dist too

Should I use Go instead of TS for now?

[civic yacht]

At this moment, if you are using my PR's code, then I would suggest either:

1. Use JS/TS but w/ raw graphql queries (so no codegen needed).
2. Use Go
   Those are what I've tested in my PR so far

[bronze hollow]

ls

Oops

[civic yacht]

I always accidentally type vim keybinds into discord 🙂

:wq

[ancient kettle]

:q!

[bronze hollow]

😂

[wild zephyr]

haha, same

[cosmic cove]

<@&1003717314862129174> Community Call starting. See you there!

[tepid nova]

Sorry @mellow bolt we should have started with the later timezones in the call. Next time we'll do that!

[mellow bolt]

No prob, I'm back

[tepid nova]

Thanks @bronze hollow @round granite @untold shoal @scarlet gate for joining!

@civic yacht how hard would it be for the yarn extension to extend Filesystem ? Is that technically feasible today or still at the 'idea for later' phase?

[civic yacht]

It worked when I tested it 2 weeks ago, we have not made any changes that should have broke it since (🤞 ). The only issue was that the codegen clients can't really take good advantage of it yet, but it should work w/ raw graphql queries today

Can double check it still works if you're interested

[tepid nova]

It would be really neat to do:

{
 host {
  workdir {
   read {
    yarn {
      run(script: "build") {
       output {
        id
       }
      }
     }
    }
   }
  }
 }
}

Instead of having to stitch 2 different calls together with filesystem(id: ...)

the chain is longer (and that's a lot of curly braces...) but easier to explain than the filesystem() thing

[tepid nova]

Yeah definitely interested 🙂

It opens the question of namespacing, related to the "is it ok to have empty containers" question Andrea asked yesterday

Same question in top-level namespace ({ git { remote(address: ...) } } and type extensions (extend type Filesystem { yarn { build } })

(but less urgent than @bronze hollow 's problem 🙂

[wild zephyr]

I've been told the shopify graphql docs are one of the bests out there: https://shopify.dev/api/admin-graphql. I don' think it's OSS buy maybe we can see if we know someone who works there?

[tepid nova]

Question for the JS experts: is there a way for a nodejs script to retrieve the path of its own source code?

(Looking for a way to get rid of the CLOAK_CONFIG= hack)

[civic yacht]

I did this in a previous commit in my PR, I will find it, I switched to CLOAK_CONFIG because it felt less ugly that the way of retrieving its own source code

[civic yacht]

https://github.com/sipsma/cloak/blob/433479a2419f7e803c9ecc882c3b1b96c62a6987/examples/todoapp/app/workflows/build/index.mjs#L2-L7

This is another thing that varies from esm vs. cjs...

[tepid nova]

Perhaps it's uglier (debatable) but only you have to look at it 🙂

[tepid nova]

of course

[civic yacht]

I just don't like it because it's unavoidable boilerplate in the code (can't really be farmed out to and hidden within the sdk), but yeah it's subjective

[tepid nova]

Actually I take that back. Either way it is visible in the demo

I had missed the part where you can't move it into the SDK, which makes sense

So far the least dirty solution is to embed it in the code with cloak generate IMO

OK ignoring that one for now

[civic yacht]

It's that or move cloak.yaml to be next to package.json and shared, but that's the whole other thread we're ignoring for now

[tepid nova]

Yes that may be where we end up for sure

OK I have another suggestion for short-term solution

How about cloak generate drops an executable entrypoint, say at WORKFLOW_DIR/run.sh which just does:

export CLOAK_CONFIG=$(pwd)/cloak.yaml
node index.mjs

So in my package.json I only need to do "foo": "./workflows/FOO/run.sh"

(we could rename run.sh to just run)

It could be a cross-SDK convention too

Which you can bypass at will if you know what you're doing

If we switch to project-centric it may turn out to be unnecessary and we can drop it

[civic yacht]

You could also put something like #!/usr/bin/node at the top of index.mjs I think

[tepid nova]

but it may remain useful either way

[tepid nova]

Yes but then we're missing the CLOAK_CONFIG var

[tepid nova]

Correction: not $(pwd) but whatever hack bash has available to find its own script's path

Honestly this is the strongest argument for project-centric... Not having to deal with this stuff

[civic yacht]

I see, yeah I think it would work, I'm just concerned at the complexity. cloak.yaml is loaded into buildkit as part of the workdir, so the script has to look at its workdir and find the path of cloak.yaml relative to that. Not actually hard to implement, it just makes my head hurt atm. Also, I just realized that if your cloak.yaml wasn't a subdir of your workdir, then it gets even more complicated.

I'll think about it some more once I finish the current fix (and after the demo at 1)

[tepid nova]

@civic yacht where can I get your modified yarn extension ? 🙂

[civic yacht]

https://github.com/sipsma/cloak/blob/07ed249a8630ad9865866ae55e56184bd1f0bbb3/examples/todoapp/app/workflows/build/cloak.yaml#L2-L5

[tepid nova]

https://tenor.com/view/ifht-carbon-dealer-mtb-mountainbike-gif-20969945

[civic yacht]

https://tenor.com/view/hellmo-elmo-yarn-gif-24795700

[bronze hollow]

I always like to get the monorepo argument in early too, what about multiple build contexts in a work dir for each cloak file?

The cloak file might need a dependencies key. If all my cloaks are node, I can import dependant jobs- but multi language means we need a platform way to define those too?

[tepid nova]

Related: https://github.com/dagger/cloak/blob/main/demos/v2/demo-v2.md#project-centric-or-workflow-centric

We have two viable options for where the cloak.yaml should live, and what the scope of each file should be. We have not yet decided.

Either way there is definitely a dependencies key.

Common terminology for both options @bronze hollow :

• You can use a Dagger SDK to write either a workflow (client code) or an extension (server code).
• A workflow may depend on extensions
• An extension may depend on other extensions

See also this diagram: https://github.com/dagger/cloak/blob/main/docs/introduction.md#running-a-workflow

[bronze hollow]

If I have a Go workflow that has a file output and I need that in a TS workflow, what does that look like? Does each need named and referenced in GQL?

Thanks. I’ll read the links 😀

[tepid nova]

You can't integrate workflows with each other. You need at least one of them to become an extension

If you need to compose your Go and TS workflows, that is a sign that you need to move logic into an extension. Then composition happens via the cloak API

[bronze hollow]

Hmmm

[tepid nova]

--> https://github.com/dagger/cloak/blob/main/docs/introduction.md#writing-an-api-extension

[bronze hollow]

It’s quite common to have an isolated service to generate types from protos and consume from many services

I can’t think what that looks like here yet

Need to kick more tyres

With single language I wrap the generation in a function and each workflow consumes, but multi language gets weird. Maybe single language is my bet

Oh, the penny just dropped about your extension idea. I’d have a compileProto extension and pass it a path

Depending on the context that might work

[tepid nova]

HOLY 💩 @civic yacht I just copy-pasted your custom yarn extension in my cloak file, ran cloak dev, read the docs, figured out how to use, and boom. Amazing...

[tepid nova]

I think that would be an extension that has a dependency on many other extensions, and acts as middleware to them

It shouldn't need to do any custom generation though

That pattern may be redundant, since cloak itself takes care of this aggregation for you

so maybe the need for that service just goes away @bronze hollow ? Not sure

[bronze hollow]

Perhaps if there’s a cloak.yaml with a “root” annotation (like editorconfig) and every subsequent cloak.yaml becomes its own build kit context, cloak can handle the rest for me and I never need to specify a context again?

If the child cloak has a dependency, both contexts are loaded

Maybe I’m using the wrong terminology

This was a nice thing with 0.2, core. Source

[tepid nova]

Yeah definitely. Extensions will get an equivalent of core.#Source I think it will not be a problem

Client code (workflows) are TBD because cloak doesn't run them: you run them on the host with your native tools. So we have less control (hence the discussions of various hacks to figure out where the damn files are)

Basically everything is simpler in an extension. But extensions have a bit of a learning curve so we need client-side scripts to have a great DX too

[bronze hollow]

If I run “ts-node workflow.ts` how does buildkit get the local source?

[tepid nova]

you have to call { host { workdir { read } } } which loads your current directory into the engine

[bronze hollow]

I haven’t gotten that to work yet. I’ll try again next week

Thanks for answering my questions 👍

[tepid nova]

Anytime 🙂

[scarlet gate]

You can try something like that: https://stackoverflow.com/questions/3133243/how-do-i-get-the-path-to-the-current-script-with-node-js

[wild zephyr]

👋 I was setting up cloak locally for the first time and I realized that performing cloak dev actually has to go through the process of building each extension. Just wondering if we thought about doing that in a lazy manner. Couldn't find any issues about that.

It kind of hurts the DX to run cloak dev and having to wait for all the extensions to be ready the first time 🙏

[north jay]

Or a cache server ala go module proxy (note I haven't actually run cloak yet)

[civic yacht]

Yep it should be trivial to switch the current sync loading of extensions to be lazy (it might literally be a one line change actually). The tradeoff is between waiting for cloak dev to start or waiting for the extension to build once you submit a query that targets the extension. But either way, the idea is that buildkit's caching means you only have to wait when the extension actually changed and needs a rebuild (especially nice once integrated with remote caching). I went with the sync approach for now because I like knowing whether the extension build succeeds or fails before I try to invoke it, but that's particular to me as a developer and a demo-er. I think we could make the behavior configurable pretty easily too.

[civic yacht]

We actually just get all the buildkit caching features for free since extensions are themselves just built using buildkit; cloak is kind of the ultimate dogfooding framework in that respect 🙂

[tepid nova]

OK I'm at the deploy with go part...

[civic yacht]

Awesome

1. Implementation that works for me is here: https://github.com/sipsma/cloak/blob/07ed249a8630ad9865866ae55e56184bd1f0bbb3/examples/todoapp/app/workflows/deploy/main.go
2. go.mod has a replace directive you'll need: https://github.com/sipsma/cloak/blob/07ed249a8630ad9865866ae55e56184bd1f0bbb3/examples/todoapp/app/go.mod#L7-L9
   • I just realized I pushed my local replace, that will work for you, but for the actual demo you'll want to use the normal github.com/dagger/cloak repo. This is a private repo though and go makes it extremely annoying to pull it. I got it to work though, I'll send the instructions after this.
3. To generate the client stubs you can do cloak generate -p workflows/deploy/cloak.yaml --sdk=go --output-dir=workflows/deploy/ --client
   • (TODO: update cloak generate to remove the need for --sdk=go, probably can also make the output-dir flag optional)

Oh and you'll need a netlify token too of course. I just read it from an env var in mine but whatever works: https://github.com/sipsma/cloak/blob/07ed249a8630ad9865866ae55e56184bd1f0bbb3/examples/todoapp/app/workflows/deploy/main.go#L19

When you want to specify a private git repo in your go.mod you have to:

1. export GOPRIVATE=github.com/dagger/cloak (or whatever the repo is)
2. Add something like this to your .gitconfig:

[url "ssh://[e-mail]/sipsma/cloak"]
  insteadOf = https://github.com/sipsma/cloak

Which is incredibly annoying, and hopefully there's a better way of doing this, but that's the instructions I found on first google and that worked for me

[tepid nova]

That's going to be a huge PITA for all Go SDK devs until the cloak repo is public, right?

maybe we could have cloak generate drop an vendored version or something?

@civic yacht so for now, export GOPRIVATE=github.com/sipsma/cloak ?

[civic yacht]

Yes, that for now

[tepid nova]

btw how does it find the right branch?

[civic yacht]

maybe we could have cloak generate drop an vendored version or something?
Yep I can't think of anything better than that. It should work pretty well actually, can just run go mod edit in the sdk generate container.

[civic yacht]

I just paste the commit ID after the repo and then run go mod tidy and it does it automatically

[tepid nova]

@civic yacht what's the minimum sequence of commands to get my new go workflow to run?

• cloak generate
• go run ?

So far I have this:

% yarn run deploy
yarn run v1.22.19
$ GOPRIVATE=github.com/sipsma/cloak go run ./workflows/deploy
workflows/deploy/main.go:4:2: no required module provides package github.com/sipsma/cloak/engine; to add it:
        go get github.com/sipsma/cloak/engine
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[civic yacht]

Yes, provided cloak generate takes care of all the annoying details above, then that would be it

[tepid nova]

(I stuck the GOPRIVATE=... in the package.json)

[civic yacht]

I think that just means you have to run go get ... or I usually just run go mod tidy and it takes care of everything. That's the part where you need GOPRIVATE=... actually

(at least until we stick this in cloak generate)

[tepid nova]

Ah ok so:

1. cloak generate
2. go get
3. go run?

[civic yacht]

At this moment yeah. 2. should be merged into 1. though. I think I will have time tomorrow to work on that

[tepid nova]

cloak generate isn't doing anything:

% cloak -p ./workflows/deploy/cloak.yaml generate
#1 local://.workdir
#1 transferring .workdir: 497.59kB 0.1s
#1 transferring .workdir: 1.69MB 0.3s done
#1 DONE 0.3s

#2 copy / /
#2 DONE 0.0s

(no files written)

[civic yacht]

cloak generate -p workflows/deploy/cloak.yaml --sdk=go --output-dir=workflows/deploy/ --client

[tepid nova]

Feels like --sdk and --output-dir could be inferred from -p?

and --client should just default to true

[civic yacht]

Yep, I mentioned above (TODO: update cloak generate to remove the need for --sdk=go, probably can also make the output-dir flag optional)

[twin crow]

I did the generate and the git config, here is what I get (using the main.go from your branch Erik)

[tepid nova]

it worked though 🙂

(generate I mean)

[twin crow]

looks like git still does not use ssh

the generate works on my side as well

[tepid nova]

I have a variation of the same problem

[civic yacht]

--client defaulting to true sounds good too, that was a quick and dirty change to add --client and --impl, can clean it up more

[tepid nova]

 % go get ./workflows/deploy
github.com/dagger/todoapp/workflows/deploy imports
        github.com/sipsma/cloak/engine: cannot find module providing package github.com/sipsma/cloak/engine

[civic yacht]

Hm okay let me go run it again on my side, one sec

[twin crow]

nevermind, the go module did not have the right name

it's all good now

I mean, better

[civic yacht]

do you have GOPRIVATE=github.com/dagger/cloak exported?

[tepid nova]

sipsma/cloak right?

I'm getting confused on which one

[twin crow]

same error

[civic yacht]

(starting thread)

[wild zephyr]

Well... the counter argument against always building upon running cloak is that we're pulling in dependencies and API's that will not be used in any workflows and since there's no way to know which extensions could be potentially used like we have in Dagger Europa, I guess a "safe" default would be not to build them on-demand and let it happen upon invocation time. In any case, we can make this configurable as you suggested

[wild zephyr]

shouldn't go run do the get part automatically? (can't join audio, baby background noises :D)

[twin crow]

@civic yacht @tepid nova 🥳

@civic yacht btw if you merge your PR #95, it will break the todoapp. If you could keep your branch on your fork, in case you merge 🙏

[civic yacht]

Oh for sure, I'll let you know when I'm merging it all and we can coordinate

[tepid nova]

Quick demo update, we're going to tweak the demo v2 script (let's call this version 2.1 🙂 to start with a more realistic scenario. Overall structure remains the same.

1. Write a simple deploy workflow, the hard way. Orchestrate yarn and netlify from a JS workflow.
2. Deploy the easy way: use the yarn and netlify extensions. Massively simpler code.
3. Write the same deploy workflow, in Go. Demonstrate multi-language + generated client stubs.

Steps 1 and 2 are different. Step 3 is the same.

[civic yacht]

That will probably fit into the time slot better and even though build is cool it's too meta

(for now)

[tepid nova]

As a bonus, we can start with a non-containerized version of the script. Hopefully it will be worse than the containerized version in some way: slower, harder to read.. something

[twin crow]

same yarn / build / netlify deploy in javascript, I am pushing the workflow now

(it's the easy way, using Erik's netlify extension in Go)

https://github.com/dagger/todoapp/blob/cloak-demo-v2/workflows/deploy-easy/index.mjs

[wild zephyr]

graphiql v2 looks dope (https://graphql.org/swapi-graphql). Has been released yesterday.

Looks interesting as the replacement of the current playground 😊

[scarlet gate]

I created a new app in the example folder. Now I wanna execute the cloak generate command.
Here I use your command in this way: cloak generate --output-dir=examples/radar/ts context . -p examples/radar/ts/cloak.yml --sdk ts

This gives me that back

#1 transferring .projectContext: 60.72kB 0.1s
#1 transferring .projectContext: 79.51kB 0.1s done
#1 DONE 0.1s

#2 copy / /
#2 CACHED
Error: failed to solve: input:5: core.filesystem.loadExtension open /tmp/buildkit-mount2903404526/examples/radar/ts/cloak.yml: no such file or directory

exit status 1

As you can see on my screenshot everything should be there.

[tepid nova]

🤔

[scarlet gate]

@civic yacht & @tepid nova I have seen in the examples the graphql queries (in TS) which are quite straight forward. But if they become more complex it will be more difficult to write them correctly. What do you think about a little lib that helps to write queries more easily?
Something like that: https://github.com/munichbughunter/graphql-client-js
I already tested it in a easy way: https://github.com/munichbughunter/graphql-client-js/blob/main/__tests__/GqlStatement.test.ts

[civic yacht]

For the cloak.yml: no such file or directory I think it's because your file is named cloak.yaml but you are specifying cloak.yml in the command line.

Another side thing: you can leave out --sdk ts from that command for now. The codegen for ts is half baked at the moment which is why it's currently left out of the instruction: https://github.com/dagger/cloak/blob/main/docs/writing_extensions_typescript.md#writing-an-api-extension-in-typescript

If you leave out --sdk ts you still will get schemas written to gen/ in the output-dir, but you'll still have to make some config files and run yarn generate yourself. Once this issue is taken care of we can make the ts codegen actually nice-to-use: https://github.com/dagger/cloak/issues/126

The codegen for go is in a much more usable state (since it's pretty much a necessity for go).

For the cloak.yml: no such file or directory I think it's because your file is named cloak.yaml but you are specifying cloak.yml in the command line.
For this, I think we should make it possible to have -p just point to a directory holding cloak.yaml, in which case you can leave out writing the full filename. You could specify the full filename too of course, but that tiny amount of sugar might make small typos like this harder to run into

[tepid nova]

Yes please 🙏

[civic yacht]

Neat! So if I'm understanding correctly this is sort of like a query builder? We have been thinking about something like this a bit; the current graphql codegen tools we use are okay but something like a codegen'd query builder could be much more powerful

[scarlet gate]

Yeah correct. I don't wanted to build up the queries by hand. So I wrote a little helper for me.
And if I see that right with small changes types.ts and we can use that for cloak.

[wet mason]

Nice! Yes graphiql and playground joined forces a while ago, playground is deprecated now. We should switch

[wet mason]

Perhaps even in the dependencies config, point to the directory rather than the file?

[tawny flicker]

I had a similar headache when I did my hack to embed .so library in a Go binary to be able to have a fake "static" binary, with no external deps.
My solution was to write the embedded .so to a temp dir, and then load it from here. Ugly, hacky, but it works.

code: https://github.com/dolanor/sobed/blob/main/dlopen.go#L41-L83

I've heard about a tool that can help alleviate the pain. You can install those utility into something it calls a "container". And from there, you can run your tools without polluting your host.
It's pretty fantastic if you ask me.
It's called Docker. They have been around a while.

[hasty basin]

also this! https://github.com/dagger/cloak/issues/125

[tawny flicker]

Yes, it does, for a while now, if you have your go.mod already there

So FYI, @round granite demo the other made me want to test the same way.
So I did a leftpad extension instead of a ToUpper.
So, if I don't make the CI ecosystem better, at least, I can break it like any other else. Which is something, I guess. 😅

testing/checking @civic yacht PR for automated buildkitd start

no, first I'm gonna submit my typo/fix found during my dagger ramp up

[mellow bolt]

@civic yacht do I have access right if I want to use [e-mail]:sipsma/cloak.git ?

I get the following error :

#3 0.094 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2287/fs/
#3 0.796 [e-mail]: Permission denied (publickey).
#3 0.797 fatal: Could not read from remote repository.
#3 0.797 
#3 0.797 Please make sure you have the correct access rights
#3 0.797 and the repository exists.
#3 ERROR: failed to fetch remote [e-mail]:sipsma/cloak.git: exit status 128
------
> git://github.com/sipsma/cloak.git#workflow-clean:

[dense dust]

Same here

[mellow bolt]

ok It works for me know...

just did an eval and ssh-add

maybe try this @dense dust

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/your_rsa_key

[dense dust]

Thanks!

[wet mason]

Oh no actually you should use npm to install yarn 🤯

Same dance as easy_install pip in Python

npm ships with node (just like easy_install ships with python). yarn is a faster (and compatible) 3rd party package manager by Facebook which you can install with npm (like pip)

I’m guessing that custom apt repo comes with its own nodejs, that’s the gigantic list of dependencies you’re seeing (yarn is a JS program)

Note for later: we should probably retool our demos around npm instead of yarn

yarn is much better than npm, but we shouldn’t make that decision in example code (people familiar with yarn upgraded from npm, so basically everyone knows npm)

React for instance defaults to npm but if it detects yarn is installed, it’ll use it instead

Pulumi shows both npm and yarn commands in their docs (npm first)

/cc @mellow bolt @dense dust
/cc @civic yacht @tepid nova

[dense dust]

Yep, that's the rule of thumb generally

I'm currently doing that with async/await and try/catch blocks in my fork

Right now, this is working. /todoapp/workflows/build/index.mjs. Just missing building and writing the output.

import { gql, Engine } from "@dagger.io/dagger";

new Engine({
    ConfigPath: process.env.CLOAK_CONFIG
}).run(async (client) => {
    // 1. Load app source code from working directory

  try {
    const sourceCode = await client.request(gql`
      {
        host {
          workdir {
            read {
              id
            }
          }
        }
      }
    `);

    console.log(sourceCode);

    const image = await client.request(gql`
      {
        core {
          image(ref: "index.docker.io/alpine") {
            exec(input: { args: ["apk", "add", "npm", "git", "openssh-client"] }) {
              stdout
              fs {
                id
              }
            }
          }
        }
      }
    `);

    // console.log(image.core.image.exec.fs.id);

    const installDeps = await client.request(gql`
        {
            core {
                filesystem(id: "${image.core.image.exec.fs.id}") {
                    exec(input: {
                            args: ["npm", "install"],            
                            mounts: [{path: "/src", fs: "${sourceCode.host.workdir.read.id}"}],
                            workdir: "/src"
                         }) {
                        stdout
                    }
                }
            }
        }
    `);

    console.log(installDeps.core.filesystem.exec);
  } catch (err) {
    console.log(`error: ${err}`);
  }

});

[wet mason]

Another note for later: I see there are “accepted” ways for npm packages to ship binaries

We could consider for the cloak node SDK to ship with the cloak API binary. So in order to get started you only need to add the sdk as a node dependency, no other setup required

/cc @tepid nova @civic yacht

[dense dust]

If you only have to run npm install <package name> without building anything from source, you greatly improve DX IMO

[wet mason]

Oh btw — regarding .mjs— we’re doing that because of some problems we had with “execa”

We couldn’t get commonjs imports working with execa

@mellow bolt and @civic yacht have the context

Basically in the SDK, in engine.ts, we’re exec’ing “cloak”

However we didn’t manage to get it to work with the default tsconfig.json

[mellow bolt]

There is a PR for the tsconfig

[dense dust]

Yep! I joined the call yesterday, we are working on it 🤞

[wet mason]

Oh amazing

[mellow bolt]

https://github.com/dagger/cloak/pull/127

[wet mason]

I’m on PTO and didn’t check the latest developments 🙂

[dense dust]

No worries!

[wet mason]

I’ll be back on Monday

Anyway, happy to see so much progress 🙂

[wet mason]

Yep you need the ssh agent running

[mellow bolt]

yep it works now

[wet mason]

Basically you should be able to git clone the repo from the same terminal you’re running cloak. If that works then cloak will also work

[mellow bolt]

So happy to be able to contribute to Dagger itself and not just front-end side 🙂

[tawny flicker]

I mean, without the front-end side, dagger is nothing either

I mean, nobody would know about it, so close to nothing.

It's like the best blacksmith of the country is worth nothing if nobody knows about it.
You need both.

[civic yacht]

@tepid nova @twin crow pushed some updates, major one being automating the go.mod setup nonsense. The process now looks like:

1. From todoapp root, run go mod init github.com/dagger/todoapp.
   • It's possible to include this as a step in cloak generate too, but it feels awkward since we can't automatically determine the name of the module. We'd need a way of passing it to cloak generate which would just transparently pass it to go mod init. Also not needed for any project that already has a go.mod. Let me know if you disagree though, it's trivial to implement either way.
2. Run cloak generate -p workflows/deploy/cloak.yaml --workflow
   • This will run go mod edit, get, tidy , vendor etc. with GOPRIVATE env setup to get you a working go.mod. When everything is merged into main, will update this to skip the replace of github.com/dagger/cloak w/ github.com/sipsma/cloak
   • It also writes a main.go file with a skeleton implementation to be filled in.

The only caveat remaining, which I think is fine for the demo since it can be setup beforehand, is that you still have to configure .gitconfig. I could not find a way to configure that w/out editing .gitconfig and automating changes to that file would have been in "egregious, un-mergeable hack" territory for me. For the long term, this will be addressed once we move cloak generate to actually run in containerized extensions as part of the overall SDK concept: https://github.com/dagger/cloak/issues/126

Also merged a fix for the problem David run into yesterday and some other minor things (e.g. I use yaml.UnmarshalStrict now to improve that yaml indentation situation yesterday)

I think the biggest remaining wart is the CLOAK_CONFIG hack, so gonna do what it takes to squash that. Then just some other minor cosmetic issues we've brought up.

Also, once that's done, I'll start disentangling and merging commits from the PR (somehow have 22 in there now...). Will coordinate as needed. After skimming through it again, there's only one truly egregious hack that is so bad I have to fix it before merging into main: https://github.com/sipsma/cloak/blob/a2fc1db433542e80b197cd15cd9aef6ae17559a5/core/exec.schema.go#L159-L173

Fortunately the fix is pretty easy and just involves adding more features to Filesystem.exec that we'll want in the long-term anyways. So shouldn't be that bad to get all the new demo features main-lined soon

[twin crow]

nice, I will try this today. It should be much easier indeed with those helpers.

[tawny flicker]

What about the dependabot PRs? Should we just merge them? They don't seem to break things we use (after reading the changelog of each of them).

[tepid nova]

Thanks @civic yacht !

We should consider removing the todoapp content altogether

(from the cloak repo)

[civic yacht]

Yes should be fine, utterly worst case scenario they are easy to revert, thanks for looking at that! Been in my backlog for a bit now

[civic yacht]

Yeah I agree, makes much more sense in the separate repo now

[tepid nova]

Bonus is that’s probably what the majority of dependabot PRs are for

[civic yacht]

Trying to fix CLOAK_CONFIG hack and going back to the .sh wrapper. After thinking more, I still am hesitant because this is complexity that the user has to be aware of (they have to know to invoke the .sh script). I like that right now they can just do what should be natural to them (js devs call node ..., go devs can use go run or go build if they want a standalone binary).

With the script wrapper we have to explain an extra step to users and when I imagine trying to answer the question "why do I have to use this wrapper script" I just imagine their eyes glazing over as we explain these minute technical details about workdir vs config file vs source code path, etc.

I am just going to give a shot at the project centric approach and see what it looks like. Hoping it only takes a few hours so we can decide what to do later this afternoon. Will keep the commit self-contained so it's easy to revert if we change our minds in the future too.

[dense dust]

I managed to make the "hard way" build work with npm. it's basically the same as yarn.

The thing is, is it normal for the npm install query to not be cached if it doesn't fail?

[civic yacht]

In general, things should be cached if none of their inputs have changed at all, so depends on the exact details here

[hasty basin]

@dense dust what does your code look like?

[tepid nova]

Sounds good

Since we're all collaborating on todoapp now... Maybe we should make it a standalone repo instead of a fork

Will make it easier for everyone to have their own forks, open PRs etc

@civic yacht how does project-centric remove the need for CLOAK_CONFIG=? We'll still have the fundamental problem of the code being runnable from any directory, outside of our control

I have a feeling the only truly clean solution is to not require reading a config file at all...

(at runtime I mean)

Otherwise there will always be a situation where we need contorsions to find the location of a config file, which is highly language-specific and just generally annoying

[dense dust]

https://github.com/crjm/todoapp/blob/cloak-demo-v2-npm/workflows/build/index.mjs

It cached last time I ran though

But now I ran again and didn't cache, weird.

[tepid nova]

@civic yacht We're discussing step 4 of the demo (todoapp extension). We need to choose either JS or Go. Any thoughts on which one has a better DX at the moment?

[civic yacht]

If you choose JS, you will have to do raw queries, codegen is half-baked for js/ts right now. Go requires that you do codegen. Other than that I think they are both fine. If I was forced to choose, I'd do go

[tepid nova]

Starting a demo-v2 thread to reduce noise..

[bronze hollow]

Interesting: https://stepzen.com/blog/sequence-directive-sunrise-api

Who needs code when cloak could implement its own directives like this 😀

[scarlet gate]

Sounds interesting

[civic yacht]

Super interesting, they basically enable arbitrary chaining by matching resolver output fields -> resolver input args. End effect is sort of like a pipeline of functions for mapping json blobs. Requires very careful coordination of field and arg names I guess, but the general approach is intriguing nonetheless.

Directives in general are almost limitless in power (both a good and bad thing). I think we've previously briefly discussed something like being able to annotate fields as a named result (@result("foo")) that could be referenced later somehow. But it just felt like creating a sort of scripting DSL on top of graphql, at which point it becomes more like "just another language" rather than "the one language to rule them all". But that doesn't mean we should rule out neat DX things similar to this pipelining idea, especially if it could also be useful to codegen clients built on top of gql in addition to the "raw gql" experience too.

[wild zephyr]

interesting indeed. One "neat" thing about the graphql ecosystem could potentially be that if cloak allows injecting/registering this crazy directive resolvers dynamically (as long as they're written in Go I guess?). This way if someone has a very specific workflow requirement (like steps in this case), they could inject or create their directives and plug them directly through the Cloak CLI.

[scarlet gate]

Yes... @civic yacht my first cloak example is running. Nice...
Thank you for your help.

[scarlet gate]

Next step improve our Salesforce deployment with cloak...

[proven rock]

Hey folks! Much later than anticipated (life got in the way), I've done a writeup of the caching model differences between Dagger/Nix I was talking about before. I expect most of this to not be news for anyone, but IMO it is useful to write it down and straighten out any possible misunderstandings: https://github.com/dagger/cloak/discussions/135

This has huge implications for where both of the systems fit in the overall stack, and how they can interact.

[cosmic cove]

Keep us updated if you need anything for that evaluation! Also, would love to see a demo of what you got running at the next community call this week!

[ancient kettle]

Something we talked about with dagger was dagger undo. Have we given any consideration to the same idea in cloak, @wet mason @civic yacht @tepid nova?

[tepid nova]

Welcome to project cloak @hybrid widget 🙂

[civic yacht]

My immediate thought is that this probably would not be baked into cloak specifically. Whether there would be the equivalent of undo would be up to each extension as it depends heavily on what each extension is doing. E.g. netlify could definitely have a both deploy and undeploy, but yarn wouldn't want build and unbuild since unbuild doesn't really make sense in that context.

[wild zephyr]

👋 just some flying by feedback here. Without having any context, at a first impression reading about dagger undo gives me no idea about what the command could potentially do. Reading Erik's answer I can imagine what we're talking about but the UX still quite doesn't feel right to me. Specially since dagger doesn't have any context of state, so by doing undo each extension+action combo would have to define what undo actually is?

[wild zephyr]

@civic yacht this weekend I came across TSX (https://www.npmjs.com/package/tsx). Seems like another typescript node executer alternative to ts-node. Just bringing this up in case you haven't seen it

[ancient kettle]

Some good examples:

• terraform apply vs terraform destroy
• cloudformation deploy vs cloudformation delete-stack

I’d say it mostly is useful in the context of infrastructure, rather than build, as noted by @civic yacht.

It’s about undoing or destroying some thing that was created by the do cmd.

[cosmic cove]

<@&1003717314862129174> based on the notes from last week's meeting, it looks like the majority thought it would be a great idea to start recording our community calls. Therefore, I have created an official calendar invite, invited all of you, and added a zoom link that we will use from now on. The notes are also accessible to everyone through the calendar invite.

@hasty basin @twin crow @tepid nova I made you all alternative zoom hosts, so you can record or lead if I am not around. I did set it to record automatically though, so you shouldn't have to worry about it 🙂

[tepid nova]

In parallel to the demo we can start planning a few contribution streams that could be started in parallel to the core work

[scarlet gate]

I will not attend the next two community calls because of vacation. About what I did is pretty simple.
We have a tech-radar in our organization. It is a simple JS application which has easy commands in the package.json file.

 "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject",
    "lint": "./node_modules/.bin/eslint src"
  },

First I started with the build to get everything working. Deployment is not possible because of my rights on our kubernetes.

[obsidian rover]

As I'm ramping up on Cloak: MIND BLOWN 🤯

Dynamic actions on top of normal languages, GraphQL as a representation of DAGs is so intuitive.
The imports of packages in the playground, the types directly accessible, the fast feedback loop. Side effects are easy, OMG

[cosmic cove]

ah yes, have a wonderful vacation! And thanks for the summary!

[dense dust]

Demo v2 is now TS compatible
https://github.com/crjm/todoapp/tree/cloak-demo-v2-ts
Run yarn run deploy-1 or yarn run deploy-2 (executes ts-node-esm) to have JIT support for /scripts/{deploy-1|deploy-2}/index.ts.
The main problem is that the Netlify package (https://github.com/netlify/js-client) does not have type definitions, so we either have to create our own types for the package or change the compiler restraints (strict: false in tsconfig.json) . The latest was chosen as it takes considerably less time to implement, but we should avoid this in the future, as it omits many checks the compiler does.
(thanks for the help @mellow bolt )

[ancient kettle]

Ok... I'm fairly confused now. I'm trying to create a new go project (dagger-classic), and attempting to use dagger/cloak: go get github.com/dagger/cloak

go: downloading github.com/dagger/cloak v0.0.0-20220829171855-01ea736a9c09
go: github.com/dagger/cloak@v0.0.0-20220829171855-01ea736a9c09: verifying module: github.com/dagger/cloak@v0.0.0-20220829171855-01ea736a9c09: reading https://sum.golang.org/lookup/github.com/dagger/cloak@v0.0.0-20220829171855-01ea736a9c09: 404 Not Found
    server response:
    not found: github.com/dagger/cloak@v0.0.0-20220829171855-01ea736a9c09: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/b46a797880c1419bb8db03bbea7898d15401d1ab8150416524436c67cd3f4e11: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
    Confirm the import path was entered correctly.
    If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

I keep getting that error. I've tried the both of the methods here: https://go.dev/doc/faq#git_https

.netrc + .gitconfig

Am I missing something obvious?

[ancient kettle]

@tawny flicker Do you know what I'm doing wrong? 🙂

[ancient kettle]

This helped: go env -w GOPRIVATE=github.com/dagger/cloak

[wild zephyr]

yeah.. the whole "pulling go modules from private repos" UX is horrible 😝

[tepid nova]

Big day for Cloak tomorrow! We're showing a live demo in the expo hall at VMware Explore in Moscone Center, around 10am Pacific. We'll share links here!

[tepid nova]

Thinking about @tawny lion ‘s feedback on cloak yaml… about it being weird to have more than one extension

It makes sense but I couldn’t find the right field to express a single place to define your own graphql api…

well I think it’s just that: api 🙂

scripts:
 - …
 - …
dependencies:
  - …
  - …
api:
  path: …
  sdk: …

Welcome @tawny lion by the way 🙂

[tepid nova]

https://twitter.com/solomonstre/status/1564855297855741953

Solomon Hykes (@solomonstre)

Tomorrow I will be presenting at #VMwareExplore DevOps Loop. See you at the pop up theater, 11am. There will be live demos of unfinished code lovingly prepared by the @dagger_io team. Rumour says the one and only @BettyJunod may even be attending 🤞

Feels like old times!

Twitter · Aug 31, 2022, 5:59 AM

[scarlet gate]

Running yarn run deploy-2 works.
But when I run yarn run deploy-1 I get an error:

yarn run v1.22.15
$ ts-node-esm scripts/deploy-1/index.ts
Error: spawn cloak ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:477:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn cloak',
  path: 'cloak',
  spawnargs: [
    'dev',
    '--workdir',
    '/Users/TestLab/private/myPlayground/radarapp',
    '-p',
    './cloak.yaml',
    '--port',
    '8080'
  ]
}

I guess it comes from this line const engine = new Engine({ ConfigPath: process.env.CLOAK_CONFIG }); right?
What should I set for CLOAK_CONFIG?

[tepid nova]

you don’t need that config key anymore

i recommend pulling the latest cloak-demov2 branch of todoapp it has moved a LOT in the last 24h

[scarlet gate]

I am on the latest cloak-demo-v2-ts branch.

[mellow bolt]

hey @scarlet gate . You should be using this branch https://github.com/dagger/cloak/tree/demov2-branch

then you can get rid of the Engine param like so: const engine = new Engine();

[tepid nova]

Not sure what that branch is, sorry about the confusion

[scarlet gate]

I will have a look on that after my vacation.

[obsidian rover]

Questions:

• the clientdir core query is not working atm ?
  I like the fact that exec is a core type directly not exposed at root, even though it surprised me at first

[dense dust]

I actually solved this one by building cloak from source, from the workflow-clean branch instead of main. It seems that some args are not compatible with the cloak version in main

[wet mason]

@obsidian rover it is, however it takes an ID and you need to map ID in localdirs when you do an engine.Start

[obsidian rover]

cloak dev -p examples/test_guigui/cloak.yaml --local-dir src="."
with query:

query {
  core {
         clientdir(id: ".") {

returns -> "NotFound: no access allowed to dir \".\""

[obsidian rover]

(all from graphQL playground)

[obsidian rover]

Ooooh, found out! So, the current way to mount in GraphQL playground is to rely on the CLI with the --local-dir, that maps the given ID to a string ID. As it shall also be a graphQL query, let's see how to do it manually

[wet mason]

@obsidian rover Yeah, there’a an option in engine.Start if using through the API

Local directories must be declared in advance in buildkit

[obsidian rover]

I thought this exposition was done with the loadLocalDirs func, and that we could add this key/path using a GraphQL query inside the playground

I don't see where this buildkit declaration happens in our code, exploring; Thanks 😇

[obsidian rover]

Ok, my bad. Found it: it's a config of the engine, thanks 😇

• If i may, what is a shim ? its purpose

[civic yacht]

It's a binary we inject into every ExecOp as the entrypoint. It lets us capture stdout/stderr (which is why those are available to be selected in the graphql output) without the user having to manually specify a command that redirects those to a file. In the longer term it could also be used to provide stdin to processes, hypothetically maybe some network proxying features, etc.

[cosmic cove]

<@&1003717314862129174> just a reminder to all that we are moving to zoom tomorrow for the community call, and will be using Google Docs for community notes. You should all have a direct invite on your calendars, which is where you will find the zoom link. Please let me know if you have any demos or topics that you'd like me to add to the agenda!

[scarlet gate]

Does it make sense to have a creation dialog similar to npm init when creating a new dagger project? What do you think?

@cosmic cove is it possible to get a recording from the VMExplore and Solomon’s demo?

[cosmic cove]

Yup! They will have a recording as soon as the demo is done, so I'll share it here when it is ready 🙂 He will do a 20 minute demo and then 20 min podcast interview

[civic yacht]

Yeah I think that could be a great thing to add in time. It could be part of cloak generate. Each SDK could optionally vend templates that get you started, with an interactive menu for configuring them 👍

[scarlet gate]

Yeah. That would be awesome. For JS devs especially. 😁 Should I open a feature request in GitHub?

[tepid nova]

I’m curious about Backstage and how cloak might integrate with it. Does it have an API? Does anyone have experience with it?

[civic yacht]

Should I open a feature request in GitHub?
Yes, please do!

[hasty basin]

Her talk was great.

some plugins

Features of the CircleCI backstage plugin: https://github.com/backstage/backstage/tree/master/plugins/circleci#features

[cosmic cove]

<@&1003717314862129174> We just completed the Cloak demo at VMware Explore. You can check out the live demo and interview here -https://www.twitch.tv/videos/1578035654

[ancient kettle]

@wet mason I'm having some more issues with dagger-classic... 😢

module github.com/dagger/dagger-classic

go 1.19

require github.com/dagger/cloak v0.0.0-20220831001332-9250e5d71e38

package main

import (
    "fmt"

    "github.com/dagger/cloak/engine"
)

func main() {
    if err := engine.Start(ctx, &engine.Config{}, func(ctx engine.Context) error {
        fmt.Println("It has begun")
    }); err != nil {
        panic(err)
    }
}

When I try to run go mod tidy...

$ go mod tidy
go: finding module for package github.com/docker/libnetwork/ipamutils
go: found github.com/docker/libnetwork/ipamutils in github.com/docker/libnetwork v0.5.6
go: finding module for package github.com/Sirupsen/logrus
go: found github.com/Sirupsen/logrus in github.com/Sirupsen/logrus v1.9.0
go: github.com/dagger/dagger-classic imports
    github.com/dagger/cloak/engine imports
    github.com/dagger/cloak/core imports
    github.com/moby/buildkit/frontend/dockerfile/builder imports
    github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb imports
    github.com/moby/buildkit/frontend/dockerfile/instructions imports
    github.com/docker/docker/opts imports
    github.com/docker/libnetwork/ipamutils imports
    github.com/docker/libnetwork/osl imports
    github.com/Sirupsen/logrus: github.com/Sirupsen/logrus@v1.9.0: parsing go.mod:
    module declares its path as: github.com/sirupsen/logrus
            but was required as: github.com/Sirupsen/logrus

It appears that isn't in github.com/dagger/cloak incorrectly...

[civic yacht]

You need a replace directive https://github.com/dagger/cloak/blob/9250e5d71e381aeb142c53650b9f80546cb48d2f/go.mod#L111

(That will be included with the new docs coming soon)

[ancient kettle]

🙏🏻 Thank you!

[twin crow]

it was posted before here, but I guess it's worth repeating it, the code that Solomon demo'd live with Cloak is here: https://github.com/dagger/todoapp/tree/cloak-demo-v2

[wild zephyr]

@tepid nova I've spent quite some time working with backstage.

Backstage on it's core exposes a "service catalog" API which describes relationships between API's > Components > Resources (https://backstage.io/docs/features/software-catalog/system-model). On top of that core it allows anyone to extend that model and come up with new entities that can have relationships with the core entities.

My first ideas about how cloak and Backstage could play together nicely are the following:

• Have a cloak dev cache-persistent playground embedded on my Backstage UI where I can dynamically add / remove private and public extensions and prototype queries super fast to create workflows for my organization.

• Make Backstage be cloak aware of the services that are currently using cloak workflows and provide "Governance" about who triggered what and when. For example, I foresee a UI where I can trigger a specific workflow for a specific service and backstage stores all the transactional metadata about what outputs that workflow generated and by whom. On top of that we could leverage on the Backstage RBAC model to add/remove capabilities to users for specific Cloak workflows

[ancient kettle]

@civic yacht Question for you...

package main

import (
    "context"
    "fmt"

    "github.com/Khan/genqlient/graphql"
    "github.com/dagger/cloak/engine"
)

func main() {
    ctx := context.Background()
    if err := engine.Start(ctx, &engine.Config{}, func(ctx engine.Context) error {
        fmt.Println("It has begun")

        var resp *graphql.Response
        err := ctx.Client.MakeRequest(ctx, &graphql.Request{Query: `
            query {
                core {
                    image(ref: "alpine") {
                        exec(input: {
                            args: ["echo", "{\"hello\": \"world\"}"]
                        }) {
                            stdout
                        }
                    }
                }
            }
        `}, resp)
        fmt.Println(resp)

        if err != nil {
            return err
        }

        return nil
    }); err != nil {
        panic(err)
    }
    return
}

is giving me this error (after running everything seemingly cleanly):

#15 /_shim echo {"hello": "world"}
#15 CACHED
panic: failed to solve: json: Unmarshal(nil *graphql.Response)

goroutine 1 [running]:
main.main()
    /Users/joel/src/dagger-classic/main.go:39 +0x6c
exit status 2

Any idea why I'd be getting that panic?

Oh... I didn't initial resp 🤦🏻‍♂️

[civic yacht]

Might be this line: var resp *graphql.Response

[ancient kettle]

Yeah... definitely that.

https://giphy.com/gifs/robert-downey-jr-sigh-confused-116a8zosxwA0SI

[civic yacht]

No worries at all, I have gotten stuck on a tooon of little things like that when trying to use the raw gql client in go. It's so verbose it's easy to overlook tiny stuff.

Just so you know, you can get autogenerated client w/ go. I'm actually updating that exact doc at this moment, but the basic idea is A) declare a dependency in cloak.yaml and B) run cloak generate (no args needed anymore) from the dir containing cloak.yaml or any subdir

[ancient kettle]

Thanks, I appreciate it. 🙂

I look forward to seeing that doc. I think I might have created one while mucking around... 🙂

[civic yacht]

Will send it over shortly, once I've done a test run

[tepid nova]

Welcome newcomers!

[tepid nova]

I just realized a downside of keeping a runtime dependency on cloak.yaml: as it stands you can’t use the cloak SDK to produce a standalone tool (like dagger-classic)

That is pretty major actually

[cosmic cove]

<@&1003717314862129174> welcome newcomers! You should see a calendar invite for our Cloak community call. During the call, the community brings questions, shows demos, discusses use cases, etc. It is every Thursday at 10 am PDT. We hope to see you there!

@loud mica here is the channel that I was mentioning 🙂 Feel free to any questions here, and we will be sure to answer!

It will be interesting to see how Cloak helps your current use case

[loud mica]

Sure @cosmic cove , thanks for the invitation 🙏

[civic yacht]

Ran through this successfully in case it's helpful: https://github.com/dagger/cloak/pull/95/commits/61647779c41236fc5bd6965c10a390c4b33643fd
The instructions technically will only work exactly as is once the PR is merged into main (since they refer to the main branch), but hopefully the gist of it is still helpful if you haven't figured it all out on your own already.

Next up is the same for TS, then I think we can finally merge and you won't have to base off my branch anymore...

[ancient kettle]

I experienced this today. Probably worth loosening this restriction, if possible.

[scarlet gate]

I love the jacket 😁

[tepid nova]

Really, twitch?

The algorithm said "that's it. the perfect preview frame"

https://tenor.com/view/yodeling-intensifies-yodel-mountain-the-price-is-right-cliffhangers-gif-13442945

[scarlet gate]

@civic yacht here it comes… https://github.com/dagger/cloak/issues/152

[obsidian rover]

From yesterday, the clientdir action has disappeard. We now have the host core api. Hmm, that changes things 🤣

[tawny flicker]

https://tenor.com/view/big-oof-size-small-medium-switch-gif-17355313

[wet mason]

Migrated from Playground to GraphiQL

[cosmic cove]

<@&1003717314862129174> reminder that the community call is at 10 am PDT today! I won't be there today, but you are in amazing hands with @tepid nova and @hasty basin ! Bring your questions, demos, use cases, or just come to listen in. We look forward to seeing all the new faces!

[tepid nova]

Thanks for a great call everyone

Here are the areas of contributions I mentioned earlier. This is not an exhaustive list:

1. Bundling buildkit. We want cloak to be a standalone binary
2. “Dagger classic”. Cue SDK with europa compat. We have a design. Prototyping can start.
3. New use cases. Do something real with cloak. Real world projects are best.
4. Bash SDK. 🙏
5. Adding core API buildkit bindings
6. multi-platform.
7. Go DX. Needs a lot of polish
8. Typescript DX. Same.

More discussions in this thread: https://discord.com/channels/707636530424053791/1014065280835661905

Areas of ongoing core design:

1. Adapting core API to GraphQL best practices
2. Define extension points. What can an extension do, and how are they integrated into the API?
   • Option 1: supergraph model. Each extension is a mini-API, cloak aggregates them (more flexibility; less interop)
   • Option 2: regular graph model. Cloak has its own API, extensions can extend some of the types in some ways (less flexibility; more interop)
3. Project file layout and format (cloak.yaml)

[tepid nova]

@civic yacht @wet mason can we catch up on DX and aftermath of demo for a few minutes before Andrea logs off?

[hasty basin]

@round granite https://www.twitch.tv/videos/1578035654

[hasty basin]

Don't get too stressed out, the audio-visual difficulties get overcome and the demo happens 😄

[tepid nova]

@wet mason @civic yacht @hasty basin @round granite @tawny flicker thanks for the discussion, I added a list of core design topics to the summary above 👆

[tepid nova]

I'm reading up on GraphQL interfaces to understand how they could fit in our extension model: https://graphql.org/learn/schema/#interfaces

[tepid nova]

@civic yacht is workflow-clean still your primary branch?

I'm going to start from there for my core API proposal

[civic yacht]

No everything is in main now

[tepid nova]

Ah ok

oh right I missed type Host

saw type LocalDir and assumed it wasn't merged. But I guess you added Host on top o fit

congrats on merging btw 🙂

I've been reading up on type extension, I'm only now realizing that extends is part of the core graphql spec (not an apollo thing)

it looks like you can extend any type, including enums 🙂 Could come in handy as we explore best extension model

[civic yacht]

Thanks! Glad the pain of a separate branch is over for everyone.

Also, this PR updates the todoapp branch to use main when you have time: https://github.com/dagger/todoapp/pull/10

saw type LocalDir and assumed it wasn't merged. But I guess you added Host on top o fit
Yeah I suppose there's mixed terminology there still, probably should be HostDir.

[tepid nova]

eg.

Core:

extend type Query {
  repository(namespace Namespace!, address String!): Repository
}
enum Namespace {
  Git
  HTTP
  OCI
}
interface Repository {
  artifact(name String!) Artifact
  artifacts [String!]

Debian extension:

extend enum Namespace {
  Debian
}
DebianRepository implements Repository {
  artifact(name String!) DebianPackage
}

[civic yacht]

Oh nice I didn't know about enums being extendable. Apparently interfaces are too: https://spec.graphql.org/October2021/#sec-Interface-Extensions

[tepid nova]

I couldn't find anything about type extensions in the docs, so went looking in the specs. They're pretty readable.

[civic yacht]

Agree, much better than most specs.

Another thing to look into if you haven't already are fragments: https://graphql.org/learn/queries/#fragments

I don't have any super concrete ideas immediately, but they feel potentially useful

[tepid nova]

Yeah I was wondering about a pattern to reduce stitching overhead in the case of branching out a pipeline, using gql fragments + buildkit dedup

(when pipeline A branches out into pipelines B1 and B2)

You could define A as a fragment, then use it to make only 2 queries:

• A + B1
• A + B2

Buildkit deduplication takes care of the rest

[obsidian rover]

Btw, I don't know if most of you saw. But Andrea bumped the playground version to one of the latest gaphiql version, and now it's even more awesome.
-> Literally a bump in DX in 20 min

[tepid nova]

Oh cool! Trying now. What's changed?

[obsidian rover]

Linter integrated, cmd + k to search inside docs. No more bugs (it used to freeze when searching to deeply in docs)

Autocompletion works well. We literally benefit from the web community tooling on this

[tepid nova]

What happened exactly in your case? What change to the cloak command would fix your issue?

Trying to understand the smallest possible change we can get away with

[tepid nova]

Looks really nice! But I don't see the docs?

[tepid nova]

Oh! I see it

Designers being too clever

There's this thing called "text"

where you write down what a button does...

[obsidian rover]

There's also the history of the requests below 🤯

[tepid nova]

@civic yacht what's the story with IDs, are we going to keep using "llb cookies" (as @twin crow calls them 🙂 or are switching to random IDs?

(could influence the core API design one way or the other)

[civic yacht]

I don't think we are going to ever switch to random. We wanted to switch to them being hashes of the LLB, but after I wrote the code for that I realized it was much trickier than I thought: https://github.com/dagger/cloak/issues/12#issuecomment-1230578506

So for now no clear path away from them literally being the LLB

The best we can do is compress them so they aren't stupendiously long, which is a bit silly but the only change I can see in the short term

[tepid nova]

Right sorry - llb cookie or llb hash then

The interesting bit is that if they are cookies, then we can rely on them as content in addition to just a content address

Which is perhaps useless and better left hidden as an implementation - or perhaps could be used for API design ?

(no idea how)

[civic yacht]

Yeah I actually like that a lot. I just don't like that they can grow to indefinite size and have to be passed around so much. Not that we have to optimize for performance right now, but I do get concerned that could bite us someday in the indefinite future.

[tepid nova]

I think thats OK actually

fundamentally they are code. Code size scales with human effort which is to say: not a lot

("because humans are lazy and weak" my computer would say)

[civic yacht]

The other part is that they are so long they pollute output. That can obviously be fixed, but is actually trickier and more development/maintainance work than it seems

[tepid nova]

the compression doesn't fix that?

[civic yacht]

It helps, but it's like instead of taking up multiple pages of my screen, it only takes up a 3rd of my screen

When I experimented quickly the other day

[tepid nova]

I see. yeah that's not great

OK so I will assume that it could be changed to a hash at any time

[civic yacht]

Yeah probably the best assumption. Actually doing that will not be straightfoward, so won't happen soon, but may need to someday

[obsidian rover]

There's a use-case I don't understand (in types imbrication) ->

client: host {
    workdir {
      read {
        exec(input: {args: ["ls", "-a"]}){
          stdout
          stderr
        }
      }
    }
  }

In that context, read returns a Filesystem. We have access to the exec, which will inevitably fail because it's not running in a real image (am I right?)

Btw, I have this issue:

"message": "process \"/_shim /bin/ls -a\" did not complete successfully: exit code: 1",

But truth is, I don't know how to debug (change verbosity)

[tepid nova]

@obsidian rover correct unless you happen to be sitting in an unpacked linux rootfs as your workidr on the host machine

I think it will make more sense to move exec to a Container type (I'm drafting a proposal for that right now)

[obsidian rover]

My current biggest confusion, as ramping up, is debugging atm. Will explore tomorrow (maybe read the docs better). Whole experience feels so much better, explanations are ok too. I'm just still a bit confused on how to debug some parts here and there

[tepid nova]

More graphql noob questions (sorry). Do we want to give everything an ID? That seems like a very common pattern, in the github API for example

[civic yacht]

Yeah the debugging experience is painful right now, please bring up and create issues for whatever is most painful

[tepid nova]

In fact it may be necessary for the core API design we want. Right now Filesystem and Secret are special because they can "checkpoint" the llb state so far for composition across queries. We may want more types to be able to do that.

[ancient kettle]

iirc, it panics on missing file or similar when I try to run my go code.

[civic yacht]

I think things that are persisted state that you want to reference should have an ID. I suppose this is probably the same as any other graphql API that is in front of a database. I guess in our case Buildkit's cache is our "database"? Secret IDs are similar in that they have to be stored in memory in a store hooked up to the buildkit session

[tepid nova]

Right

[civic yacht]

I don't know if we need a hard rule that "all objects must have an id" though. E.g. we'll probably have an ImageConfig object at some point, not sure if that needs an ID

But maybe it could?

[tepid nova]

In my proposal of having a Container type which has a rootfs: Filesystem field, would Container need an ID? I suppose so (to retrieve the rootfs)

[civic yacht]

I don't know in this case. That would imply that either:

1. We actually persist the mapping of container id -> container somewhere in the cloak server
2. We embed the definition of the container into its id (essentially what filesystem is where id is the serialized llb)

[tepid nova]

Right, I was thinking 2 (since 1 would be a major architecture change it seems)

How much work is option 1 if I wanted to try? 🙂

[civic yacht]

You will run into the same problem I ran into where I tried to change fsid to be a hash I think. Doesn't work when you get cached values from previous executions. The only thing I can think of is to have our own persisted DB (or re-use buildkit's/containerd's boltdb somehow).

[tepid nova]

Sorry I meant option 2!

If I wanted to try option 2 (assuming it's the easiest), how much work?

[hasty basin]

qq on tags and docs for dagger/cloak repo
is the current demov2 tag stuff going to remain as is for now?
or are we bringing that content to a branch?

Are the most current general docs on main branch?

[civic yacht]

Oh haha, I mean all objects are json serializable, so you'd just have to serialize it. Might want to run it through b64 too to avoid lots of json-in-json escaping, but not that hard i don't think

[civic yacht]

You shouldn't need the demov2 tag anymore, I merged everything in main yesterday. I'm not deleting the tag until the todoapp repo itself has been updated (e.g. this PR is merged https://github.com/dagger/todoapp/pull/10) and I'm sure no one else is using it anymore though.

Yes most current docs are on main, updated them before merge yesterday

[civic yacht]

See these ones in particular:

• https://github.com/dagger/cloak/blob/main/docs/writing_extensions_go.md
• https://github.com/dagger/cloak/blob/main/docs/writing_extensions_nodejs.md

[tepid nova]

actually I should be able to use the same IDs for container and fs, si ce everything I need is in tbe llb state

unlike in Europa we don’t need to disable parts of the llb state, we can use it all the way

might make multi arch easier too

and of course no need for the docker/core split for push, pull etc

no more ImageConfig either, just individual resolvers for each field that would query the llb state

[civic yacht]

I thought this too, but for some reason when you serialize llb.State to LLB protobuf, all the image config meta fields are dropped and not thus honored if you deserialize. Discovered this after I told @twin crow the other day it would be super easy to add support for honoring image config values in exec and then soon realized it was not so easy after all.

However, we can fix this by slightly modifying FSID to have both the LLB protobuf and image config metadata and serializing all that instead of just the protobuf. So we can still achieve what you're describing relatively easily (just not as easy as it seems like it should be).

[tepid nova]

Ok I see. I guess it would make sense to eventually just serialize everything into our own polyfill format, which would be a strict superset of llb

[tawny flicker]

I'm hitting some blocking today in trying to create an extension:

dolanor@eastpeak:~/src/cloak/example/test$ cloak generate -p ./example/test/cloak.yml --workdir ../../
#1 local://__cloak_workdir
#1 transferring __cloak_workdir: 7.76MB 0.1s
#1 transferring __cloak_workdir: 17.63MB 0.2s done
#1 DONE 0.2s

#2 copy / /
#2 DONE 0.1s
Error: failed to solve: input:5: core.filesystem.loadProject yaml: unmarshal errors:
  line 2: field sources not found in type project.Config

[tepid nova]

but conceptually what you are proposing is the same

[civic yacht]

Yes we can get progressively fancier as time goes on

[tepid nova]

@tawny flicker your cloak.yaml is probably different than what the latest version of cloak supports

[civic yacht]

There was a change in the yaml format, sources has been split up into scripts and extensions

Yeah what Solomon said

[tawny flicker]

ok, I guess we need to update the examples as well

[civic yacht]

Oh where are they out of date?

[tepid nova]

@tawny flicker look at that you just discovered a useful PR to make 🙂

[tawny flicker]

(and dogfood test so we detect that on CI)

[tawny flicker]

I know, makes me feel like a winner

[tepid nova]

Any ideas on how to express docker push, git push, netlify deploy in a graphql-friendly way?

I see how to design a graphql-ey schema for downloads that doesn't have verbs, but uploads are harder

[tawny flicker]

I guess I am out of date.
Better go in bed.
The examples are actually ok. My eyes were too tired to see the difference

[civic yacht]

If there is data to select on the pushed/uploaded/deployed noun (i.e. select the url of the deployed site), then that's one path. I guess we may run into cases where we are just triggering a side-effect and nothing else, in which case it's kind of awkward. I saw people saying they just use Boolean as return types in those cases. But they were random github commenters, so not sure if it qualifies as a best practice

[civic yacht]

No worries, yeah get sleep!

[wet mason]

The graphql-friendly way would be for those to be mutations

But I don’t know how we’d want to translate them into cloak

[tepid nova]

OK. Do you think we should pursue that? Are there mutations in our future?

Or is there an immutable implementation problem that prevents their use

(I know you guys have researched that but I'm not up-to-date on that)

[wet mason]

I think mutations make sense for things that have side effects in the API (like addSecret, loadExtension)

For “uploads” (netlify deploy, git push, …) I don’t know

On the one hand it would make sense for them to be mutations, in a graphql-friendly way. On the other hand, we lose chainability by doing so

[civic yacht]

AFAIK the only actual underlying difference is that w/ mutations the selection set is executed in serial: https://spec.graphql.org/October2021/#sec-Mutation

Which is just meant to handle race conditions when updating state

[tepid nova]

Does addSecret really have side effects? Secret IDs are handled differently than FS IDs (ie no cookie)?

So there's a global lookup table I guess

For extensions I was thinking we could borrow llb's with... pattern to remove the side effects. Could be an extension of Query?

extend type Query {
  withExtension(input: ExtensionInput) Query!
}

[wet mason]

Yeah, but in the same way as in REST, the only actual difference between POST/PUT is the header name. However there’s semantic meaning associated with them

Also the other difference is mutations are “flat”

mutation { foo { bar } }

Foo is the mutation. Bar is a query against the return type of foo

[tepid nova]

that one is a pretty big difference

If we use mutations, and we want extensions to create their own, we will need a good way to namespace them both between extensions, and within each extension

[wet mason]

And semantically speaking, queries are nouns (data), mutations are verbs (actions)

(typically — not in the spec but that’s how virtually all APIs around do it)

[tepid nova]

Right

query Example {
  withExtension(input: {
    git: {
      remote: "github.com/dagger/cloak"
      ref: "main"
      path: "examples/yarn"
    }
  }) {
       ...
     }
   }
}

https://tenor.com/view/dr-cox-giggling-happy-victory-smile-gif-7858617

[civic yacht]

While I hate to be a party pooper, I would be surprised if that worked, if I'm understanding correctly 😔

The full schema has to be available at the start of the query. Also I guess you'd lose the ability to e.g. see the docs for the schema before you submit it

[tepid nova]

Ah right

Forgot about that

Mutation is fine for loading an extension since it's kind of a special case

For adding a secret it feels more awkward to me, but I can't express why quite yet

[civic yacht]

Yeah so you're just saying that you have to decide to be a mutation at the start, e.g. you can't do a big chain of queries and then say go do this mutation? You can do as much chaining as you want after you do the mutation, just has to be at the beginning

[civic yacht]

The thing is we can load extensions in parallel, so even though I agree on the semantic level it makes more sense, it would be a limitation on the actual implementation

[tepid nova]

Yeah, no opinion on that

Basically I'm going to ignore the loading of extensions as a special case, as far as my core API proposal is concerned

[civic yacht]

I think that's my main hangup overall here, like the semantics of mutation vs query make perfect sense and we shouldn't ignore that aspect, it just feels awkward that we could be imposing arbitrary limitations (serial execution of top level selection set) when they don't need to be imposed. We are just different than other graphql use cases in that our "database" (buildkit) is highly optimized to handle parallelization, so our higher level API has less of a need to be specialized for that. Overall, still not sure

Edit: actually, it's more that our core api doesn't really have the equivalent of "incrementCounter" where the return value of queries are impacted by past mutations. We are just querying immutable content-addressable content. My words are failing me here, but something along these lines

[tepid nova]

Can gql input types be composed of other input types?

[civic yacht]

I think yes. Not embedded (like go structs can embed), but you can have a field in an input type that is itself an input type

[tepid nova]

We could organize the extension API around these constraints. For example, I'm assuming serializing extension loading is only a problem because we build them on the fly (which can take time and could be serialized). What if we separated the build part into a graphql-ey query, and it returns an extension ID. Then the mutation to load takes that ID as input. The loading itself would be quasi instant, and serialized mutations are no longer a problem. Could even be a good thing since we can predict the order in which extensions are loaded

[civic yacht]

That makes sense, I just don't think we currently care about the order they are loaded either. I think the schema merge is commutative (? if I remember terminology right). Any overlap is an error. But maybe we will care about the order someday, in which case yeah then a mutation is exactly what we'd want

[tepid nova]

I just learned that you can't pass a union type as input type 😭

[civic yacht]

there's an issue and proposal about this somewhere, let me find it again

I had the same sad realization a while back

[tepid nova]

https://github.com/graphql/graphql-spec/issues/488

That?

https://tenor.com/view/crying-neverending-story-gif-20148013

[wet mason]

The top-level field is the mutation, that’s the part supposed to alter the state. The rest is query

[civic yacht]

Ah yeah, but that RFC got replaced by this one: https://github.com/graphql/graphql-spec/pull/825

[tepid nova]

lol

[wet mason]

Yes, exactly that last point, incrementCounter.

That’s what I meant with addSecret, which is an exception compared to querying immutable content-addressable content

[tepid nova]

In the case of secrets, doesn't it depend a lot on how we implement them under the hood?

[civic yacht]

Yep, makes perfect sense. So the general rule might be "if your query has side-effects that could change the result of subsequent queries, it should probably be a mutation" ?

[wet mason]

Basically our “exec” for instance is not exec-ing— it’s querying buildkit for immutable content

addSecret and loadExtension are kinda different as they will impact future queries

[wet mason]

(And turns out that buildkit will actually exec if it doesn’t have that content handy, but it’s still a query for state at its core)

[wet mason]

Yeah I guess so

[tepid nova]

Right but in the case of addSecret, in practice queries are only affected once they know the secret's ID. So there is no risk of race conditions etc

Even though technically the secret can be looked up by any query as soon as it's added... in practice no query will know what ID to lookup until the addsecret returns

so if we wanted, we could present it as a querying immutable content

(I think?)

[wet mason]

True. Because we have no hypothetical updateSecret (which would love alongside addSecret in a typical gql api)

But even our secrets are immutable by nature

[tepid nova]

we could also scope the secret to the query, if we found a way to make the with.. pattern work

[wet mason]

But let’s say we do have removeSecret

[tepid nova]

For example

type Filesystem {
  WithSecret(name: String!, cleartext: String!): Filesystem
}

I guess in that case you coud dispense with secret IDs altogether. Caller passes a scoped name

[wet mason]

If you look at GH for instance, there’s a addComment() mutation that returns a Comment

If you do mutation { addComment(…) { author } }, the “author” part is not really part of the mutation, that’s a query on Comment (you could get the same data with comment(id) { author }).

So the mutation really ends at addComment

[tepid nova]

Maybe there's a lightweight Environment type to carry that metadata instead of Filesystem same pattern as llb state but expanded to our own sticky metadata

[civic yacht]

I guess if we had listSecrets that would also mean the addSecret should be a mutation (since now there's a query that is impacted by addSecret)

[tepid nova]

After all the llb API is a masterclass in chaining 🙂

extend type Query {
  "Load an existing environment or create a new one"
  environment(id: EnvID): Environment
}

type Environment {
  id: EnvID!
  withSecret(name: String!, cleartext: String!) Environment
  secrets: [String!]!
  image(...): Filesystem
  git(...): Filesystem
}

[civic yacht]

If I understand, yeah I think I agree. This issue about being able to chain along multiple mounts is in the same direction: https://github.com/dagger/cloak/issues/18

You're just saying pass along secrets too I guess. But then why stop there, why not:

type Universe {
  noun
  withNoun
}

But for every noun that we could want.

I'm sort of joking here (I think)

[tepid nova]

Well the difference is that secrets work that way in buildkit

[civic yacht]

I mean the way graphql execution works is that the parent object is passed to all child field resolvers, so I think you can put whatever you want in that object. Each resolver can just update its state and then pass it along to the next one. So I don't think we are limited to secrets and filesystems

I may be misunderstanding the suggestion though

[tepid nova]

To rewind, my understanding of the problem:

• We need a way to reference Secrets and Filesystems by ID (to mount them, copy them etc)
• Filesystems can be created, then referenced, in the context of a chained query, because no global state is mutated
• Secrets do mutate global state when created (the engine's global secret lookup table), so in theory we should use a graphql mutation to do so
• But I argue that maybe a mutation is not the right fit, and instead we can manage secret IDs in a way more similar to filesystem IDs
• I then explore various options for doing so.
• One option is the with.. pattern. Possibly a bad idea, I don't know.

[wet mason]

addSecret is a weird one btw. We put it there in a rush but I don’t know if it makes sense to be an API call

[tepid nova]

@civic yacht I think this pattern:

type Universe {
 nouns
 withNoun
}

Actually makes sense for every value of "noun" that 1) needs to be referenced by ID , 2) we want to manage those IDs in a way similar to Filesystem IDs (ie. either global or mutable, but not both), and 3) they're only used in one type

[wet mason]

The reason it’s there is just so we can use SecretIDs everywhere, so that when we have a better solution we don’t have to rewrite everything that touches secrets as we’re already using the ID indirection

[civic yacht]

Yes, that's right, much more precise than my original "every noun we could want" 🙂 I also don't know if it's a good idea, it's a very stringent pattern, but is intriguing in a way.

[civic yacht]

One use case would be that an extension creates a new secret (i.e. private key or cert or something) and then needs to pass it to another one

But doesn't want that stored in buildkit cache

[tepid nova]

It could be stored in the engine's own cache in memory?

via the polyfill cookie serialization format (or PCSF as I like to call it)

yes I just made that up

[civic yacht]

Inputs are passed to extensions via a file mounted into the extension's execop, so if you don't mark an input as a secret then you'll just be writing the secret plaintext to a file

[wet mason]

Yeah, for that particular use case it makes sense

I think ideally, long term, we should follow bk’s pattern and somehow have extensions be secrets providers

[civic yacht]

This is actually convenient for us atm because it means that the extension's execution is cached based on the inputs. Technically other ways of accomplishing that though, so PCSF is still a possibility 😄

[tepid nova]

@wet mason even with secrets providers (which would be awesome to have) scripts will still want to load secrets from plaintext themselves

I mean, sometimes

[wet mason]

Yeah

True

[civic yacht]

Yeah I agree that we should have secret providers, I just don't know if it covers every case. Like the one I mentioned where you have an extension that creates a new secret and wants to pass it around or return it. We could force users into the secret provider pattern there, and maybe we should, but it would be more awkward for some use cases than others

[tepid nova]

@civic yacht here's what I mean by "we could cache it in the engine memory"

• Encode a json-serialized description of how to retrieve that secret (maybe "here's the plaintext", maybe "get it from this provider with those settings")
• Return a "hash" of that description as secret ID
• When buildkit calls for a secret, look up the description, retrieve the plaintext, return to buildkit

In that case the "hash" better be a real hash, otherwise you might be copy-pasting base64-encoded, gzip-compressed secrets around...

[wet mason]

Agreed

I guess my point is, the whole point of SecretIDs is to have secret providers, or something, one day

By itself the current system is not better than just passing secrets around in plaintext, just with extra steps

[civic yacht]

I mean today if you call addSecret(plaintext:"shh") from an extension, the plaintext is never written to a file and thus not the cache. It goes over the unix socket to the cloak engine but is only stored in memory there. Then you get back the hash of the plaintext and that's what you actually pass around (and what gets stored in buildkit cache)

So I think even the current system is better than just directly passing around plaintext secret everywhere, IIUC

[wet mason]

Right, because of input caching

[wet mason]

Maybe like:

internal://<hash> (what we have today)

vault://<whatever> (tomorrow)

Maybe extensions can register handlers in the future

[tepid nova]

Hard to say how secret providers will work until we have more clarity on the rest of the API IMO

Would it be crazy if IDs were hashes of graphql queries?

[wet mason]

I was looking at 1password yesterday and saw they had some cool that would have made our life easier in the past (invite script for instance)

Basically you can set environment variables as GH_TOKEN=op://daggerinc/github/token, then you run your programs with “op run — myscript.sh” and it translates those env variables with plain text secrets

[civic yacht]

Okay yeah makes sense, I think you are both describing the same thing more or less, and I see how we could continue to have the same method that exists today (plaintext provider) but also support more

[tepid nova]

Or if IDs were actually just string-encoded graphql queries??

[wet mason]

Was thinking it would have been super nice if I could somehow use them with cloak

[tepid nova]

would spare us the trouble of inventing a polyfill on top of llb serialization... just use our gql schema as the polyfill

[tepid nova]

Same pattern as Doppler. I wonder if one copied the other

[wet mason]

I don’t know if 1password copied but they definitely came out later

[tepid nova]

Well you can write a script that embeds cloak, and run your script with op run 🙂

[wet mason]

They’re getting into b2b, with GH actions etc

[civic yacht]

Also like the vault wrapper thing you linked to previously I think?

[wet mason]

That was my plan

[tepid nova]

embedding ftw!

[wet mason]

Internally

Like everyone on the team could run yarn deploy

Without having to cruft a bunch of API tokens in env variables

[wet mason]

Yeah aws-vault

The thing I like about 1password is we’ve been talking about using aws-vault for a while, but it’s such a pita to set up AWS infra that we procrastinated and ended up with the mess of copying/pasting tokens manually until now

1password I could see myself using it in minutes for our CI/dev (especially since we already have a corp account to share e.g. typeform logins)

[tepid nova]

what if you sent several queries in the doc, and you could reference another query in the doc as an id?

basically nested queries

(orthogonal to secret providers)

[civic yacht]

You just have to pay an AWS Solutions Architect to help you, it's working exactly as designed 🙂

[tepid nova]

todoapp deploy would be just one request

https://tenor.com/view/dr-cox-giggling-happy-victory-smile-gif-7858617

[civic yacht]

What would the ID be?

[tepid nova]

just the name of the query in the current graphql doc

[civic yacht]

I feel like @wild zephyr mentioned something like that recently, but I could be misremembering

Everything is becoming a blur

[tepid nova]

An attempt at a cleaner filesystem API, using the buildkit-style with pattern https://github.com/dagger/cloak/issues/161

I'm going at it in small digestible pieces... This one is not too blurry @civic yacht 👆 , the scope is intentionally limited

[wild zephyr]

👋 yes, I mentioned about the experimental "@export" directive some graphql servers are implementing: https://sangria-graphql.github.io/learn/#batch-executor. Here's the corresponding graphql-js issue: https://github.com/graphql/graphql-js/issues/462

cc @tepid nova seems like somehow related to what you're thinking about

[tepid nova]

It's similar but not exactly the same

From what I understand, Sangria is trying to make that pattern usable for any graphql schema, using directives. In our case we only need it for our own schema, so we can probably do it without directives (I think)

I'm staring at this from their example, to make sure I understand it right:

query NewsFeed {
  feed {
    stories {
      id @export(as: "ids")
      actor
      message
    }
  }
}

query StoryComments {
  stories(ids: $ids) {
    comments {
      actor
      message
    }
  }
}

[wild zephyr]

I don't think it's straightforward since the graphql AST <> resolvers currently don't evaluate references across queries. If we want this to work for Dagger's schema without directives I'd assume we'd have to rewrite a good part of that.

I guess Sangria gets away with it since they also have an additional argument for operationNames where you need to specify the multiple nested opreations separately.

[tepid nova]

What I'm imagining we could do:

query Base {
  image("alpine") {
    exec(input: {
      args: ["apk", "add", "yarn"]
    }) {
      fs
    }
  }
}

query Install {
  filesystem(contents: "Base") {
    exec(input: {
      args: ["yarn", "install"]
    })
  }
}

[wild zephyr]

I'm a bit lost about what's the point goal of doing this. Avoid passing ID's around and having deep nesting?

[tepid nova]

Way less manual stitching in code. Same benefit as chaining, but in cases where chaining isn’t possible

[wild zephyr]

Agree.. but seems like the graphql community is also suffering from this limitations. I think I'd personally prefer to deal with having to write a bit more code in the extension and have more flexibility there to handle this behavior (which could be simplified with helper functions) rather trying to fiddle with the graphql spec ourselves.

[tepid nova]

I wouldn't call it fiddling with the spec (it doesn't require any deviation from the spec) but that doesn't mean it's a good idea 🙂

Generally graphql seems to be very OK with relying on introspection, so it wouldn't go against the spirit of the spec either IMO

I'm more worried about the additional implemetnation complexity on our end: you'd need to compute the graph of queries to resolve them in the correct sequence (sounds familiar? 🙂

[wild zephyr]

true, but wouldn't be adding a behavior that's not defined in the spec which "could" potentially break the spec in some way? (still TBD)

[tepid nova]

I don't think so but could be wrong

again there are many other reasons why this could be a terrible idea anyway

[wild zephyr]

yeah.. agree.. just thinking out loud about trade-offs here. I guess we're thinking of this idea along the DX aspets mostly?

[tepid nova]

Yes it's part of a wider conversation about core API design, how to make it as "graphql-ey" as possible, how to manage IDs, etc

Like @civic yacht said it's all becoming a blur 🙂

[wild zephyr]

let's grab a graphl gurus/gurvis without any context and ask them "how'd you express this in grahpql?". Done, ship it 😄

[tepid nova]

lol 🙂

"you want to express WHAT?"

[wild zephyr]

https://giphy.com/gifs/fallontonight-funny-lol-88iYsvbegSUn9bSTF8

[tepid nova]

welcome @vagrant goblet 🙂 The rest of your early access should arrive shortly but let's start with the discord channel!

[vagrant goblet]

hello friends! 👋
excited to play with cloak!

[tepid nova]

Welcome to the blur 🙂

Does buildkit getMount work only on regular mounts, or also on cache, tmp and secret mounts?

[civic yacht]

Only regular

[tepid nova]

TIL graphql has an ID type

[tepid nova]

For reference later on the topic of multi-operation documents, and their reasonable use: https://github.com/graphql/graphql-spec/issues/29#issuecomment-118994619

Note the 3d. point 🙂

[astral widget]

👋 thanks for adding me

[tepid nova]

Welcome 🙂 Beware that the repo is still work in progress, docs may be out of date or incomplete, etc

[wet mason]

Typically, yes

[tawny flicker]

I don't know about graphql, but yeah, I think ID is good, even if we don't use it everytime.
When you need it, you will regret not having any ID to reference in the first place, and then you'll have to start breaking stuff all around.

[wet mason]

For different reasons:

• There's more not just a SINGLE edge to reach a node (e.g. you can query the same object using different paths)

• un-chaining: You might want to split a query into smaller ones. Let's say both "users" and "organizations" have Todo edges. You might craft one query each to get Todo IDs and have another query that goes Todo ID -> Todo item, so you can re-use generic code

• It's part of the relay spec, which goes a notch beyond: you should have a top-level node and nodes query where you can pass ANY id in the system to retrieve a node (e.g. can retrieve a user or a todo using query { node(id: $id) }). Relay spec standardizes a few things on top of graphql to help with some usage patterns (e.g. if you follow it, you can re-use UI paginators etc that expect that spec). GitHub, buildkite, facebook implement the relay spec

• It's required for caching / synchronization. Apollo client (and I'm sure other libraries) use a combination of id and __typename as a synchronization/cache key. Again with point 1), there's multiple paths to reach the same node. So you can't cache a query, that's pointless (the user "Andrea" might have been retrieved by user(name: "andrea") but also through workspace(name: "dagger") { members { ... } } -- if you cache by query you'd have 2 cache entries for the same object. If I change my name to al and you have multiple entries on screen for the same object, that may display conflicting information). Instead caching/sync is keyed by id+__typename

[wild zephyr]

Interesting... I thought it was a custom scalar.

[vagrant goblet]

@wet mason your mention of the Relay spec immediately makes me think that some iOS engineer is going to put a deploy button on some iPad for the office

Unrelated: https://twitter.com/capileigh/status/1565690002012606467

Leigh Capili (@capileigh)

@invariantly Taking a deeper look this morning -- I like how having this API machinery in front of buildkit creates both a multi language interface and runtime

I do wonder how verbose the primitives will get and what sorts of fragmentation might come out of language specific abstractions

Twitter · Sep 2, 2022, 1:16 PM

[vagrant goblet]

By "3D point", are you referring to the content about server-side batch and pipeline requests?

[obsidian rover]

It may be a dumb question, but our router has a graphQL schema ? I mean, there is some graphql schemas there ?

Oooh, so it's the communication schema. There's a layer I don't get yet then

[wild zephyr]

you're referring to Cloak's core schema Guillaume?

[tepid nova]

Yes 👍

[obsidian rover]

I think so yes

[wild zephyr]

the schema is currently being resolved through go code. i.e: https://github.com/dagger/cloak/blob/main/core/core.schema.go#L26. Is that what you were looking for?

[civic yacht]

The "router" is the thing that we use to merge multiple schemas together. So it doesn't have it's own schema per-se. It's more that it takes a bunch of schemas (e.g. core plus any that have been loaded) and combines them together. We call it a router because it knows the mapping between each (field in the schema -> resolver for that field).

[tepid nova]

@civic yacht @wet mason I am getting a little obsessed with the potential of multi-op documents + directives to help us handle the remaining API design issues, specifically:

• Secrets
• Uploads (push, deploy, publish etc)
• Proper use of mutations
• Unintuitive "stitch by ID" pattern (per @tawny lion feedback)
• Dependencies

[civic yacht]

So like what you were describing here? #maintainers message

If so, I think I'm just missing the connection between that and the various mutation related issues and dependencies

[tepid nova]

My brain kept spinning its wheels on this topic last night since the discussion. As I'm writing down the more straightforward parts of the API in https://github.com/dagger/cloak/pull/163, the blur is expanding to other areas 🙂

I'm finishing up some polish on 163 then will open a separate PR on top for the more "blurry" stuff

Let's just say when I look at a multi-op document, I'm having a hard time not seeing the stages of a Dockerfile 🙂

[civic yacht]

Oh totally, there's definitely a connection there, something to do with DAGs and all that 🙂

[tepid nova]

I think there's huge potential. If we could give a special role to your graphql document in the DX, that could simplify things in the SDK too, for example make it a standard practice to write your graphql document for a given extension and even script.

Add the right directives to express dependencies, and you have perhaps a way out of the dilemna of A) dependencies in yaml, dragging a weird runtime dependency along, or B) dependency in native code, making it harder to share and collaborate. Now maybe there is option C: everyone write a graphql document, that's where dependencies are expressed.

Secrets:

query netlifyToken @secret {
  host {
    variable(name: "NETLIFY_API_TOKEN")
  }
}

query run {
  container(address: "index.docker.io/alpine") {
    afterCommand(args: ["apk", "add", "yarn"] {
      withSecretVariable(name: "TOKEN", secret: "netlifyToken") {
        afterCommand(args: ["yarn", "install"]) {
          // ...
        }
      }
    }
  }
}

EDIT: simplified further by using op name as secret

https://tenor.com/view/dr-cox-giggling-happy-victory-smile-gif-7858617

I'm wondering if we could do the same thing with named volumes to facilitate passing Directories around

query appSource @volume {
  host {
    workdir(exclude: ["node_modules"]) {
      id // FIXME: can we simplify further and not need ID?
    }
  }
}

query run {
  container(address: "index.docker.io/alpine") {
    afterCommand(args: ["apk", "add", "yarn"] {
      withVolume(path: "/src", volume: "appSource") {
        withWorkdir(path: "/src") {
          afterCommand(args: ["yarn", "install"]) {
              afterCommand(args: ["yarn", "run", "build"]) {
                // ...
              }
          }
        }
      }
    }
  }
}

EDIT: simplified further by just using operation name as volume name

[tawny flicker]

Can somebody do a live with me to see what's wrong with my config? I don't quite get it

Error: failed to solve: input:5: core.filesystem.loadProject open /tmp/buildkit-mount2407612308/hugo/cloak.yml: no such file or directory

[tepid nova]

Loading an extension:

query netlify @extension {
  git(url: "ssh://[e-mail]/dagger/cloak") {
    branch(name: "main") {
      contents {
        directory(path: "examples/netlify") {
          id
        }
      }
    }
  }
}

query run {
  netlify {
    // go about my business here...
  }
}

[civic yacht]

I like @secret and @volume but I also can't tell yet if this would be a better experience for, e.g., a js dev who just wants to write a script. Before we asked them to write a yaml file and then their code. Yaml file is annoying I completely agree and there has to be a way to make it optional or swappable for other formats. But if the change is just "write a graphql operation document with these custom directives" instead of yaml, then I don't know yet whether that's going to make them happier or sadder. Also, with the query run you have, does that entirely invalidate the need to write a script in e.g. js at all? What would the native code be doing?

Overall, I like the general idea a lot, I just am always hung up on whether we should be trying to find the "one correct format" or finding a way to support multiple possible formats. But I also don't have concrete suggestions at this point, so I think it's worth it to keep going down this operation document path

[civic yacht]

Yes, joining cloak audio

[tepid nova]

One step further, perhaps the resulting build is stored back into a volume?

query appSource @artifact {
  host {
    workdir(exclude: ["node_modules"]) {
      id // FIXME: can we simplify further and not need ID?
    }
  }
}

query appBuild @artifact {
  container(address: "index.docker.io/alpine") {
    afterCommand(args: ["apk", "add", "yarn"] {
      withArtifact(path: "/src", artifact: "appSource") {
        withWorkdir(path: "/src") {
          afterCommand(args: ["yarn", "install"]) {
              afterCommand(args: ["yarn", "run", "build"]) {
                directory(path: "/src/build") {
                  id
                }
              }
          }
        }
      }
    }
  }
}

EDIT: simplified further by just using operation name as volume name
EDIT: renamed "volume" to "artifact"

Mmm maybe "volume" is really "artifact

[tepid nova]

I think the trick would be to make these optional improvements. So you start really simple, then you gradually learn the features and make your gql document more and more powerful, while making your JS code simpler and simpler

Another benefit is that it facilitates the transition from script to extension

Since that document is the first step towards defining an API of your own

[tepid nova]

It also opens the way to a manageable set of mutations, if you can reference named artifacts in a local namespace

eg.

mutation export(artifact: ArtifactName!, remote: RemoteName!)

[wild zephyr]

I was thinking about this. I like where it is going since refactoring the query to make it an extension should be relatively straightforward

[cosmic cove]

If anyone missed the community call yesterday, you can watch the recording here or check out the notes in the community call invite 🙂

https://dagger-io.zoom.us/rec/share/DbsMFjH4FNoPr_Bk8e2w2c9ZXvs6-B-nyy8qHw5hGCMpFkCIGfWtpgVqsl3xj0X5.DkytfISEcgFjVQr2

Passcode: kgSc.9z9

[wild zephyr]

the downside of making this a mutation is if you have multiple exports, they'll have to be executed sequentially given the spec and current implementation. Not sure if that's a desired behavior.

[tepid nova]

query netlifySite($name: String!) @remote {
  container(address: "samalba/netlify-cli") {
    withArtifact(path: "/site", artifact: "appBuild") {
      withWorkdir(path: "/site") {
        withSecretVariable(name: "NETLIFY_API_TOKEN", secret: "netlifyToken") {
          afterCommand(args: ["netlify", "deploy", "--site", $name]) {
            lastCommand { stdout } // FIXME return URL instead
          }
        }
      }
    }
  }
}

mutation push("netlifySite")

# Push to a remote
$ cloak push netlifySite

# Export the contents of an artifact
$ cloak export appBuild -o ./build

[wild zephyr]

what would be a @remote? Is that a kind of artifact?

[tepid nova]

It's a target for a push/upload

[tepid nova]

Possible answer to this 👆

[wild zephyr]

got it

[tepid nova]

• @secret: create a named secret backed by the annotated query
• @extension: load an extension backed by the annotated query
• @artifact: create a named artifact backed by the annotated query
• @remote create a named upload target backed by the annotated query

[wild zephyr]

thinking if push should be a mutation since the operation seems idempotent to me. If nothing changes from the remote and the cache is used, it should return the same site URL?

[wild zephyr]

checking what's the idempotency rule for graphql

[tepid nova]

We won't be able to force all remote implementations to guarantee that

Even our own implementation doesn't (we just force to always: true)

Separately from that, I don't know how to express a deployment / docker push / upload without using a verb... And the only accepted place to use a verb in graphql, apparently is a mutation

[wild zephyr]

just checked and seems like mutations and idempotency are orthogonal (https://shopify.engineering/building-resilient-graphql-apis-using-idempotency), so using push as a mutation seems to be ok

[tepid nova]

One alternative is to skip the explicit push mutation altogether, and just auto-execute all @remotes in the document

This rigid classification (which implies a rigid sequence of execution: extensions first; then secrets and artirfacts; remotes last) reminds me of Bazel @tawny lion 🙂

[wild zephyr]

cries in Bazel

[tepid nova]

OK I'm really liking the direction of this draft. The examples feel fun to write and graphql-compliant

Taking a quick break then joining the fun in audio 🙂

Feedback welcome: https://github.com/dagger/cloak/pull/163

[civic yacht]

Yes I completely agree with that, as long as the first step is intuitive we can guide them the rest of the way as they need more advanced stuff

[tepid nova]

OK I pushed a bunch of updates. I'm feeling pretty good about it. Whenever you have a few minutes to go over it and share your impressions:

Another good reason for JS to embrace a DX where they write a graphql document then interact with it from their code : it’s a graphql best practice which they will benefit from adopting even outside of cloak

Allows for various optimizations which are considered good in graphqlistan

[hasty basin]

@civic yacht @tepid nova love a 👍 on the docs website PR. https://github.com/dagger/cloak/pull/103 I just caught it up to current. Can go over it briefly at next community meeting if we merge it. Makes our Cloak docs website the first class way to consume the Cloak docs. Consistent with the current Dagger docs. Incremental revolution of the docs! 😄 (since it takes more time than you would think to get everything in the right spots).

[tepid nova]

How do I browse the docs? are they hosted somewhere?

[hasty basin]

Have an issue for hosting them open. Today you can browse the markdown in GitHub:

• https://github.com/dagger/cloak/blob/main/docs/unxpq-introduction.mdx
• https://github.com/dagger/cloak/tree/main/docs/guides

or view the actual site locally from the cloak repo:

cd website
yarn install
yarn start

That ☝️ is explained in https://github.com/dagger/cloak/blob/main/docs/_README.md

[civic yacht]

I did just realize that when one markdown file links to another on github the links are broken (also just saw that this is mentioned "The links in the docs assume you are viewing via the website.", so that's my bad for not noticing before).

The links in the main README still work, which is good, but is there any fix to that issue of links between markdown docs? Just to hold us over until we have them hosted

[tepid nova]

That would be nice. Also browsing via the repo is pretty useful even in parallel to the docs

[hasty basin]

I did a look around before and didn't see a well-trodden path, but will look again.

Looks like we can go to relative file links: https://docusaurus.io/docs/markdown-features/links, which are the kind we had before. Means files are less move-around-able, which is prob not as much of an issue for now. seems the slug in each page will take care of us. I'm learning 😄

[civic yacht]

Yeah that sounds great, then it's still pretty easy to browse directly on the github repo thanks to the index in the main README

[tepid nova]

FYI I'm adding a detailed overview of the proposed API design in https://github.com/dagger/cloak/pull/163

[civic yacht]

(finishing something else up, will take a look in a bit)

[tepid nova]

Looks like github is down?

[hasty basin]

was for me too. seems to be back up

[tepid nova]

how cool would it be if cloak.yaml became cloak.gql 🙂

Could even include a query marked @source to make it completely standalone, this making the API sandbox way more useful (just copy paste the file’s contents into the web editor and you’re good to go)

With perhaps the exception of extension loading, unless we find voodoo incantations to bootstrap the full schema at runtime 😇

Otherwise that would require patching the playground with a hook to send extension queries before introspecting the schema

Logging off for now, enjoy tbe weekend everyone

[tepid nova]

For reference, apparently GraphiQL has plugins: https://github.com/graphql/graphiql/blob/main/packages/graphiql/README.md#plugins

[hybrid widget]

I was able to access graphql from PHP (using ext/curl) and build an image for my PHP webapp via cloak and a PHP build script. The "build" includes installing deps via composer and configuring apache docroot. Attaching the code in case anyone is interested in having a look

PHP sample code here

Now I'm wondering: if I have my PHP app code + above build/deploy script in gitlab and I normally deploy to heroku on every commit - could I do that with cloak? how?

[tepid nova]

Thanks for sharing! I will try to run it on my machine

[tepid nova]

Yes you could absolutely do that with cloak.

How to do it depends on how you usually deploy to heroku. The simplest would be to write a pipeline similar to the netlify deploy, but that deploys to heroku instead. Then call your script from gitlab on each commit

Ideally there would be a heroku extension and you would be able to use that with a very small query

[north jay]

Building out some stuff in cue (classic dagger). Everywhere I need a utility to do something I either need to hardcode an image to use (yuck) or pass through a ref or docker.#Image.
It makes the API for these packages kind of messy.

I think the approach with cloak with extensions compiled from source as needed may be really nice.
Hoping to play soon.

[tepid nova]

Happy to help anytime, if you want to just talk through your use case, let me know!

[dense haven]

been playing with cloak a bit coming from classic dagger. I'm trying to build some binaries in my extension (all good), and then write some output from that to the host fs. seems like my best bet is writing a script and manually chaining the action output between the two? is there an easier way to chain queries like that in graphql? I saw https://github.com/dagger/cloak/issues/58 for example but I don't think this quite works for writing back to the host, since I need a LocalDir not a Filesystem.

seems related to https://github.com/dagger/cloak/issues/41 as well, but I assume that would look more like LLB / buildkit changes instead of trying to find a way with LocalDir

[tepid nova]

Hi there! With the current API you should be able to do:

query {
  host {
    workdir {
      write(contents: FSID)
    }
  }
}

Note that the API is still work-in-progress, we're still defining the final design. See https://github.com/dagger/cloak/pull/163

Note: you can use the interactive API playground with cloak dev to explore everything available under query { host } : all interactions with the host system go there

[dense haven]

So right now I've got something similar to go.#Build, but as an extension -- this might be more illustrative:

query ($source: FSID!) {
    gomod {
        build(source: $source) {
            # sanity: binary is there!
            exec(input: { args: ["/_out/bin/main"]}) {
                stdout
                stderr
            }
            # now I have a Filesystem
            # TODO: how to write the binary back to host?
        }
    }
}

seems like I could write a script, and then have the script take the output FSID and run the query you shared, but i'm mostly playing with cloak dev/do so far

[tepid nova]

Yes that's how you'd do it at the moment. You need to "stitch" multiple queries together in your code, by retrieving a FSID from query 1 then passing it as parameter to query 2

See an example here: https://github.com/dagger/todoapp/blob/cloak-demo-v2/scripts/deploy/index.mjs

This particular script doesn't write to the host filesystem, but there's the same idea of stitching queries together in code. Pretty unavoidable as soon as you want to do something "real"

Note that we're discussing a possible alternative method of stitching in the new API discussion: https://github.com/dagger/cloak/pull/163

But still exploratory

[dense haven]

the todo v2 demo was super helpful, thanks for that! I'll take a crack at scripting the queries together. the ability to chain things off of Filesystem with e.g. extend type Filesystem was pretty neat to get a lot done in one query (although maybe not an explicit goal) it looks like the approach with afterCommand/lastCommand is similar, will keep an eye on that 👀

[tepid nova]

@dense haven in case you're using the Go SDK, there's also a Go example in that same branch

https://github.com/dagger/todoapp/blob/cloak-demo-v2/api/magefiles/main.go

As well as an example of wrapping your pipelines in an API extension

https://github.com/dagger/todoapp/tree/cloak-demo-v2/extensions/deploy

To recap, that branch has 3 examples:

1. Script (gql client) in Javascript
2. Script (gql client) in Go
3. Extension (gql server + client) in Typescript

[hybrid widget]

@tepid nova to save you some time, i have shared the github repo which has the php app and the above build code in scripts/build

[hybrid widget]

OK I see. I understand about the custom deploy script. But how would gitlab be able to run it (since gitlab doesn't have cloak pre-installed)? Would it be necessary to include the cloak binary in the gitlab project or some other way to enable it?

[dense haven]

from my own experimentation the scripts like https://github.com/dagger/todoapp/blob/cloak-demo-v2/api/magefiles/main.go#L17 and https://github.com/dagger/todoapp/blob/cloak-demo-v2/scripts/deploy/index.mjs#L30 can be run standalone (in the Go case as a standalone binary). the engine in the SDKs is the same as what drives cloak do/dev https://github.com/dagger/cloak/blob/main/cmd/cloak/do.go#L65

[dense haven]

I figured something out for my copying binaries out of Filesystem but not without some minor core changes. I has issues with core { workdir { write } } because it copies the entire FS, so I got a full FHS tree output to my workdir each time. I ended up changing localDirWrite to take an include: [String!] parameter just for testing, and did llb.Copy() into scratch before the export with the IncludePatterns set.

this is pretty well covered with cue classic since you can do that sort of thing already, so I expect it'll get sorted out 🙂

[tepid nova]

Unfortunately the Go SDK is a special case, since it benefits from the engine itself being written in Go: that makes it easy to embed without requiring the cloak binary.

The Javascript/Typescript SDK however does require the cloak binary to be installed. A php SDK will require this too. @hybrid widget since a php sdk is not yet available, you will need to do that part yourself. It involves running cloak from your php code, then connecting to its socket

[tepid nova]

Nice! Do I sense a PR in the near future? 🙂

You are correct, one way or the other this will all br covered since it’s still buildkit under the hood. We just need to find the right API design in Graphql-istan

[tawny flicker]

Oh the irony.
I got throttled while I was searching for some message about ratelimiting in buildkit, and how to deal with authentication.
And for the life of me, I'm pretty sure @civic yacht talked about it last week, but I can't find the message/issue, etc.
So if anyone would be so kind to find it again, @stray heron needs it.
And if no such thing existed, I'm gonna do therapy about pretend issues that exists in my head.

[stray heron]

Let's make this discussion public so that it's visible to everyone: https://github.com/dagger/dagger/discussions/2997

When we have a solution, I think that this is something that should be in the docs too.

[hybrid widget]

Thanks. I have tried the approach you suggested using the following .gitlab-ci.yml

image: docker:20.10-git
services:
  - docker:20.10-dind

stages:          
  - build

build:
  stage: build
  before_script:
    - |
      apk add go
      apk add php8
      cd /tmp
      git clone https://$GITHUB_USER:$[e-mail]/dagger/cloak.git
      cd /tmp/cloak
      go build -v ./cmd/cloak
      mv cloak /usr/local/bin/
      cloak dev
  script:
    - php scripts/build.php```

However it fails at the go build stage with exit code 1. There's no further info. I've attached the build log for reference. Could you please see if there is some mistake or something I should be doing differently? Or any way to get more detailed debug messages?

[wild zephyr]

However it fails at the go build stage

[wild zephyr]

interesting to see what Netlify is doing with their Graph UI experience (https://youtu.be/t8FIcX2UyXg?t=702) . I can see some ideas we could take for how extensions are loaded in cloak's graph...

[wild zephyr]

I particularly like the graph <> examples flow. That' neat.

[tawny flicker]

Does input:5: core.filesystem.exec failed to unmarshal result: unexpected end of JSON input: ring a bell to you?
It's an error from graphql.Client.MakeRequest or at least from the code we generate. I don't want to dig into this if it's something somebody already faced it.

[tepid nova]

doesn’t ring a bell sorry. Can you open an issue with details?

[tepid nova]

FYI I have pushed a major simplification of the Core API proposal. Thank you @civic yacht for the detailed feedback and notes. I think you're right that I pushed too far with custom directives, creating the risk of diverging from vanilla graphql.

I do have a few add-on proposals in store, but will leave them for future PRs to keep scope narrower. This proposal stands alone in its current form.

https://github.com/dagger/cloak/pull/163

README: https://github.com/shykes/cloak/blob/gql-builtins/builtins/README.md

Feedback, notes and questions welcome!

[tawny flicker]

I'll create a repro in an issue

[tawny flicker]

I'm getting some docker issues with my repro

dolanor@eastpeak:~/src/io/ci/repro$ curl -I https://registry-1.docker.io/v2/library/golang/manifests/1.18.2-alpine
HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/golang:pull"
date: Tue, 06 Sep 2022 12:34:52 GMT
content-length: 157
strict-transport-security: max-age=31536000
docker-ratelimit-source: 92.148.192.56

dolanor@eastpeak:~/src/io/ci/repro$ go run .
#1 local://__cloak_workdir
#1 transferring __cloak_workdir: 4.68kB done
#1 DONE 0.0s

#2 copy / /
#2 CACHED

#3 resolve image config for docker.io/library/golang:1.18.2-alpine
#3 ERROR: failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/1.18.2-alpine": dial tcp: i/o timeout
------
> resolve image config for docker.io/library/golang:1.18.2-alpine:
------
panic: failed to solve: input:24: core.filesystem.loadProject.install failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/1.18.2-alpine": dial tcp: i/o timeout

is it related with the authentication from buildkit?

Tried from my original example, same problem. It seems to be a docker issue

https://status.docker.com/pages/history/533c6539221ae15e3f000031

There is a shutdown from the hub? Is that the problem? It seems it was just a test

Ok, so the test was on the 2nd of september, to prepare for the real shutdown of the hub v1 API.
But the curl that fails is on the registry API, it seems somewhat unrelated.

[wet mason]

I think you hit the rate limit quota

[tawny flicker]

I can't believe it

On my tethered phone, it works. But on my brand new kick ass fiber connection, I get a timeout…

[mellow bolt]

I'm deploying dagger.io with cloak. It works (except for some images issue that I need to figure). However, it's deploying to my own Team on Netlify. I want to be able to choose Netlify account dagger. I remember we use to have this for Europa:

env: {
  NETLIFY_SITE_NAME: site
  if (create) {
    NETLIFY_SITE_CREATE: "1"
  }
  if domain != null {
    NETLIFY_DOMAIN: domain
  }
  NETLIFY_ACCOUNT: team <------- HERE ----------
}

[tawny flicker]

It's even weirder, my phone is actually connected to my wifi router, to which my computer is connected. So it's actually the same router in the end…

[mellow bolt]

I guess we should implement it for cloak cc @wet mason @civic yacht

[wet mason]

yep, that's just some example code

[wet mason]

@civic yacht Continued a little bit the query builder experiment in Go, added binding capabilities (like in flag parsing): https://github.com/dagger/cloak/compare/main...aluzzardi:query-builder-test?expand=1#diff-1b6185b0a44826e95e82f077b90cc982a13af4c6db19799cfdeba9391bccb9cd

[civic yacht]

If you sign into dockerhub (via docker login), cloak will pick up and use those creds to authenticate (commit here: https://github.com/dagger/cloak/commit/8f3b2b71b7aad379260c4f921f25fec9bad23ba7)

Not as generic and useful a solution as we'd like in the longer term, but it was a one-line change so we added it last week before the demo

[civic yacht]

Updating the docs with that here: https://github.com/dagger/cloak/pull/175

[wet mason]

@civic yacht Hey are you aware of the netlify setup? Not sure where it came from and all PRs are red

[civic yacht]

No I don't know where the PR checks came from, we merged the website changes end of last week but that didn't have any netlify config at the time

[hasty basin]

@civic yacht @wet mason I'm guessing it's from the team working on getting the Cloak website to deploy.

@mellow bolt @dense dust ☝️

Could we only run those checks on PRs to /website or /docs?

[mellow bolt]

yes error are coming from cloak docs (sorry about that)

[hasty basin]

let me know if I can help with anything 🙂

[mellow bolt]

I'm in the train and my connection is unstable. I'll have a look when I'll be home 🙂

[dense dust]

I'm looking at it too, will update if I find something

[mellow bolt]

Ok I've just updated the branch with Github and it seems to fix the problem

I'm means for docs deployment at least

[dense dust]

Yep, seems to work! Thx

[tawny flicker]

NICE!

[tepid nova]

@civic yacht want to talk live about DX? The core API proposal shrank back to a pretty reasonable scope, nothing too surprising in there I think (still plenty of bikeshedding opportunities of course)

However we need to talk about doc-centric DX and all the ramifications of that

I allude to it in 163 README but it deserves a whole discussion on its own

[civic yacht]

Yeah for sure, I'll join cloak-audio in 5 mins

[tepid nova]

eta 5mn 🙂

[wild zephyr]

https://youtu.be/t8FIcX2UyXg?t=3084

^ @tepid nova encourages to watch a few mins of this for inspiration. Particularly how we could start approaching a doc-centric DX

[civic yacht]

@twin crow lemme know if I can help with the question you were asking before the meeting started

[twin crow]

yep, basically while reading the code, I realized that the runtime functions (project/sdk.go) that generates a base layer with some LLB for each language are only used with SDK, it's a bit misleading at first, but logical. I then noticed that the ts SDK (engine.ts) is simply shelling out a server from the cloak binary with some args. I am replicating that in Python.

so not really a question anymore 😛

[civic yacht]

Ah okay, yes you got it then, the runtime stuff in project/sdk.go is only used for extensions at the moment.

[hasty basin]

👋 @radiant vortex

[radiant vortex]

Hey yall thanks so much for the invite!

[hasty basin]

Welcome! Love to hear more about what you want to do! We have a community call every week on Thursday 10am PT.

[radiant vortex]

Just keeping up with the dagger tbh. I'm more or less just experimenting to see if we can abandon jenkins one day. Got a rather large Jenkins shared library setup, which is a bit exhausting to work with sometimes.

Really like the non-yaml ness of dagger, imports, and ability to run local. Trying to read through the docs here on cloak.. seems the non yaml bit may have changed 😱

[hasty basin]

I wondered about what the large shared library experience was like with Jenkins. I'm giving a talk on Jenkins + Dagger/Cloak at the end of the month. Figured that with shared lib, you get sharing across Jenkins, but not with local dev enviros or other CI or CD systems you might have to interop with.

[hasty basin]

The YAML bit of cloak today is not set in stone 😄 Just has the list of cloak scripts and dependencies.

We're constantly interating on better DX.

[civic yacht]

^ Yes it's just holding us over until we have something better 🙂

[hasty basin]

Welcome back to YAML hell! 👿 jk 😇

[radiant vortex]

gotcha hah. yeah, some guys on the team are pretty anti yaml (at least for controlling execution flow) . one of the main reasons we're still in jenkins.

I'm actually a tekton fan, but implementing any kind of looping/conditional logic in yaml gets ugly quick, and thus we're still on the 50 year old jenkins train

I'll have to tune into the talk!

[hasty basin]

https://events.devopsworld.com/widget/cloudbees/devopsworld22/conferenceSessionDetails?tab.day=20220928&search=dagger

[tepid nova]

What @hasty basin said: no yaml hell planned in cloak 🙂 it’s just a small config file without actual pipeline logic in it

@civic yacht @wild zephyr @hasty basin we tried writing a mock dagger.gql for todoapp… And were surprised to discover that in this particular example, an extension is not actually needed. Backend and frontend team can just reuse the operations file

i find that very promising because it would push the creation of extension further down the onboarding path, which probably would make the learning curve less daunting

basically extensions would be a power user feature

Not necessarily needed for team collaboration. Perhaps more of a “scale collaboration to hundreds of engineers” feature

Extensions could still ship operation files as examples -> which plugs into the workflow @wild zephyr described

[civic yacht]

Also needed if you are trying to do something like import a library in some language. Can't import a go library into graphql code. But that very well may end up being a power user feature, hard to say right now.

But yeah I think that's very promising. From my perspective, if you squint this is almost like being able to author an extension with the GraphQL SDK 🙂 And that's a good thing since in the current extension authoring DX you have to write graphql anyways, why not just write the whole thing in graphql when possible. Like I said before, my dream is that we can abstract graphql away from the extension author DX, at which point you'd still have this GraphQL SDK but also equally nice ones in other languages like Go where you can ignore graphql entirely. But if that dream doesn't pan out, then the graphql sdk grows even more in importance. Point being that either way I think we should keep going down this route.

Interested to see the mock dagger.gql whenever you have time to push it somewhere

[civic yacht]

Yeah or similarly, it might make sense that whatever operations you define in dagger.gql are made available to anyone depending on you (maybe)

[tepid nova]

Will do tonight 👍

[wild zephyr]

also friendly reminder about the doc-centric DX discussion kick-off 😄

[twin crow]

Open to early feedback: https://github.com/dagger/cloak/pull/177 - there is still a bunch of work to do (listed in the draft PR), but at least I am able to do embedded cloak queries from Python

there are many ways to expose the Engine, I started copying the Javascript implem with a callback, but ended up using a context, more python-native to my taste... Anyhow, if anyone wants to join the fun, I opened the draft PR to avoid duplication of effort.

[tepid nova]

As soon as the kids are asleep 😅

[tepid nova]

https://tenor.com/view/you-crazy-son-of-a-bitch-you-did-it-jurassic-park-ian-malcolm-gif-16042346

[median charm]

Hey 🙂 Thx for the invite - excited to take a closer look

[tepid nova]

Welcome @median charm ! Fair warning the docs are still quite rough and partially out of date, as the code is moving fast. Please don't hesitate to ask any question here at any time. No such thing as a dumb question here!

[median charm]

Thx @tepid nova - and thanks for the heads up 🙂 I will dig in and see what's there

[tepid nova]

Do you have a use case in mind for testing it? Maybe an existing pipeline, or a tool you'd like to integrate with?

[tepid nova]

"Load netlify API token from host environment"
query netlifyToken {
  host {
    secretVariable(name: "NETLIFY_API_TOKEN") {
      id
    }
  }
}

"Load source code from host working directory"
query source {
  host {
    workdir(exclude: ["node_modules"]) {
        id
    }
  }
}

"A builder container with the app mounted"
query builder {
  container(id: "index.docker.io/alpine") {
    afterExec(args: ["apk", "add", "yarn", "openssh", "git"] {
      withMountedDirectory(source: "source", path: "/src") {
        withWorkdir(path: "/src") {
          withMountedCache(path: "/var/cache/yarn") {
            withVariable(name: "YARN_CACHE_FOLDER", value: "/var/cache/yarn") {
              id
            }
          }
        }
      }
    }
  }   
}

"Build the app"
query build {
  container(id: "builder") {
     afterExec(args: ["yarn", "install"]) {
      afterExec(args: ["yarn", "run", "build"]) {
        directory(path: "./build") {
          id
        }
      }
    }
  }
}

"A container setup to deploy the app"
query deployer($contents: DirectoryID!) {
  container(id: "index.docker.io/alpine:latest") {
    afterExec(args: ["apk", "add", "npm"]) {
      afterExec(args: ["npm", "-g" "install", "netlify-cli@8.6.21"]) {
        withMountedDirectory(source: $contents, path: "/site") {
          withWorkdir(path: "/site") {
            withSecretVariable(name: "NETLIFY_API_TOKEN", secret: "netlifyToken") {
              id
            }
          }
        }
      }
    }
  }
}

query deploy($siteName: String!) {
  container(id: "deployer") {
    afterExec(args: ["netlify", "deploy", "--site", $siteName]) {
      stdout // FIXME: return URL instead
    }
  }
}

In-document stitching in action 🙂

[obsidian rover]

Solomon, how can I play with that ? It seems that you do everything in graphql (in the playground), and you're now able to cross-reference queries. Is it on a specific branch ? (I presume: PR #163)

[tepid nova]

Yes it's all in that PR. With one small detail...

There is no implementation 🙂 I'm just writing mockup queries that I hope to run one day soon! But first I want to get feedback on the DX. wdyt?

There's a detailed README in PR 163 where I go over design highlights and the rationale behind them

• full graphql schema with comments

This example above is in the README, along with a few more

[obsidian rover]

Thanks, will look more closely today 😇

[mellow bolt]

same

[tepid nova]

Note that if we copied the ideas from Netlify Graph re: shareable and runnable examples, then soon all those examples could be runnable in one click from your Dagger Cloud account 🙂

[mellow bolt]

that would be awesome

[tepid nova]

I really think the in-doc stitching (query can reference other query) is a gamechanger. It's like multi-stage dockerfiles... but cleaner

And all graphql spec-compliant and supported out of the box by GraphiQL playground

[obsidian rover]

Btw, I had in interest into building an SDK/seing how to make one. I saw Sam's draft PR on Python. Shall we focus on helping him or making a parallel one (ruby?) [bash seems to ambitious for me, as no type system]

[tepid nova]

That would be a fun project, as long as you consider it to be a throwaway prototype

[obsidian rover]

Just want to help the most, ramp-up as much as possible

[tepid nova]

DX will most likely keep moving in the next few weeks, in a way that will change how SDKs work and integrate with the core

[mellow bolt]

we get used to.... 😏

[tepid nova]

Personally I recommend using it first, before writing a SDK (unless you write the SDK and use it for something)

Just using cloak to do something

Now that kids are finally sleeping, I should go to sleep but I'm too excited 🙂

[mellow bolt]

I've used cloak to deploy dagger.io. Now I would love to get a preview URL on github (like Netlify)

ok

have a good sleep

[tepid nova]

@obsidian rover @mellow bolt I recommend watching this video if you haven't, it shows the potential of a query-centric collaboration workflow

[obsidian rover]

Thanks, it was lost in the messages. Would have missed it 😇

[tawny flicker]

Here is the issue + repro: https://github.com/dagger/cloak/issues/179

[obsidian rover]

Is anyone working on https://github.com/dagger/cloak/issues/144 ? I might be interested, but don't want to take it from someone 😇

[mellow bolt]

Do we have cloak implementation for Github action ?

Naive question: Is the current dagger-for-github repo can just use the cloak binary ?

[tidal spire]

Maybe someone more familiar with genqlient can help me out, I'm probably overlooking something simple here. How come gen/core/generated.go does not have an implementation for addSecret(plaintext: String!): SecretID!?

[tawny flicker]

Not yet, and cloak is not yet a replacement for dagger

[mellow bolt]

Ok, so maybe we could duplicate this repo and call it cloak-for-github

[civic yacht]

At the moment, generated clients only include methods for operations that have been explicitly defined (operations are different than the schema). For extensions under examples, you'll see the operations defined in operations.graphql. core is special because it's builtin to cloak, so the operations are found here: https://github.com/dagger/cloak/blob/a3aad6f11e994def546a954ccc238877d1e4b56b/core/core.schema.go#L65-L91

If we want AddSecret to be included in the generated stubs for core, we'd need to just add an operation for it there.

To be very very clear, this is all unnecessarily convoluted and just an artifact of how we could quickly get things going in the earlier prototyping phase. The approach outlined here should be our saving grace and let us skip defining custom operations that are just wrappers around really simple calls to schema: https://github.com/dagger/cloak/pull/174

[tidal spire]

Ah I see, thanks for the explanation! So in this case if we wanted an addSecret method to be generated, we'd need to add it to the operations here https://github.com/dagger/cloak/blob/a3aad6f11e994def546a954ccc238877d1e4b56b/core/secret.schema.go#L62-L70 or implement it in another extension

[civic yacht]

Yep! That's exactly correct

[wild zephyr]

@tepid nova this link returns 404 here https://github.com/dagger/cloak/pull/163#issue-1359797853

[tepid nova]

Thanks, should be fixed now

[hasty basin]

@stray heron @wild zephyr @hybrid widget @tidal spire @twin crow any of y'all considering demos/mini-demos for tomorrow's Cloak community call? Anyone else? 🙂 Even if you don't have a demo, we'd love to hear use cases, wishlists, half-baked project ideas, etc.
https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[wild zephyr]

I have a surprise demo Jeremy

[hasty basin]

Very well. The demo contents will remain a mystery until tomorrow 🪄 🔮 but we'll be waiting with expectation!

https://tenor.com/view/bard-cape-rainbow-gay-dungeons-gif-18646358

[stray heron]

Yes! I have something. I know that @mellow bolt has been working on something too. @tawny flicker may also have something for us. At least one of these three potential demos is going to happen. My wish is that all of us will share what we have, even if it doesn't fully work. Being stuck is especially helpful at this stage of Cloak.

[hasty basin]

excellent! I propose we start with simpler/working demos to give more new folks a feel for things and then get into the thornier things afterwards.

[tepid nova]

@hasty basin sign me up for a demo of my demoV3 mock-up featuring doc-centric DX 🙂

[hybrid widget]

Hi guys, I have a graphql query running in PHP

        query {
          core {
            image(ref: "php:8.1-apache") {
              exec(
                input: {mounts: [{fs: "{$workdirFsid}", path: "/mnt"}], workdir: "/var/www", args: ["sh", "-c", "cp -R /mnt/* /var/www/ && rm -rf /var/www/vendor"]}
              ) {
                fs {
                  exec(
                    input: {args: ["sh", "-c", "apt-get update && apt-get install -y curl git zip"], workdir: "/var/www"}
                  ) {
                    fs {
                      composer: exec(
                        input: {workdir: "/var/www", args: ["sh", "-c", "curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-scripts"]}
                      ) {
                        exitCode
                        stderr
                        stdout
                      }
                      apache: exec(
                        input: {workdir: "/var/www", args: ["sh", "-c", "sed -ri -e 's!/var/www/html!/var/www/public!g' /etc/apache2/sites-available/*.conf && sed -ri -e 's!/var/www/!/var/www/public!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf && a2enmod rewrite"]}
                      ) {
                        exitCode
                        stderr
                        stdout
                      }
                    }
                  }
                }
              }
            }
          }        
        }

It fails with this error dump

                    [message] => process "sh -c curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-scripts" did not complete successfully: exit code: 1

Any suggestions for why?

[twin crow]

happy to share what I am doing with Python, knowing it's 100% 2%-half-baked!

[twin crow]

pretty sure one of this exec is failing. I had trouble debugging those earlier. @civic yacht @wet mason - in case you have hints, I am interested as well

[tawny flicker]

That's what I got executing your query in my cloak dev:

{
  "data": null,
  "errors": [
    {
      "message": "failed to unmarshal fs bytes: illegal base64 data at input byte 0",
      "locations": [
        {
          "line": 4,
          "column": 15
        }
      ],
      "path": [
        "core",
        "image",
        "exec"
      ]
    }
  ]
}

for me, it must be because I don't have any FS var set up

(I actually don't know how to read from workdir from a graphiql instance, is it possible?)

[civic yacht]

@hybrid widget @twin crow Right now if an execop fails the output will not be included with the graphql error returned to you. You currently have to look at the buildkit progress logs from cloak, which will have output from the command that failed

So if you are using cloak dev you have to look at the output of that binary

[hybrid widget]

@tawny flicker I have obtained the FSID in a separate query

    public function buildContainerWithDeps() {

        $workdirFsid = $this->getWorkdirFsid();
        $containerWithDepsQuery = <<<EOF
        query {
          core {
            image(ref: "php:8.1-apache") {
              exec(
                input: {mounts: [{fs: "{$workdirFsid}", path: "/mnt"}], workdir: "/var/www", args: ["sh", "-c", "cp -R /mnt/* /var/www/ && rm -rf /var/www/vendor"]}
              ) {
                fs {
                  exec(
                    input: {args: ["sh", "-c", "apt-get update && apt-get install -y curl git zip"], workdir: "/var/www"}
                  ) {
                    fs {
                      composer: exec(
                        input: {workdir: "/var/www", args: ["sh", "-c", "curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-scripts"]}
                      ) {
                        exitCode
                        stderr
                        stdout
                      }
                      apache: exec(
                        input: {workdir: "/var/www", args: ["sh", "-c", "sed -ri -e 's!/var/www/html!/var/www/public!g' /etc/apache2/sites-available/*.conf && sed -ri -e 's!/var/www/!/var/www/public!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf && a2enmod rewrite"]}
                      ) {
                        exitCode
                        stderr
                        stdout
                      }
                    }
                  }
                }
              }
            }
          }        
        }
        EOF;
        
        $response = $this->query($containerWithDepsQuery);
        return json_decode($response);
    }

[civic yacht]

It is possible to include the output w/ the graphql error, but actually trickier than it may seem. I'll create an issue describing how we could do it since it's clearly needed once someone has time to implement it

[hybrid widget]

In my case I don't think I would have access to the buildkit log since I'm running cloak dev in a gitlab runner, not locally. BTW when I ran this PHP script/query with cloak dev locally it worked fine (AFAIR)

[civic yacht]

Ah okay, yeah at the moment not having access to the output of cloak dev is going to be an impediment. To workaround you could temporarily append || true to the end of the command that is failing, so then you will be able to get the output without the command failing

[hybrid widget]

In case it helps, this is the output I see in the gitlab output

#30 sh -c curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-scripts
#30 0.813 All settings correct for using Composer
#30 0.814 Downloading...
#30 2.842 
#30 2.842 Composer (version 2.4.1) successfully installed to: /var/www/composer.phar
#30 2.842 Use it: php composer.phar
#30 2.842 
stdClass Object
(
    [data] => 
    [errors] => Array
        (
            [0] => stdClass Object
                (
                    [message] => process "sh -c curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-scripts" did not complete successfully: exit code: 1
                    [locations] => Array
                        (
                            [0] => stdClass Object
                                (
                                    [line] => 12
                                    [column] => 15
                                )
                        )
                    [path] => Array
                        (
                            [0] => core
                            [1] => image
                            [2] => exec
                            [3] => fs
                            [4] => exec
                            [5] => fs
                            [6] => composer
                        )
                )
        )
)```

[tepid nova]

Looks like we're still need Dagger Cloud to debug even with cloak 🙂

[hybrid widget]

The weird thing is that it says

#30 2.842 Composer (version 2.4.1) successfully installed to: /var/www/composer.phar
#30 2.842 Use it: php composer.phar```
which is the expected output for a successful command, yet it indicates failure

[civic yacht]

This is how we can fix it: https://github.com/dagger/cloak/issues/182

which is the expected output for a successful command, yet it indicates failure
@hybrid widget Yeah that's odd, is the command exiting non-zero even though it succeeded? I'd try seeing if you do || echo $? if you get any output. Could also split the commands combined with && up into separate steps just temporarily to make it more clear which one is exiting non-zero

[hybrid widget]

Using ||true gave me some clues

[stderr] => Do not run Composer as root/super user! See https://getcomposer.org/root for details
Composer could not find a composer.json file in /var/www
To initialize a project, please create a composer.json file. See https://getcomposer.org/basic-usage

Again the weird thing is that the file composer.json should be there as it's part of the source repo. However it looks like it's not being found. I'm now checking further to see if I can figure out what's going wrong

[civic yacht]

The other thing we should do to help here is add the ability to drop into a shell on any execop (even failed ones) so you can poke around manually, run commands, etc. for debugging. If we do it right, should work both from command line and cloud.

[tawny flicker]

That would be superpower level. It something we really missed in CUE and could be a PITA at some point

[civic yacht]

I have alluded to it in the past but realized I don't have an explicit issue for it, just added one with a description of how to do it here: https://github.com/dagger/cloak/issues/183 It's surprisingly simple

[hybrid widget]

Figured this out, the issue was that I was running cloak dev from the wrong directory so the wrong set of files were being copied over to the image. Corrected and it works fine now. Next question is: once the image is prepared, how can I publish it to a registry?

[civic yacht]

https://github.com/dagger/cloak/issues/144 We haven't implemented that quite yet unfortunately

[tepid nova]

This reminds me that we should remove the use of cloak dev to run the embedded engine. IMO part of the "bundle buildkit" contribution thread.

/cc @civic yacht @wet mason

[civic yacht]

https://github.com/dagger/cloak/issues/47#issuecomment-1208324918 We'll still need a binary to run from languages that aren't Go. But I agree it shouldn't just re-use our debugging util. Should be a subprocess communicated with over anon pipes or a socketpair so we don't hog localhost

[tepid nova]

Yeah I just mean it should be different subcommands

(at the very least)

(also the port conflict you mentioned)

Longer term (but not that long term, launch horizon) there is the possibility that one moves into the user-facing client tool (dagger) and the other remains in a separate engine binary (cloak) to be embedded and instrumented by dagger. Not sure we'll end up doing that but it's plausible and would have advantages

[hybrid widget]

I read this and it looks like you can export to a local directory, though not an image What does this mean? Can that export be submitted to a registry or be input to a docker push command?

Just for context, my use case is: build the app container on Gitlab with cloak (done), then push to registry (pending) and deploy from registry on heroku (pending)

[civic yacht]

If you export a Filesystem to a local directory it just means the contents of the FS will be unpacked into that local dir. At the moment, we only support exporting to the root of your project workdir, so I suppose in your case that would mean to the place where you are running cloak on the gitlab runner, which is not really very convenient I'm guessing. Until we add the image export support, that's really all you can do though unfortunately. It's very easy to add support, we just haven't had time yet.

[tepid nova]

cc @tawny flicker 😇 👆

[hybrid widget]

I'm running cloak dev in the root directory of the cloned repo. So if I understood correctly, exporting the FS would unpack the entire image filesystem contents into that location?

[civic yacht]

Yes that's currently the behavior. As barebones as it gets for the moment. There's also a PR out that at least lets you specify a subdir of the local project dir, which I'm guessing would be slightly less annoying for what you are trying to do: https://github.com/dagger/cloak/pull/173

[hybrid widget]

OK I see

[tepid nova]

Speaking of which @civic yacht did you see the tweaked version of container push in 163?

[civic yacht]

Not yet, planning on taking a look again tonight

[hybrid widget]

After preparing and building the container image with cloak, what would be the next step I can take (with the currently available functions)?

[tepid nova]

FYI in latest iteration of new core API proposal, @hybrid widget 's query would look like this:

query($workdir: DirectoryID!, $repository: String!) {
 container(id: "php:8.1-apache") {
  withMountedDirectory(source: $workdir, path: "/mnt") {
   withWorkdir(path: "/var/www") {
    afterExec(args: ["sh", "-c", "cp -R /mnt/* /var/www/"]) {
     withoutDirectory(path: "/var/www/vendor") {
      afterExec(args: ["sh", "-c","apt-get update && apt-get install -y curl git zip"]) {
       composer: afterExec(args:["sh", "-c",
        """
        set -e
        curl -sS https://getcomposer.org/installer | php
        php composer.phar install --no-scripts
        """]) {
         afterPush(repository: $repository, tag: "composer") {
          ref
         }
        }
       }
      }
      apache: afterExec(args: ["sh", "-c", """
        set -e
        sed -ri -e 's!/var/www/html!/var/www/public!g' /etc/apache2/sites-available/*.conf
        sed -ri -e 's!/var/www/!/var/www/public!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
        a2enmod rewrite
        """]) {
       afterPush(repository: $repository, tag: "apache") {
        ref
       }
      }
     }
    }
   }
  }
 }
}

[hybrid widget]

I believe that even the rm -rf /var/www/vendor could be replaced with withoutDirectory(path: "vendor")?

[tepid nova]

Ah true 🙂

Maybe we could add afterCopy or something like that?

[civic yacht]

Until we have image export, I'm not there's any good options for your use case (deploy to heroku from a registry). All of them would take more effort than just adding image support. We should be able to have that extremely soon I think so we can unblock you

[tepid nova]

I didn't think of the case where you want to copy a directory to a new location on the same root dir

@civic yacht might be worth adding a binding to llb.shexpr given the prevalance of args: ["sh", "-c", ...] ?

[hybrid widget]

The reason I am trying to have this flow working is that I think it would be a great topic for a tutorial eg. "build/deploy your [language] app from gitlab to heroku with cloak". Once the image is pushed, then I can probably use the heroku cli to pull the image from docker hub and deploy it - but as you say, this is blocked until the image push function is available. In the meanwhile, I might try to replicate in github actions what I have done so far in gitlab CI...expect more questions soon! 🙂

In the meanwhile I'd appreciate any suggestions to improve this Gitlab CI script. It works but could probably be better

image: docker:20.10.16-git
services:
  - docker:20.10.16-dind
variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_VERIFY: "1"
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "/certs/client"
stages:
  - build

setup:
  stage: build
  script:
    - |
      apk add go wget php8 php8-curl php8-iconv php8-openssl php8-phar curl
      DIR=`pwd`
      cd /tmp
      git clone https://$GITHUB_USER:$[e-mail]/dagger/cloak.git
      cd /tmp/cloak
      go build ./cmd/cloak
      mv cloak /usr/local/bin/
      cd $DIR
      cloak dev &
      wget -qO- https://raw.githubusercontent.com/eficode/wait-for/v2.2.3/wait-for | sh -s -- --timeout 200 localhost:8080 -- echo 'cloak is up'
      php scripts/build.php

[civic yacht]

Yeah, to be clear the feature you're asking for, image export, is not some strange exotic thing at all, it's a really really basic feature that we just happen to have not had an explicit request for until now. That sounds like a great tutorial to have. But good to hear you have something else to look at while blocked. Please keep asking questions and requesting features for all the basic things we're still missing 🙂

[hybrid widget]

Thank you!

[tepid nova]

If I understand correctly the state of the art in gql client generation... there is no way currently to "flatten" the response by skipping intermediary structs and returning only the "bottom" of the query (the part with the scalar fields I want).

For example from this query:

query deploy {
  container {
    afterExec {
      file {
       contents
      }
    }
  }
}

I would ideally get this generated response type:

type DeployResponse struct {
  Contents string
}

And retrieve my value with resp.Contents

But instead I will get this:

type DeployResponse struct {
  Container struct {
    AfterExec struct {
      File struct {
        Contents string
      }
    }
  }
}

And I must retrieve my value with resp.Container.AfterExec.File.Contents.

This has 2 downsides:

1. It's verbose
2. It leaks implementation detail of the underlying gql query, which I may not have authored

Do I understand the situation correctly? And if so, is there hope? 🙂

I looked through the genqlient custom directive bud sadly, I didn't see any mention of this problem

[civic yacht]

I think this is the best hope https://github.com/dagger/cloak/pull/174

Worst case scenario and that doesn’t pan out for whatever reason (not expected) then I think I’ve heard of a @flatten directive that other tools have used to address this

[tepid nova]

genqlient has a @genqlient(flatten: true) but it seems unrelated sadly

[tepid nova]

Is that the "straight from graph" generation method, which skips the operations file entirely?

Or more like the Hasura style "create your own go types on the fly" ?

(I don't know anything about that PR sorry)

[civic yacht]

Yeah, the idea is that with a builder pattern like that you don’t need predefined operations in many (or most) cases because it’s just as easy to build them in the code directly

There are still cases where predefined operations are nice, just much less often

[tepid nova]

I believe that's how our blocklayer API client worked back from before we open-sourced dagger

I have memories of using that pattern and that was my only gql project

We need to go over the pros and cons of each generation method, and decide together which path to take. Doc-centric and query builder seem mutually exclusive and they have vastly different implications for the entire DX. I'd say that decision is in the critical path and should be next on the docket.

(on the flatten front: I found this https://github.com/chasingmaxwell/graphql-leveler

(looks abandoned, js only and pretty invasive syntax)

[civic yacht]

I don’t think they are mutually exclusive at all, they are probably very complementary actually. But agree that we need to discuss all that asap, if we are going to switch cloak.yaml->cloak.gql and the other related changes (which I generally agree with) we need to get that done soon before it starts blocking things

[tepid nova]

I guess you're right that a doc-centric DX doesn't necessarily entail using only generated types. But it does assume your queries are already written, and don't need to be created in code. You could still reference the pre-written query by name but build your response type in code with the builder pattern. But it would still suffer from lack of flattening: verbosity and abstraction leak would still be there, no?

Or maybe you meant that "flattening magic" can always be implemented as a convenience in either generated clients or a query builder?

[civic yacht]

More this. But gotta continue discussion tomorrow, getting late. Worth a live chat tomorrow if time

[rancid turret]

Is it going to be recorded?

[hasty basin]

yes!

[fickle sedge]

hi everyone, I was wondering if Cloak is released to public in the future and success, what will be the plan for the dagger/dagger?

[wet mason]

@fickle sedge dagger/dagger will still be the main thing. Plan still TBD on whether cloak code is ported inside dagger/dagger, or if cloak is a component of dagger/dagger

[hasty basin]

@here In about 15 minutes we have our next Cloak Community Meeting! Last week folks indicated that they wanted to see more Cloak demos and we've got 6 or 7 this week!! 🚀

See you at 10am PT! Agenda here: https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[wet mason]

Zoom link:

https://dagger-io.zoom.us/j/89360226840?pwd=aEZqU2Zsai9ScXhCcUwyZlp4RUpvQT09

[tepid nova]

That demo app @tidal spire 😂

[tidal spire]

Expect to see it return as I try other extensions like lambda deploys, terraform apply, and more

[wet mason]

Amazing. That's the thing we're missing the most: real world usage. The conversation the vault extension sparked regarding auth/configuration/tokens was really valuable, with terraform/aws we'll get a ton more data to draw conclusions

[tidal spire]

That is my mission

[tepid nova]

Thank you for the amazing demos @tidal spire @wild zephyr @twin crow @stray heron

[tawny flicker]

That was an INTENSE session!

[stray heron]

If there is interest, I can follow-up with the finished version in a few weeks, after all the travelling.

[tawny flicker]

Yeah, I missed part of the conversation, but I'm gonna re-watch it when the recording is published. I will need it very soon.

[tepid nova]

fyi @wet mason @civic yacht https://github.com/dagger/cloak/pull/163#issuecomment-1241092518

[wild zephyr]

@civic yacht @wet mason we should sync to align about the code generation / debug containers thing so we can keep adding more useful things to the playground for rapid extension / workflow prototyping

specially if the stitching proposal is going to be under review few a bit more time

also @dense dust tomorrow talk about the possibilities about embedding that in graphiql

[dense dust]

sounds good, that's what I'm working on

[civic yacht]

Yeah absolutely, I'm a bit meeting'd out at the moment, but happy to chat async or can chat live tomorrow, whatever works

[tepid nova]

@wild zephyr the code generation topic is going to take more than a call to align. It’s a major unresolved DX point, that deserves its own discussion thread. I can give you a quick summary if you want, it will help me think through it

[wild zephyr]

Sounds good Solomon. Let's have a quick call tomorrow to get more context about what the current challenges are. We can help by testing different approaches to see if we finde one that we like

[tight socket]

Where can I view some of the demo apps?

[tepid nova]

I’m writing it all in a discord thread right now 🙂 let’s start there

[tepid nova]

https://github.com/dagger/todoapp

there’s a branch called cloak-demov2

[tight socket]

Thanks, reading docs. I'll have a look momentarily.

[tepid nova]

There’s a demo video from vmware explore if that helps 🙂

[dense dust]

We also have the recording from today's community call right? That should be a great source of inspiration too

[fickle sedge]

Hey guys, i tried the cloak-demo-v2-extension and got this error:

yarn run v1.22.17
$ node scripts/deploy/index.mjs

                Netlify site name not specified in $NETLIFY_SITE_NAME.
                Defaulting to "tuongaz-dagger-todoapp".
        
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

Error: spawn cloak ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn cloak',
  path: 'cloak',
  spawnargs: [
    'dev',
    '--workdir',
    '/Users/tuongaz/dev/cloak-demo',
    '-p',
    './cloak.yaml',
    '--port',
    '8080'
  ]
}

any idea why? I have started the cloak dev and confirmed it's working

[dense dust]

Hey 👋
I'm reproducing it, will update

[bronze hollow]

Is there a recording? I've been having some airport issues and missed the call

[tepid nova]

Is cloak findable in $PATH ? It looks like the sdk can’t find the cloak cli

either that or can’t find cloak.yaml maybe?

[civic yacht]

Figured it out in thread, gonna update the docs #1017656093498937405 message

[tepid nova]

Ah! Glad you figured it out 🙂

Adding the cloak binary to our npm package would solve it neatly

Also will proactively avoid the inevitable version mismatches (sdk should bundle a specific version and update it on its own schedule)

[tepid nova]

Issue: https://github.com/dagger/cloak/issues/188

[civic yacht]

Agree that's the right way. I think python has similar abilities to include binaries. If we ever encounter a language w/out it I guess the fallback is to just put a URL+hash in the language's SDK to download the binary from, but can avoid that for as long as possible

[hasty basin]

We'll look to have it up tomorrow. May need some assistance from @cosmic cove or someone with a bit more Zoom privs than me 🙂

[tepid nova]

@wild zephyr @dense dust there is no scenario where having a cloud playground with graphiql + an interactive console ala chrome devtools is not AWESOME

I would use that all day every day

[hasty basin]

I'm building a bridge between classic Dagger (CUE) and Cloak without meaning to 😆 It was the easiest way for me to do a multistage build of the Jenkins Agent image with Cloak inside that I'm working on for a demo 😇 https://gist.github.com/jpadams/689ba6d655b506d02aab0a47986b6d4b

[rancid turret]

Here's another of those small GraphQL things that come batteries included: https://spec.graphql.org/October2021/#sec-Field-Deprecation. Context: https://github.com/dagger/dagger/issues/2251

[rancid turret]

Can't wait 🙂

Is cloak handling image metadata? Just see Filesystem.

[tepid nova]

Working on it, see pr 163 🙂 Type Container

[rancid turret]

Ah yes, I skimmed through that. I've been meaning to read that PR more attentively for the past couple of days but felt like it'll take a while 😛 I'll jump into it 🙂

I see the potential in query chaining, but for anything past basic you can get very deeply nested which feels a bit icky to me. "Flat is batter than nested" and all. Has anyone said anything about it?

[tepid nova]

No there has been no bikeshedding whatsoever

(that’s a joke of course)

[tepid nova]

in practice it works quite well. The whole point of graphql is graph traversal. In our case, traversal of the llb graph

So it maps directly to the llb api itself (chaining function calls). Feels weird in the beginning because of all the curly braces but actually works well

[rancid turret]

Yeah, makes sense.

[tepid nova]

Then when the query gets too long -> you collapse it with a graph extension. Literally a shortcut edge in the graph 🙂

[rancid turret]

Good point!

Cloak is turning out more powerful than I imagined, before July.

[tepid nova]

Yes I feel the same way 🙂

graphql started out as an implementation detail but has turned into a major design win

[rancid turret]

Yes, who thought it would be useful to have a QL for a DAG? 😆

The more I look at it the more it seems a good fit. I like your points on https://github.com/dagger/cloak/discussions/24

[wild zephyr]

shall we start consolidating the demos into #main? I had to use a branch for my yesterday's community call and it might not be obvious for some users where the latest code is

[tepid nova]

you mean for todoapp?

[tawny flicker]

Question about extension transitive dependencies: if an extension requires a dependency, does an extension or a cloak app requires to declare that transitive dependency as well?
(my hunch would be no, as it's mostly for code generation, and that code would only be required on the extension, not on the user of the extension itself)

[cosmic cove]

@bronze hollow and rest of <@&1003717314862129174> here is the recording from yesterday if you need it 🙂 The link is added to the community notes in the invite as well. https://dagger-io.zoom.us/rec/share/ZumLg6dMYuO_NFsWof1ktQuMb03XgC_ZzzuabE8aGHRQRCiboVGoOpZg6MYoQyTL.FJOZp1Ar2wEgRv05
Passcode: Zn.HGsV5

[tepid nova]

Transitive dependencies are not yet supported (you have to declare all dependencies yourself at the top level). We plan on supporting them eventually. (note: we need sn issue for that 🙂

[tawny flicker]

On it

[civic yacht]

Actually we've had transitive dependency support for a while now

[tepid nova]

What!!

[tawny flicker]

Oh, let me take that back, then :p

[tepid nova]

how, when

https://tenor.com/view/speed-gif-5593963

[tawny flicker]

yesterday, in 5mn when Erik took a break

[civic yacht]

I gotta go do the demo in a minute, but I did that week or two ago, can find the commit after

[tawny flicker]

(jk)

[tepid nova]

me trying to keep up with cloak development pace 👆

[tawny flicker]

Yep, that's my feeling as weel

[tepid nova]

don’t worry about finding the commit I’m just happily surprised 🙂

https://tenor.com/view/bh187-spaceballs-faster-speed-increase-speed-gif-19224699

[strong coral]

What passes arguments from cloak to extensions?

[strong coral]

Actually, that's not the actual issue. Someplace in the Go SDK is screwing up paths on Windows

I think

where does go build -o /entrypoint -ldflags -s -d -w come from?

found it