maintainers

[wet mason]

@strong coral That's the "Go Runtime" (when you use sdk: "go"), comes with its own default builder

[strong coral]

I "fixed" it https://github.com/dagger/cloak/pull/192

[hasty basin]

👋 @hot smelt

[tepid nova]

Welcome 🙂

@ancient kettle re: your message about Classic: yes I agree 100% that just porting to the Go SDK is best. Also will get transparent compat with zero changes to the cue code, which would be perfect

[ancient kettle]

Cool! Glad we're on the same page there. I'll likely spike on that today. It's already starting to feel a bit cumbersome to write gql queries in CUE

[tepid nova]

https://tenor.com/view/macintosh-happymac-mac-winkingmac-gif-21349986

[wet mason]

I'm getting a Error: exit status 1 from a fresh cloak build, anyone seen something similar? /cc @civic yacht

[wild zephyr]

yep

[tepid nova]

I think it's too early to move the cloak version of todoapp to main, since that would confuse Dagger 0.2 users. But we should move the working demo cloak-demov2 to just cloak and just remove the other branches to avoid confusion

Everything else should be on personal forks anyway

[rancid turret]

Yes, in my case I didn't have docker running 🙂

[wet mason]

oh yeah, it crashed

just realized this

[civic yacht]

buildkit crashed? or docker?

[wet mason]

docker

[tepid nova]

Actually @wild zephyr I'm going to just bite the bullet and reset the todoapp repo to not be a github fork. It makes it harder to manage branches

[wet mason]

@civic yacht i think it's just because the logger isn't plugged in

the buildkit bootstrap code uses zerolog

[civic yacht]

Yeah I ended up just replacing the zerolog stuff with fmt.Printf for now

[wild zephyr]

this

[civic yacht]

Do you have any issue w/ zerolog as we use it in dagger today? If not we can just copy-paste that over too, we really need a proper logger in cloak now (https://github.com/dagger/cloak/issues/162)

[civic yacht]

Do you mean put todoapp back into the cloak repo? I am good with that if so, just double-checking

[wet mason]

Yes. The buildkit progressui is not that great, and moving to zerolog requires porting progressui over

(otherwise they fight for the terminal, not pretty)

[civic yacht]

Oh right, okay yeah we still gotta do that but not exactly trivial

[tepid nova]

No, just recreate the same dagger/todoapp repo without the "fork of mdn/todoapp" metadata in github

[civic yacht]

Ahh okay, makes sense

[tepid nova]

Which makes it complicated (or at least unfamiliar) to create your own forks of dagger/todoapp, etc

I twice accidentally made PRs to mdn/todoapp for example 😉

https://github.com/mdn/todo-react/pull/33

https://tenor.com/view/shame-gif-26321140

[wet mason]

Yeah, was planning on asking Alex next week since I think he did something pretty good in bass

[civic yacht]

Yeah that would be perfect

[tepid nova]

A note on the progressui that might be relevant if you're planning to refactor parts of it. It's not actually great to get raw buildkit events (regardless of how they're formatted). I would much rather have graphql-specific events

ie events mapped to a specific path in the graph

[civic yacht]

Wrapping buildkitd opens up more possibilities here too, we don't have to forward the raw progress events to the client anymore

[tepid nova]

I'm mentioning this to make sure you don't engage in enormous effort just to cleanly produce an output we don't actually need in the end 🙂

[wet mason]

what do you mean? compared to dagger?

(e.g. wasn't that good enough to map bk logs to CUE paths?)

[tepid nova]

Yeah I think in the current dagger the output is pretty good. We can probably make it even cleaner now that we have simpler and more reliable paths to represent the current position in the DAG.

[tepid nova]

This is in progress

[wet mason]

Yeah, we do some "slight magic" to encode CUE paths in vertex names, we haven't taken the time to do that with graph path yet but could easily get the same dagger output we have with cloak

Although I think Alex is doing something better, would love to see his take on this

[tepid nova]

I just want to see the graph appear in real time in a web page 😄

With a big red flashing arrow pointing at the box that caused my pipeline to fail

[wet mason]

yep

[tepid nova]

and a button that says "open repl" 🙂

Then I can die happy

[wet mason]

so that's the main difference between dagger and cloak (TBD)

dagger: bk -> dagger -> terminal

cloak: bk -> cloak -> (graphql conversion of raw bk events into a graphql subscription?) -> client -> [terminal/web/whatever]

[tepid nova]

that + buildkit will be bundled so we have more flexibility in how we implement the bk -> cloak part

maybe doesn't change anything in practice

[wet mason]

cloak: bk -> cloak -> (graphql conversion of raw bk events into a graphql subscription?) -> client -> [terminal/web/whatever]

btw that could be another graphiql plugin -- query the API for logs and show them on a little box alongside /cc @wild zephyr @dense dust

so no need to switch between browser/terminal

[dense dust]

Agree, that's my main focus rn, using the websockets GraphQL server as a proxy between GraphiQL and cloak through subscriptions

Still have to figure out a few things though

[tepid nova]

@civic yacht can I remove your https://github.com/dagger/todoapp-old/tree/cloak-demo-v2-extension branch of todoapp?

[ancient kettle]

https://tenor.com/view/omg-happy-excitement-gif-21423437

[civic yacht]

I didn't think that was mine, I thought you were using it in the demo. So yes, I have no emotional attachment to it 🙂

[tepid nova]

Ah ok 🙂

https://tenor.com/view/wesley-crying-wesley-snipes-new-jack-city-snipes-crying-gif-18604810

good-bye demo branch you served us well

[ancient kettle]

https://tenor.com/view/rip-dr-evil-homies-gif-10430138

[tepid nova]

Last call before I remove https://github.com/dagger/todoapp-old

@wild zephyr going back to your original point: our demo is now available at https://github.com/dagger/todoapp/tree/cloak

[wild zephyr]

thx!

[tepid nova]

https://github.com/dagger/cloak/pull/163#issuecomment-1242328044

@civic yacht I've been playing with the cloak.yaml->dagger.gql idea you showed me. Trying to remove the yaml "scent" 🙂

query Extensions {
    yarn: git(url: "ssh://[e-mail]/dagger/cloak") {
        branch(name: "main") {
            tree {
                directory(path: "examples/yarn") {
                    id
                }
            }
        }
    }
    netlify: git(url: "ssh://[e-mail]/dagger/cloak") {
        branch(name: "main") {
            tree {
                directory(path: "examples/netlify") {
                    id
                }
            }
        }
    }
}

I kept the idea of a convention for query name, as an alternative to a custom directive

• realized that we can use aliases to group all extensions in one query

Maybe we could say "Capitalized query names are reserved for dagger use"?

Alternative with directives:

query yarn @extension {
    git(url: "ssh://[e-mail]/dagger/cloak") {
        branch(name: "main") {
            tree {
                directory(path: "examples/yarn") {
                    id
                }
            }
        }
    }
}

query netlify @extension {
    git(url: "ssh://[e-mail]/dagger/cloak") {
        branch(name: "main") {
            tree {
                directory(path: "examples/netlify") {
                    id
                }
            }
        }
    }
}

[civic yacht]

Awesome, yeah the idea makes more sense the more I think about it. Aliases work nicely, ends up looking like a config. But it also makes conceptual sense in that you can "query" which extensions are provided by this project. Also just way more cohesive than our current approach of slapping yaml on top of all our other concepts.

Maybe we could say "Capitalized query names are reserved for dagger use"?
I think it's pretty normal to give operation capital names, so I'm not sure if that would be intuitive.

Directives are another approach like you said. Or we could just reserve the names really and tell users they can't have other operations in here with that name. I'm not sure whether that would actually hurt users very badly if at all.

[tepid nova]

I'm trying to fit the rest of the cloak.yaml now. Might help figure out extensions

[tepid nova]

@civic yacht version 3, with a trick to fit everything in one operation.

mutation Init {
    init(configuration: {
        graph: {
            source: "./graph"
            sdk: "go"
        }
    }) {
        yarn: git(url: "ssh://[e-mail]/dagger/cloak") {
            branch(name: "main") {
                tree {
                    directory(path: "examples/yarn") {
                        id
                    }
                }
            }
        }
        netlify: git(url: "ssh://[e-mail]/dagger/cloak") {
            branch(name: "main") {
                tree {
                    directory(path: "examples/netlify") {
                        id
                    }
                }
            }
        }
    }
}

dagger.gql experiments

[ancient kettle]

@civic yacht Do you have a minute to chat with me about the generated go client? I think I'm not understanding or missing something.

[civic yacht]

Sure, how bout in 10 mins? Just finishing up something quick

[ancient kettle]

Perfect. Ping me when you're ready!

[civic yacht]

In cloak audio now

[tidal spire]

a little preview of what I'm playing with for the next demo 👀

{
  host {
    workdir {
      read {
        rails(runArgs: ["routes"]) {
          id
        }
      }
    }
  }
}

🤔

[tepid nova]

//go:generate dagger client --type gofile -o ./generated.go

😇

[tepid nova]

I am writing imaginary Go extension code, probably naive but trying to write what feels simplest to my non-graphql expert slef

[civic yacht]

@hybrid widget @hasty basin Merged very basic support for image export so you won't be blocked waiting for the upcoming API refactor first. Example:

{
  core {
    image(ref:"alpine"){
      exec(input:{args:["touch", "/hi"]}){
        fs{
          pushImage(ref:"127.0.0.1:5000/testpush:latest")
        }
      }
    }
  }
}

You can run a local test registry with docker run -d --restart=always --name registry --network=container:dagger-buildkitd registry:2. Note that use of --network, it's required because unlike dagger, cloak currently runs buildkitd in a non-host network namespace.

[tepid nova]

I'm remembering your comment @tidal spire about the vault extension not being able to "side load" its token env variable. I'm starting to think it might be OK to let extensions access the host (properly virtualized and sandboxed of course). Could be powerful for example if yarn { build } or netlify { deployment } could default to using the workdir if a directory is not passed?

same for vault which could default to an env variable if token is not passed

[tidal spire]

Yeah I think at least being able to specify which environment variables to pass through would be good. For the vault extension specifically we could assume the user wants to pass through VAULT_TOKEN. Its a bit more complicated for yarn where we might want to forward REACT_APP_*. It's a pretty easy solve on a case by case basis but maybe there's a more general solution at the core level

Maybe that's already something that exists which I've overlooked

[tepid nova]

We have a clean foundation to try at least. Every env variable lookup would go through a graphql query, so we can mock, namespace, prompt the user for authorization, analyze who looks up what, etc

The risk is that it becomes an implicit API, if everyone starts relying on env variables as the primary mechanism for passing information, rather than properly typed arguments.

But for default values / ambient context it's pretty appealing

[tidal spire]

Yeah this specifically sounds like a really bad state. I like the idea of querying host { environment { } } and/or mounting it to an image+fs, but maybe opening up environment over other forms of args/secrets isn't the way to go

[civic yacht]

What you mentioned previously @tepid nova in terms of the entrypoints to extensions (i.e. yarn, netlify) having args might offer something of an in-between option. Maybe those args could be special and default to being received from ambient context (but also can optionally be overridden by the caller). I don't know exactly what that would look like and there's risk of it being too weird/magical. But it could result in both the env being explicitly declared in the schema while also being passed implicitly at runtime.

(thinking out loud, this may make no sense)

[tepid nova]

it makes sense to me 🙂

Enjoy your weekend everyone

[hasty basin]

Working on getting cloak working on Jenkins. I've done something similar in the past with Dagger 0.2, but running into issues at the moment.

I build some Jenkins agent images with cloak binary and snapshot of the cloak repo at /cloak. I'm not able to run anything beyond cloak version though.

When I tried to run something like cloak do -f /cloak/examples/queries/docker_build.graphql I just got back Error: exit status 1

realized later that I needed a cloak.yaml around

did a bunch of irrelevant thrashing

🌊 This could be due to a simple oversight since I'm pretty tired from running around with 12 year olds at a waterslide park today and my brain is a bit fried 🤣 🩴

I'll probably rebuild the agents with dagger in them as well and see if dagger still works like before in this setup.

[hasty basin]

I ll probably rebuild the agents with

[hasty basin]

Got it working.

the tl;dr: #1018405915277402122 message

[tepid nova]

Nice 😎

[hot smelt]

Hello, Thanks again for giving me access to cloak ! I have started working with it directly via a go file https://github.com/laupse/cloak-experimentation/blob/main/ci.go. But i'm stuck with this :

#1 local://__cloak_workdir
#1 transferring __cloak_workdir: 853B done
#1 DONE 0.0s

#2 copy / /
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:1.19.1-alpine3.16
#3 ...

#4 [internal] load metadata for docker.io/library/alpine:3.16
#4 DONE 1.9s

#3 [internal] load metadata for docker.io/library/golang:1.19.1-alpine3.16
#3 DONE 2.2s
error failed to solve: input:5: core.filesystem.pushImage failed to unmarshal result: unexpected end of JSON input:

[wild zephyr]

👀

[tepid nova]

@wet mason @civic yacht putting the finishing touches on the core API before merging.

Quick Q. the current proposal removes the ability to create a secret directly from a cleartext value passed in the query. You have to create it either from a file (on the host or in a container), or an env variable (on the host).

The reasoning is that the whole point of a Secret type is to make handling of secrets safer by reducing the exposure to the cleartext. For example if cleartext is never in the graphql query or response data, then you are less exposed to leaks, etc.

In that context, passing cleartext in the query seemed like a footgun that we could avoid simply by not implementing the capability.

WDYT?

[civic yacht]

There are probably use cases where you want an extension to create a new secret from something it generated in memory. E.g. the extension could run ssh-keygen or similar. In that case your only option would be to write that plaintext to a file and then create it from that file, which would actually be less secure than just being able to do addSecret with the plaintext (in which case everything just stays in memory). So I feel like it would be better to just have an addSecret that accepts plaintext

[wet mason]

I think that makes sense overall, however I think that would prevent @tidal spire's Vault extension to exist (please confirm?)

That being said, they way we have to do Vault today is very limiting (again, @tidal spire, please confirm):

• Secret stores would rather for secrets to be requested "Just In Time" when accessed (otherwise it'll break rotation, auditing, etc)
• Current solution requires plaintexting back & forth (the footgun)
• BuildKit is also designed in the same way (Just In Time) -- we have our own custom store to hold secrets in memory, that's the footgun holster

[tepid nova]

@hasty basin I'm also tweaking the secrets API to address your initial confusion. Will look like this:

host { variable(name: "MY_TOKEN") { secret { id } } }

host { workdir { file(path: "./my-token") { secret { id } } }

[tepid nova]

Wouldn't the vault extension be able to read it from a file on a container?

[tidal spire]

Confirmed (and +1 to what Erik said). I think you're both saying that there should be a special case just for secrets engines, so maybe that's another path to explore aside from env/file

[tepid nova]

Or perhaps stdout?

Plaintext in a file an ephemeral container (ideally written to a tmpfs) seems safer and cleaner than passing it in the query IMO

[civic yacht]

I guess I am missing where the footgun is that can replaced with something less footgunny. Even if we have a way of having a service that can provide secrets just-in-time, the secret will still need to be sent over the wire (completely in memory).

[tepid nova]

Yeah the only part where I disagree is that writing plaintext to a file in a tempfs would be less safe than sending it over the wire. In both cases they're in memory, but in one case leak risk is much higher

[wet mason]

Vault extension is using the Vault SDK, not a process. Even then, that would require to write the plaintext secret to a file, which ends up in layer caching etc

[wet mason]

If it's a tmpfs then it's gone, we can't read it

(@civic yacht ^^^ true?)

[tepid nova]

Ah right. I got over-excited about the possibilities of "get-mount"

[civic yacht]

Yeah you can't get a tmpfs after execution. I mean, in theory that would be possible to implement, but would require upstream changes

[wet mason]

I think given caching etc, we can only "read" stuff that ends up in the cache, so if you can read the secret, then the secret exists in plaintext on a layer somewhere

[tepid nova]

OK I see how we have an unresolved issue of making "secrets providers" possible in a clean way.

I'll leave the capability out for now, just because it's easier to add later than remove. To be continued 🙂

(I just mean leave it out from this particular PR we can of course make a follow-up PR at any time)

[civic yacht]

The leak risk as I see it is that we have a bug in our code (because addSecret is a core api and thus controlled completely by us). That's not to discount the risk, it's better to find a way that minimizes our opportunities to make mistakes, but I don't know if anything else is actually simpler at the end of the day.

[civic yacht]

That sounds good, can discuss it more separately

[wet mason]

@civic yacht Ugh. That actually makes me think ... does it actually work today?

e.g. "vault get the netlify token" is cached. so it won't run. we'll get back a SecretID from the extension call, but it's nowhere to be found, right?

I think there's something fundamentally wrong with addSecret/caching/etc

[tepid nova]

I have another unrelated question.

Since the client-side Go DX seems to throw off people (because of the mismatch between request and response types).

• We have a long-term solution with the query builder approach, but still lots of work to get it to work properly (quoting @wet mason )
• Meanwhile no clear way to fix the DX of the current "pre-written queries" approach.

--> Is there a stopgap we could implement to improve the situation in the short term? I was thinking of the "on the fly structs" approach that I think Hasura uses, and I believe we used in the Blocklayer client @wet mason ? It's not perfect but at least it's consistent, and it's kind of the least painful version of raw graphql queries in Go.

WDYT?

[polar ember]

Hi. I've begin my cloak journey . What the error below mean ?

[civic yacht]

The cloak.yaml is using workflows: which isn't recognized. The docs have up to date examples of cloak.yaml: https://github.com/dagger/cloak/blob/main/docs/guides/y0yh0-writing_extensions_go.md

You most likely meant scripts: there. Also, just in case you don't know, bash is not (yet) a supported SDK, so specifying it won't result in anything being generated. Issue for bash support is here: https://github.com/dagger/cloak/issues/44

[polar ember]

Ok. Thanks Erik. So, do I need to write extension before setting up a simple pipeline (go mod tidy && go build . ) ? The use case I'm trying to implement is to build and deploy (scp ) go application to remote server.

[tepid nova]

FYI last-last call for https://github.com/dagger/cloak/pull/163, I'm going to merge it after my lunch break. We can keep evolving it, but we're close enough to consensus that we can start some implementation work in parallel IMO (cc @tawny flicker 🙂

[civic yacht]

Ok Thanks Erik So do I need to write

[tawny flicker]

wrapping your dependencies is always good. Better testability, maintainability. I'm all for it

[tawny flicker]

Personally, I would think it best to have every input passed by function and not side loaded.
It would make the whole inputs explicit and would avoid leaks, IMHO

[tepid nova]

That's current situation. Everything else is pure speculation at this time 🙂

[ancient kettle]

@civic yacht I'm running this... approximately (it's embedded in a bunch of dagger code)

ir, err := core.Image(ectx, rawRef)
fs := ir.Core.Image
//saving fs.ID

// later 
resp, err := core.Exec(ectx, dagger.FSID(fs.ID), core.ExecInput{
  Args: args,
})

I'm getting this error...

failed: fs "eyJkZWYiOlsiR2pBS0xtUnZZMnRsY2kxcGJXRm5aVG92TDJSdlkydGxjaTVwYnk5c2FXSnlZWEo1TDJGc2NHbHVaVHBzWVhSbGMzUlNEZ29GWVhKdE5qUVNCV3hwYm5WNFdnQT0iLCJDa2tLUjNOb1lUSTFOanBrTWpCa1lXRXdNR1V5TlRKaVptSXpORFZoTVdJMFpqVXpZalppWWpNek1tRmhabVUzTURKa09HUmxOV1UxT0ROaE56Wm1ZMlF3T1dKaE4yVmhNV014Il0sIm1ldGFkYXRhIjp7InNoYTI1NjpkMjBkYWEwMGUyNTJiZmIzNDVhMWI0ZjUzYjZiYjMzMmFhZmU3MDJkOGRlNWU1ODNhNzZmY2QwOWJhN2VhMWMxIjp7ImNhcHMiOnsic291cmNlLmltYWdlIjp0cnVlfX0sInNoYTI1NjpmMDc1Y2MwN2RiMWFhNWNhZWI2Mjk2YjBlMGM3M2M2NTIxMzJkNjM1MjE4M2VmODZkYTE4NWM3MmQxNzc4OGMxIjp7ImNhcHMiOnsiY29uc3RyYWludHMiOnRydWUsInBsYXRmb3JtIjp0cnVlfX19LCJTb3VyY2UiOnsibG9jYXRpb25zIjp7InNoYTI1NjpkMjBkYWEwMGUyNTJiZmIzNDVhMWI0ZjUzYjZiYjMzMmFhZmU3MDJkOGRlNWU1ODNhNzZmY2QwOWJhN2VhMWMxIjp7fX19fQ==" not found

Ring any bells?

I've verified that I'm using the same FSID

[hasty basin]

Jenkins users or recovering Jenkins users 😄

For a demo I'm working on, I want to show how teams could run the same Cloak workflow locally and in Jenkins.

I'm thinking of having a nodejs dev team use yarn or npm to run cloak locally from scripts in their package.json and for the same cloak script to be run in CI by running a shell step like: https://github.com/jpadams/helloworld-dagger-jenkins/blob/cloak/Jenkinsfile#L19 or via a Jenkins Shared Library.

The shell invocation works because my Jenkins agent images (using container agents via Docker plugin) have cloak pre-loaded on them.

The project could alternatively have a Jenkinsfile that invokes a shared library function which calls cloak. Maybe there is a way for shared libraries to pull in code that is in a a git repo, so it could target the same repo where devs have their package.json?

[ancient kettle]

@civic yacht I've got simplified #Pull and #Exec working in the dagger/dagger codebase on top of cloak. Gonna be a ton of stuff to pull out of this codebase, honestly, but I think it should just work, once I've got all ported over.

🙂

[civic yacht]

Amazing! Awesome to hear it's going so well, pretty incredible that porting has been relatively pain free (other than the known cloak pain we have to fix for everyone anyways)

[ancient kettle]

yeah, it really hasn't been hard thus far. I'm sure there's going to be some API edge cases and differences that I'll need to work around... but 🤞🏻 that it's not too bad.

[tepid nova]

https://tenor.com/view/classic-youre-classic-thats-classic-haha-classic-so-classic-gif-4807951

[hot smelt]

Hello, I'm trying to do the same but i'm getting http: server gave HTTP response to HTTPS client. Didn't you get this error too ? I try to move registry to https and even installing ca-cert inside dagger-buildkitd container but can't make it worked.

[obsidian rover]

Hi,
Did you spawn your buildkit instance with https://docs.dagger.io/1224/self-signed-certificates/ ? (with the --network mention above

[tawny flicker]

Does the cloak netlify Go extension still works?
I'm failing to see where the FS containing the static site is pushed/passed

[hasty basin]

Got some more feedback on the jenkinsci Gitter: https://gitter.im/jenkinsci/jenkins?at=631fd53272ad51741fd9edcb

[mellow bolt]

Hey guys 👋 ! How can I pass a secretEnv inside an input args:
this doesn't work.

core {
  image(ref: "index.docker.io/dockette/vercel") {
    exec(
      input: {
        args: [
          "vercel"
          "--name"
          "dagger_io"
          "deploy"
          "./out"
          "-t"
          "${VERCEL_TOKEN}" <---------- my env
          "--scope"
          "dagger"
          "-c"
        ]
        mounts: [{ fs: $sourceAfterBuild, path: "/src" }]
        workdir: "/src"
        secretEnv: { name: "VERCEL_TOKEN", id: $tokenSecretId } <---------- my env
      }
    ) {
      stderr
      stdout
    }
  }
}

thanks

[wild zephyr]

Hey guys 👋 How can I pass a secretEnv

[tidal spire]

ok here's an easy question hopefully. I'm creating a new go extension under cloak:/examples. I have a cloak.yaml, operations.graphql, and schema.graphql. When I run cloak generate, it doesn't generate a main.go, models,go, or generated.go. It does create the gen/ directory, but it just contains my operations and schema. What step did I miss?

[cosmic cove]

<@&1003717314862129174> Hi everyone! I hope y'all had a great weekend. I look forward to seeing you at the community call on Thursday. If you have any topics or discussion items that you'd like to add to the agenda, please DM me or add it here -https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#heading=h.isyrk6ea5tjx

[tidal spire]

for my next trick... 🚀

> yarn run terraform
...
#37 terraform plan
#37 1.308 Running plan in Terraform Cloud. Output will stream here. Pressing Ctrl-C
#37 1.308 will stop streaming the logs, but will not stop the plan running remotely.
...
#37 23.22 Plan: 16 to add, 0 to change, 0 to destroy.
#37 45.64 
✨  Done in 50.65s.
➜  cloak-ruby git:(main) ✗

[tepid nova]

Call for feedback: a clarified and simplified introduction to the cloak docs; based on what we learned in the last 2 weeks (a lot!!) https://github.com/dagger/cloak/pull/204\

It's less content than before, because I removed everything that was no longer correct. I also wrote some new content but not as much as I removed.

https://github.com/shykes/cloak/blob/docs-cleanup/docs/unxpq-introduction.md

[still garnet]

@civic yacht nerd sniped me into a Nix/Buildkit comparison from the Buildkit side, if anyone's interested in that (feedback/clarification welcome!): https://github.com/dagger/cloak/discussions/135#discussioncomment-3639462

[cosmic cove]

@warm tundra I'd love your thoughts on how we explain extensions here. Does the explanation and diagram ease the confusion that we were discussing this morning? If not, what do you think is missing?

[warm tundra]

wow, I gotta start using mermaid in my own docs! I'd write more flowcharts if it were that easy to write flowcharts.

on-topic: I think this clarifies well what extensions can do, but leaves some questions about what they should be used for. I was turning this over in my head a bit earlier: the reason the "extension" terminology trips me up is that it sounds like a heavier abstraction; a way to share things between projects. I'm looking for a smaller unit of code, something that you would presumably use within an extension.

by way of analogy: a single, importable file in JS/TS is a "module," while an intentionally arranged collection of them, especially if published, is a "library." extension : library :: ? : module

Concretely, the build configuration for my backend is interesting and potentially reusable within the context of my monorepo -- because other services could plausibly have a dependency on its build artifacts -- but likely not outside of it. I should be able to import and use it trivially, in the same way that I can define and export a component in my React app and then import it into another component -- even if I wasn't originally building that component as a distributable, standalone thing. Does that make sense?

[tepid nova]

I think the "module" will be a custom pipeline step

we should de-emphasize the part about "writing your own extension". IMO that's the part that's confusing. An extension is something you install on top of your own work (to extend it)

So in your case:

• Your backend would be a Dagger project
• In the context of this project you can install extensions (collections of custom pipeline steps to keep your own pipelines simpler).
• Then you can develop pipelines and embed in your various tools and scripts
• Eventually you will want to create custom steps of your own. Perhaps to simplify your scripts, or perhaps for other projects to reuse yours as an extension. This is where you start writing your project's API

I think you would typically have multiple Dagger projects in the same monorepo. Each with their own API and installed extensions. Possibly some projects will use others as extensions.

I will propose this as a topic for the next community call 🙂

Thank you very much for your feedback! This is all still work in progress for sure

[warm tundra]

"using other projects as extensions" is particularly interesting to me -- I see that as core to what I want to do with Dagger, enough so that I'd want it to be a very streamlined use case.

[tepid nova]

Yes! absolutely core

our goal is to make it 10x easier than it is today

[warm tundra]

I'm envisioning being able to do stuff like

import { binary as backendBinary } from '@mymonorepo/backend';

...

export async function dockerImage() {
  return alpine.copyLink(backendBinary, '/var/app/bin');
}

I haven't actually used cloak yet so I don't know the actual syntax, but you get the idea

better still if @mymonorepo/backend could be in Go, but the imports and such would still work in a TS context and vice versa

[tepid nova]

it’s slightly different because you’re always composing your pipelines by querying the dagger api. So it’s the api server that’s loading extensions then allowing you to query them

[warm tundra]

I think in my mind backendBinary (maybe I should've written it as a function) was something that would itself produce a query, as would alpine.copy(). So this would all be a query at the end, but in writing it I might not really be thinking of it that way

[tepid nova]

the problem with that approach is that you get a matrix from hell between all language package managers and type systems

we can’t solve that properly . api extensions are a shortcut to dodge the problem entirely

[warm tundra]

Do you see writing GQL as the intended path for most users? Or do you imagine the typical user writing the queries through a wrapper of some kind?

[tepid nova]

that way every project can use every extension seamlessly regardless of language, without messing with hard problems of language interop

[tepid nova]

we want outr answer to be “whatever the state of the art of gql is in your language “

the goal is to avoid reinventing the wheel at all cost

[warm tundra]

so if I understand:

• each project has its own GQL API/schema
• the valid GQL schema for a project is the Cloak core + the loaded extensions in that project
• the language-native GQL abstraction can wrap the core and the extensions as it would wrap any other GQL API
• in statically-typed languages, this wrapping might entail a "generate" step where the project-local tooling "catches up" with changes to the APIs (analogous to how my frontend dev server always starts by running a graphql generator based on the current state of the schema shared with the backend)

right?

[tepid nova]

exactly right 🙂

[warm tundra]

that sounds fantastic

[wild zephyr]

👋 we just merged a nice feature (https://github.com/dagger/cloak/pull/198) which adds a new query explorer to the graphiql playground. This makes the experience of discovering and writing queries way better. We'll feature this on the community call, but if you want to start playing with it and provide feedback, it's already available in cloak's tip.

[wet mason]

I understand now the pain of text/template 🙂

[wet mason]

@tepid nova @civic yacht @hasty basin re: the secrets conversation

I just made a quick and dirty POC for secret providers: https://github.com/dagger/cloak/pull/210

(/cc @tidal spire)

There's a few (large) caveats around caching etc, but basically solves the need to call addSecret, does "just in time" retrieval of secrets, and there's an extension point

... cloak queries itself ... it's a bit mind boggling though 🤯

[tepid nova]

Andrea is messing with our brains again

[civic yacht]

This is needed for dagger.gql anyways so I was glad to see it is easy

[tepid nova]

https://tenor.com/view/-gif-7960871

[civic yacht]

@here If you are currently using Go for cloak scripts/extensions, a change to remove our current client codegen was just merged: https://github.com/dagger/cloak/pull/206

If you are just using JS/TS then this shouldn't affect you.

Now if you run cloak generate you will no longer get client stubs for your dependencies and instead have to make raw graphql queries (examples of those here: https://github.com/dagger/cloak/blob/62310f84a076202280fcc2f7b6b000c5cb8098db/examples/alpine/main.go#L32-L101).

cloak generate still helps out with creating extensions by creating models.go for you and some other autogenerated boilerplate, but is now more limited in scope than before.

There's some more rationale behind this in the PR, but basically the codegen we were using before was a third-party tool and had some quirks that seemed to lead to as much confusion as it helped. This confusion was poised to grow much even more in the near future with some upcoming API changes, so we decided to just rip the bandaid off now and revert back to the raw graphql client. That raw interface is tedious to use, but is at least more obvious.

Fortunately all the low-level plumbing to support client codegen is still in place and we plan to return with better support for it in the future via custom frameworks better tailored to our use case. But let me know if there's any concerns or questions about the raw graphql client in the meantime.

[twin crow]

Some teaser, today I was able to execute my first helloworld python extension via the sdk (declared via a native Python class). Still a long way to get this to a clean state, but it's moving along! (https://github.com/dagger/cloak/pull/177 - if anyone wants to see the dirt under the carpet)

currently using strawberry types declaration, but there is nothing specific to the framework in the way the code is called, so technically it should work with any graphql framework easily. Strawberry has a nice feature for automatically generating graphql schemas from python code that I plan to leverage very soon.

[ancient kettle]

I'm guessing this breaks the client I'm using in dagger-classic?

[civic yacht]

Technically the client still works, it's just that you won't get any more updates from cloak generate now. So it will most likely make sense to start moving over to the raw client.

[tawny flicker]

Why is AddSecret not available anymore in core?
I'm trying to get secret, but I'm miserably failing

From what I understand, core.Secret() gets the secret data from its ID.
But I don't know how to obfuscate a plaintext data behind a secret ID.

"old" (4 days ago) examples use AddSecret, but I don't have that available.

[scarlet gate]

@civic yacht & @tepid nova I am back from my vacation.
Today I had my session with our Salesforce Team. They explained their pipeline to me.
The goal is to transfer their hacky yml stuff to cloak.

I lost the overview in the last two weeks. So I have to start from scratch. 😅

[scarlet gate]

@obsidian rover can we start a thread, so we don't produce to much noice?

[obsidian rover]

The_Automator pairing

[mellow bolt]

Maybe it's something known, but I'm facing the following issue. I've created a new extension Vercelwith its own schema and operations. I've used the same schema type as the Netlify one :

type SiteURLs {
  url: String!
  deployURL: String!
  logsURL: String
}

My project load both Netlify and Vercel extension:

  - git:
      remote: [e-mail]:dagger/cloak.git
      ref: main
      path: examples/netlify/ts/cloak.yaml
  - git:
      remote: [e-mail]:slumbering/cloak.git
      ref: vercel-extension
      path: examples/vercel/ts/cloak.yaml

Then I get the following collision error:
Error: failed to solve: input:24: core.filesystem.loadProject.install conflict on type "SiteURLs": "url": field re-defined

Is it something that you're aware of ? Should I open an issue ?
cc @wet mason @civic yacht

[tawny flicker]

I guess there is no namespace by default in graphQL. We might have to think about it for those cases where the API is nearly the same and some naming conflict could arise

[obsidian rover]

Hi, do you know how to make a conditional build on a specific line of code, in Go ? I want a specific line of code to exist only on linux

[tawny flicker]

mmmmh, you want macros!

let's pair

but you will have to split into a func that can be protected by build tags

[obsidian rover]

Thanks for the help, it unlocked me 😇

[wild zephyr]

This is the way :mandalornian:

[tepid nova]

@tawny flicker namespacing is a major topic. There is no easy answer:

• Either 1) we implement name mangling, solving conflicts but making extension code less readable (because the type in your code does not match exactly the type in the final extension, eg. SiteURLs will map to VercelSiteURLs

• Or 2) we leave it as is, conflicts can and will happen, and it's up to developers to follow best practices (ie "always prefix your types!")

I think with the new emphasis on code-first XDX, option 1 becomes more viable (you're not writing gql schemas directly so it's less weird that we generate a slightly different grapqhql name). Another benefit of option 1 is that you can load several versions of the same extension under different names, and they won't conflict with each other

[civic yacht]

(I'm in the middle of creating an issue for this too, just to dedupe effort here)

[tepid nova]

lol I was literally opening a tab to do this. Thanks 🙂

[tawny flicker]

lol

[tepid nova]

I encourage everyone to do this: when discussing a missing feature, bug or design decision. Just take 2mn and write the issue. You'll be glad you did later.

[mellow bolt]

I wanted to do it but wasn't sure

noted for the next time

[tepid nova]

New issue: "remember to create an issue next time"

[mellow bolt]

ahahah

thanks @civic yacht for the issue

[civic yacht]

Yep! Feel free to add more context @tepid nova , I was saying essentially the same thing but with less details: https://github.com/dagger/cloak/issues/212

[civic yacht]

Agreed, in the past when there was a revolution every day it was less productive but I think we are at the point that the issues we talk about usually remain relevant and are brought up independently multiple times, which is in a way a good sign!

[tawny flicker]

organized procrastination

[tepid nova]

Exactly, the timing is right

I have docs-related question. Obviously we will need an API reference, generated from the code itself.

Are there other opportunities to have documentation content directly embedded with the code? In my experience that content is easier to keep up to date and accurate.

The other one is examples, which I know @wild zephyr is taking a close look at.

Could we go further?

[tepid nova]

"A revolution a day keeps the launch away"

[civic yacht]

In terms of "directly embedded in the code", with code-first xdx we could definitely support the equivalent of how go docs work where a whole webpage is generated from comments in the source code. We have that to a limited extent today when we leave "Comments" on our graphql schema, but you could take that much further. For go extensions, it could literally just be that we enable the generation of the go doc webpage and then have an api that serves it to you (using the service stuff @wet mason and @wild zephyr are working on).

I know there's similar things in other languages

Also, Extensions are just Filesystems containing source code at the end of the day, which means that you are free to include whatever content you want alongside their source code. So if there is a README.md in there we can serve that to users too. We can also literally show you the source code (sometimes that's the best documentation)

[wet mason]

README.md alongside the extension (including core) could be an option. That seems like the “obvious” choice (e.g. how vscode extensions do it, npm packages, etc etc). Maybe there’s a better approach but seems like a good fallback

[civic yacht]

Yeah it's the most basic, generating and serving doc web pages would be much more advanced (though in reach given how powerful the service concept could become)

[tepid nova]

May the bikeshedding begin! https://github.com/dagger/cloak/issues/213

https://tenor.com/view/madmax-thunderdome-tinaturner-gif-5505388

[tidal spire]

I also have the terraform extension ready for review and feedback! https://github.com/dagger/cloak/pull/211

[civic yacht]

@obsidian rover I thought a little bit more about how to distribute the buildkitd wrapper. I think the idea I mentioned before in terms of embedding the source code is probably too crazy for its own good and we should go the more straightforward route, but wrote out both options here. Let me know what other thoughts you have on it: https://github.com/dagger/cloak/issues/47#issuecomment-1247066547
(anyone else feel free to chime in too)

[wet mason]

do you think we should do the same for the shim?

[civic yacht]

You mean replace the thing where we embed the source code with an actual image or similar idea? Yeah, probably, though it's lower priority in my mind. The shim is less of a problem because we actually have buildkit up+running when we need to build it, so it's more straightforward. Part of the ugliness of taking that same idea and applying it to cloakd is that cloakd is our buildkitd, and thus we need to use a different builder to bootstrap it.

[wet mason]

@civic yacht Also I think there's a B.2 option:

We embed the binaries (multi platform) rather than the source

Unsure exactly how, but handwavy way we tweak the build system to build cmd/cloakd (or whatever the name is -- buildkit wrapper) for all archs we support, then we build cmd/cloak which go:embed's the binaries

it kinda "smells bad", but makes it self-contained without the need to pull stuff from a registry

or B.3, same as B2. but we go:embed the OCI image and docker load it

(basically the binary acts as its own tiny registry)

[civic yacht]

It's definitely possible, but in my mind it would have to be just a temporary solution that allows us to avoid a registry. Once you consider every platform we'd want to support (x86, the various arm variants to start, later more exotic things probably) the binary will end up being really huge

It also means that it's much harder to build the cloak binary since it has to be done in steps

[wet mason]

in the good old days of pure linux/amd64, we could have bind mounted dagger into a container and started it (that's how the docker "shim" worked in the early days)

[still garnet]

+1, i went through that with bass's shims and those are relatively tiny, you can shrink each binary with upx before embedding which helps quite a lot, but it's still pretty hefty with more platforms to support

[wet mason]

yeah I actually looked through bass's codebase to see how you were doing it 🙂

[still garnet]

i initially just embedded source and built it at runtime, but it felt awkward to have a language coupled to e.g. the golang image on Docker Hub

[wet mason]

we're doing something more basic, just go:embed the source of the shim itself and build it on the fly (pre-pend the shim build to the actual llb.Run)

[still garnet]

gotcha, yeah sounds similar

[wet mason]

since it's a single main.go file, no dependencies on internal packages

[civic yacht]

Yeah... if we rewrite in an interpreted language then that's possible! But let's not do that

[wet mason]

cloak.jar

could auto-play some music on spotify while the shim is booting

[still garnet]

lol, you'd get halfway through Lateralus

[civic yacht]

Is your main hesitation @wet mason avoiding the registry management? I agree with that concern, that's why I was thinking about what else we could possibly do, but I feel like all the other options end up being temporary solutions with big downsides

And also pushing a multiplatform image to a registry from GHA in our cloak repo is a good dogfooding opportunity

[wet mason]

@civic yacht it's part that, part version coupling, part dev/"prod" difference

but I don't think it's worth it, just wanted to point out all alternatives

[civic yacht]

I think if we expand on the whole "provisioner" concept and make pulling the image just one option among many ways of provisioning cloakd (with common ways of dealing with version matching, etc.) it may start to be more cohesive and sensible. But that would require more design, it feels right in my head but its vague

[wet mason]

yeah definitely

[tepid nova]

Are we on the same page that "cloakd" will really be dagger ?

[wet mason]

yes

[tepid nova]

(ie everything bundled in the same binary for simplicity)

[civic yacht]

Yeah @obsidian rover is making this all subcommands of cloak

I think it's cloak buildkitd at this exact moment, but that's just a placeholder

[wet mason]

just less confusing when talking until we have "names"

[wet mason]

does it have to be different than cloak serve or whatever?

(e.g. can we serve gRPC and GQL over the same socket?)

[tepid nova]

Benefit of fetching the shim from our registry: we get a rough estimate of how many engines are running out there. Not an accurate number any stretch (caching, shared IP addresses etc) but at least something we can plot on a chart, and share it at community calls, and feel good if it goes up 🙂

[wet mason]

or another socket with like --listen-clustering-api :1234

[civic yacht]

No probably not, we are just starting with a complete copy-paste replacement of buildkitd though, so it makes sense at the moment, but once we get to the next steps of actually refactoring where the engine runs and does stuff we can combine the various commands

Agree we don't want 100 different servers embedded in cloak that do slightly different things

[wet mason]

@civic yacht I'm going to show a quick demo of "services" to @wild zephyr in a few minutes (needed for graphiql plugin stuff) if you're interested (/cc @tepid nova)

[civic yacht]

I have a meeting from 11:30 to 12, but I am extremely interested. You can also just send me the commands to run and I can try it locally

or show tomorrow at the meeting

[tepid nova]

I was just going to ask about "services" and whether the builtin buildkit server could be a builtin "extension" which exposes a port via the service capability like any other service - using our port forwarding etc

[wet mason]

Yep, was planning to! The meeting is already packed with demos so not sure it'll fit

[tepid nova]

The idea broke my brain a little because port forwarding is a buildkit feature, so you'd need buildkit to port-forward to buildkit... But maybe a subset of port forwarding could be lifted out of buildkit itself, just enough to bootstrap bk?

Not sure this even makes sense

[obsidian rover]

https://tenor.com/view/inception-deeper-go-deeper-we-need-to-go-deeper-leonardo-di-caprio-gif-16756828

[civic yacht]

I think that part would make it hard (if I understand the idea correctly). It would be like trying to run buildkit in buildkit

[tepid nova]

Since you would be "forwarding" to an builtin in-process, not to an actual remote container

In my mind it would be more like running buildkit itself with the same interface than running a container on buildkit (but with different implementations)

anyway it's a small detail, you can ignore

[still garnet]

wait buildkit has port forwarding? 👀

[wet mason]

@still garnet no, but we can build that on top through the shim

[still garnet]

ah gotcha

[wet mason]

like, [container network] <-> shim <-> gRPC <-> cloak <-> host system

[tepid nova]

So, yes 😉

[still garnet]

i was just adding service networking to bass a few couple weeks ago and cut port forwarding from the scope for the sake of shipping (jeez networking takes a long time to figure out)

[tepid nova]

https://tenor.com/view/its-all-part-of-the-illusion-trent-radio-silence-teen-wolf-its-just-a-part-of-the-visions-gif-24206616

[still garnet]

so i was looking for a freebie 😛

[tepid nova]

Technically it's a hack on top of ssh agent forwarding, but buildkit devs are not only tolerant of that use, they aspire to support it more explicitly in the future

[still garnet]

huh, interesting

[wet mason]

https://tenor.com/view/look-around-drumsy-groucho-glasses-silly-umm-gif-17675181

"ssh"

[tepid nova]

https://imgflip.com/i/6te6u6

[wet mason]

oh bass has services? I'd love to get your opinion/feedback/thoughts on this

[obsidian rover]

I don't understand sorry 😇

[tepid nova]

@obsidian rover in theory buildkit only allows forwarding a socket for your client's ssh agent/ But in practice you can forward any socket and pretend it's "ssh"

[wet mason]

@obsidian rover we're hijacking buildkit's "ssh agent forward" to mount a unix socket inside the container, then use it for things like "proxying" the GraphQL API, and in the future, networking etc etc

[still garnet]

yeah, it was a process of aligning lots of stars 😅 https://github.com/vito/bass/releases/tag/v0.10.0 has some info. one key thing for example was using DNS for container <-> container networking so that IP addresses don't bust caches all over the place

but i really like how it ended up

[wet mason]

@still garnet awesome, will take a look and would really like to chat about that. Attempted the host networking stuff but realized that doesn't really work for things running in a VM (e.g. docker for mac), since host is meaningless

oh I see

bridge + resolver

[still garnet]

happy to chat about it! i tried to do hybrid bridge/host networking support too but that turned into a real pain to support, so I just stuck with bridge-only

[wet mason]

there's still local access to figure out (hence the port forwarding trick)

[tepid nova]

I think the only part we haven't figured out yet is container-to-container that works across multiple buildkit nodes, right?

client -> container is resolved IMO. There's only one correct way to do it

[still garnet]

yeah, haven't looked into cross-node yet, that went a bit above my pay grade 😛 (for bass) ($0)

[tepid nova]

Yeah it's going to be a while before we get to implementing that as well. But at least if we have a design that makes it possible later, that would be a great first step

[still garnet]

would that be multiple worker nodes behind one buildkit API, or multiple buildkit APIs? (wondering how cache sharing would work in that case too, ie deduplicating services)

[tepid nova]

since we're going to bundle buildkit into the dagger binary, it can be one or the other, whatever works best. Either way we won't expose the naked buildkit endpoint to the end user

that will definitely make things easier

I’m getting markdown linter errors in my docs PR, how can I run the same linter locally?

[wet mason]

dagger do lint markdown

[tepid nova]

in the cloak repo?

[wet mason]

oh, no, dagger

we have a linter in cloak?

oh we do

[tepid nova]

apparently 🙂

[wet mason]

or install the markdownlint vscode extension, and it'll show them right there

[civic yacht]

there's a markdown linter that runs in GHA

[civic yacht]

I found this in my shell history:

yarn global add markdownlint-cli
`yarn global bin`/markdownlint

mileage may vary

[tepid nova]

Thanks markdownlint was the magical word I was missing

[wet mason]

i just have this

it picks up the repo's .markdownlint.yaml

so same rules as CI

[civic yacht]

That's far too easy and sane 🙂

Also, while I missed the whole networking discussion, I'll just add that all our problems can be solved by just also embedding tailscale into our cloak binary and every shim

(mostly joking)

[still garnet]

can we add a split emoji of 😂 and 😐

[tepid nova]

I was wondering about wireguard and how it changes the clustering conversation

[civic yacht]

I think it will probably be an intriguing option on the table when we get there someday

[wet mason]

I feel like "network" for clustering is the "easy" part. Clustering itself is hard

[tepid nova]

great problem solved 😄

[wet mason]

even if the cache is magically distributed, there's the whole data locality thing. Like, in a naive clustering approach, it's probably slower to cluster than to build on a single machine

e.g. image(golang) { exec(go build) { stdout } } -- for that stdout to work, you have to transfer gigabytes of data back and forth. takes probably a few nanoseconds of CPU

[tepid nova]

Wouldn't a naive clustering approach be to load balance entire pipelines? Which would be probably faster than a single machine.

The difficult part is distributing each individual node on the dag, which I wouldn't call naive

The day we manage to do that though, will be a magical experience... It could be just ridiculously fast

[wet mason]

Yes. But in that case, we don't need networking right? since we're load balancing entire pipelines rather than nodes within a pipeline

(e.g. the pipeline can just talk to itself on the machine it ended up running)

[wet mason]

I guess that's what I meant. IF we need networking, THEN networking becomes the easy problem (because we're talking about "sharding" a pipeline across a fleet)

[tepid nova]

Yes that's right. We just need a networking model that allows cross-node networking to be retrofit later

[still garnet]

has there been any chat about exposing per-Run CNI params/args in Buildkit? or do they prefer the SSH agent approach over that?

[tepid nova]

Assuming I understand correctly, I think that approach would run into the "host is meaningless" problem @wet mason mentioned

[civic yacht]

I haven't seen that, but it would probably fit into the constraints model that exists decently well. It just probably can't become part of the cache key

[still garnet]

i was thinking of e.g. port mapping args: https://github.com/containernetworking/cni/blob/main/CONVENTIONS.md#dynamic-plugin-specific-fields-capabilities--runtime-configuration

[still garnet]

but yeah, it'd only be local to the buildkitd host without some extra work

[civic yacht]

Hm yeah I suppose it's debatable which aspect of networking configuration would be considered part of the cache key for the execop and which shouldn't. Like if you change a port mapping does that indicate you need to re-run the exec? Or should it still be cached from earlier executions? Probably depends on the use case

Also presumably part of why that issue has been avoided so far, kind of tricky

[still garnet]

yeah, networking in general is sort of at odds with buildkit's caching, like I usually wouldn't want a service to be cached, but I want it to be able to make use of caches (like cache mounts)

and I would want all the steps necessary to build the service to be cached, just not the final one that actually runs it

though normally these things aren't an issue since you just interrupt the service and it never actually completes

[civic yacht]

Yeah totally, actually the NewContainer interface does allow you to mount shared caches that are also accessible to exec ops and other new containers, so there actually is a route to getting mostly uncached execution plus a little caching when you need it

It's at the borders of what buildkit is meant for though

[still garnet]

yeah, and you don't get buildkit's automatic deduping, so if multiple things need the same service you'll run it multiple times

so using the regular build/solve flow ends up doing what I want, but it feels more like luck than intention

[civic yacht]

Yeah we actually went through a similar-ish thing with cloak. At first extensions were going to be frontends, but then we realized frontends aren't cached and so we switched to execops with the socket back to cloak mounted into. It worked out cleanly in that case, but I can see what you're saying about how it gets more fragile with generic services. You have to make sure you fail at all costs if using execops

[still garnet]

interesting. yeah - if something were to trap SIGINT and exit 0 and buildkit cached it you'd have to buildctl prune 😛 I have a feeling buildkit avoids caching on interrupt though, haven't checked

[civic yacht]

Yeah I think that also starts getting into the low-level executor part (i.e. runc vs containerd) which is a whole other can of worms...

[warm tundra]

so, I showed one of the simple Cloak examples to two other high-level folks on my team, and the immediate reaction from both was very strong skepticism about GQL. The need for a cross-language transport was obvious, but the reason for GQL over, e.g. protobufs wasn't immediately clear. I believe @civic yacht gave me some of the background on our last call, but I'm realizing it's not something I understand well enough to comfortably defend it.

it might be too early to think about messaging 😅 but I figured that was something you all would be interested to hear. I think it'll come up again in my POC.

[tepid nova]

lots of possible answers 🙂 Mostly depends on what bothers them about it

@civic yacht and @wet mason can tell you more about the thought process for switching from grpc to gql

[warm tundra]

I think it is largely a gut sense that this is Not What GQL Is For, which, to be fair, was also my first reaction. It's an unusual use case!

[tepid nova]

since we’ve switched, additional benefits have kept piling on

[warm tundra]

ooh, I don't think I knew that you started with grpc

[tepid nova]

yeah that’s fair. we can work on addressing that more explicitly

[strong coral]

It makes more sense when thinking about Cloak as a server.

[civic yacht]

Yeah, I can totally understand the skepticism, especially in the current (very early) state of things.

The original reason for choosing graphql was that we needed some interop language in order to support calling extensions across container+language boundaries and it just turned out that graphql was the cleanest option we evaluated. We tried protobuf+grpc early on too, but it quickly became a mess. We then tried graphql and things just kind of fell into place. I can expand more, but overall, I think that if we were going to design an interop language of our own from scratch, we probably wouldn't end up too far from graphql, so it makes sense to re-use it given the huge ecosystem around it already.

[strong coral]

There's still work to do on the UX. Is that a valid assumption?

[civic yacht]

Yes, as valid an assumption as an assumption can get 🙂

[warm tundra]

yeah, I think it'd probably be really useful to have some of that background and e.g. why grpc didn't work (I'm curious too!) as part of the docs when you roll all this out

I can definitely imagine that in a future version as the UX evolves there's less questions -- but I think you'll still get a fair few, because IMO engineers (good ones especially) skew toward the critical

[civic yacht]

I think early on I mostly assumed that whatever interop language we chose (grpc, graphql, etc.) would be an implementation detail hidden from end-users (except those that were trying to develop new cloak functionality). Since then we've realized that a lot of the aspects of graphql work so well for what we are doing that it doesn't make sense to necessarily 100% hide it. So we are still figuring out exactly where the boundaries are: how much are you forced to know and interact with graphql, how much of that can be hidden behind abstracted language-specific interfaces, etc.

I'm very much of the opinion that there should always be a path to using cloak that only involves knowing your language of choice (go, js/ts, python, etc.). You should not be forced to go read graphql docs to start using cloak, at least for the simple cases. Graphql actually works out nicely in that respect in that it's quite natural to have things like code-first servers (where schema is derived from code), generated clients, etc. It's not all that difficult to create abstractions on top of graphql that feel natural and intuitive.

For more advanced use cases, it may always be necessary to learn a little bit more how graphql works, but the other nice aspect is that graphql is really pretty intuitive and easy to pick up IME. Especially as contrasted with something very powerful+complex like CUE.

[tepid nova]

"engineers skew toward the critical"

Understatement of the year 😄

What's the "exit status 1" on cloak dev?

something about bootstrapping buildkit on docker not working?

[civic yacht]

The thing is that our current DX is nowhere near that goal, graphql leaks all over the place even in the simplest cases. We are doing a bunch of work on many fronts to change that (code-first schemas, much better generated clients, etc.). But if you look at it today that end-goal is not very apparent at all. Hopefully as we get closer the skepticism of "why graphql" gets replaced as the DX becomes more and more natural+intuitive feeling in each language.

[tepid nova]

that was it

[civic yacht]

Was that the entire output, exit status 1? Or was there more. I think someone else reported this, but hard to debug without being able to reproduce or with more information

[warm tundra]

code first schemas

ooh, didn't know you were working on that!

[tepid nova]

Nothing more. Yeah it was docker not running

[civic yacht]

https://github.com/dagger/cloak/issues/214

[civic yacht]

Well, work on it is imminent (probably with in the next week), I'm sure we'll have issues with design proposals in the very near future that we'd be happy to have your and everyone else's feedback on 🙂

[civic yacht]

Totally agree on documenting this, I did an initial braindump on one part of graphql vs grpc here if interested, but there's more to the story that we should include too: https://github.com/dagger/cloak/issues/215#issuecomment-1247268643 cc @wet mason

[ancient kettle]

I really look forward to having generated code for Go again. I'm having trouble bringing myself to write raw GraphQL queries + creating my own types for the responses. Just doesn't feel right in Go.

[civic yacht]

Totally, I don't feel good about the raw graphql interface in Go at all. It's just that the previous codegen was bad in its own ways and about to get much much worse with some of the upcoming API refactoring. So there wasn't a good choice at the moment, just a less bad one. The long-term hope is to figure out the query-builder interface (or equivalent solution).

On the bright side, even though it is really tedious to write the queries, the raw interface does give you more opportunities to take advantage of chaining, so a lot of times what used to be multiple operations (and multiple if err != nil, etc.) can now be combined into one. I know that doesn't help a ton, but it was one silver lining I found.

[ancient kettle]

Yeah, I totally get the why. Makes a ton of sense. I think in my case, because I'm focused on building out the core tasks in Dagger, I won't be able to take good advantage of the chaining. Although, I can see how that'd be helpful for more involved tasks.

One thing I'm definitely liking about this port: I don't have to worry about the internals of Buildkit so much. It simplifies some of the work I have to do by ~20-30%

[civic yacht]

Awesome, yeah that's the goal, honestly if you reach a case where it's not reducing the work that probably means we are missing something in our API 🙂

[still garnet]

trying out cloak with straight-up cloak do < foo.gql, came across @export which would be pretty nice for chaining results between queries (unless I'm missing something that already exists for it): https://graphql-api.com/guides/schema/executing-multiple-queries-concurrently/#heading-sharing-data-across-queries-via-export

the linked RFCs have been open since 2017 though 🙃

[civic yacht]

Ah yeah, we had some discussion on that previously here: #maintainers message

[still garnet]

oh cool

[civic yacht]

I think at the moment we are focusing more on getting the DX of writing extensions in higher-level languages better (where the immediate need for this is lower), but something similar to that is still on the table afaik

[still garnet]

yeah, it's not clear that we even need that level of expressiveness at the literal graphql layer

especially now that i can see that a FSID is pretty self-contained, I was worried about state there (looks like some kind of base64-encoded LLB definition?)

[tepid nova]

https://github.com/shurcooL/graphql FYI

[still garnet]

👍 I've used it via the githubv4 API (and liked it, fwiw), but wanted to dabble with graphql directly for now 🙂

also curious how succinct ruby/python/other dynamic languages could end up

[tepid nova]

To elaborate: I was just talking to our friend Gawen (will add him to this channel imminently) and showing cloak. He in return showed his current graphql setup, which is extremely neat. He uses shurcool/graphql and was telling me about his experience. The short version is that he likes it 🙂 Of particular note: it uses struct field tags for query variables, just like the hasura library we were looking at @wet mason @civic yacht , and had no complaints.

Welcome @queen mica 🙂 See above 👆🏻

[wet mason]

@wild zephyr sorry, my errands took longer than expected!

I'm free now if you are

[wild zephyr]

@tepid nova @wild zephyr #maintainers-audio

[wet mason]

@wild zephyr joining /cc @civic yacht

[twin crow]

Python support is ready for review - https://github.com/dagger/cloak/pull/177. I documented the next steps that I plan to add in a follow-up PR. The example extension works.

[civic yacht]

Follow up thought from what we were discussing: one of the cool things extensions let you do is import arbitrary third party libs and use them in your code, so we will need to figure out whether/how that could work from the extension IDE in the browser. This might be easier to accomplish via a "dumb terminal" from browser ->IDE running in a cloak container as compared with it putting the whole IDE in the webapp directly (but I would be delighted to be wrong and find out there's some cool+easy way to do that from the browser) cc @wild zephyr

[wet mason]

@civic yacht I just realized that we might have one more "layer" than services -- in bk there's container and then process

I think that's like a docker container and an exec right?

For instance, by adding a layer of indirection, it means that you could create a mysql service, run "mysql add user" then mysqld

[civic yacht]

Yeah that's right. Buildkit's NewContainer interface supports that too (exec'ing in at arbitrary times)

[tepid nova]

isn’t that just going back and forth between a service and its underlying container, as needed? Seems like we get this “for free”?

I’m assuming ’mysql add user’ is standalone ie no mysqld needs to be running?

[wet mason]

it's like docker run -it alpine sh twice vs docker create alpine + docker exec -it <alpine> sh

in the latter, it's the same namespace/fs/etc etc, two processes on the same container

[tepid nova]

ok got it. I was too focused on the mysql example

(which may probably be easier to do without exec in the same ctr)

[wet mason]

I realized this doing when talking about the redis extension, with a hypotentical start, stop and flushCache

flushCache would probably need to run a redis command inside the SAME container

[tepid nova]

but tons of use cases would benefit

[wet mason]

and the bk API does provide that (but I hid it to simplify)

but maybe I should expose it as it is

[tepid nova]

yeah I understand now. Super powerful

So Container, Service and probably Command ?

Or are Command and Service just the same thing?

[wet mason]

Probably yeah. bk calls it a process (ContainerProcess, but they call services containers)

[civic yacht]

Maybe withCommand is a field on Service that lets you exec a process in?

[wet mason]

yeah that would make sense

[wet mason]

each command has its own stdout/stderr/stdin

service has the fs + mounts + network settings etc

[tepid nova]

That’s just a container though

(my first non-bk-expert reaction 👆)

probably safer to model docker concepts when possible, for familiarity

Could we somehow collapse “container as in llb state” and “container as in a live namespace with one or more processes running right now” into the same Container for simplicity?

So a container is something you can 1) pull from a registry 2) exec commands in 3) start / stop long-running processes in ?

This is kind of “docker compose model” vs “docker engine model”

[civic yacht]

It's appealing to try to find a way, but on a practical level the two entities in buildkit are pretty different. LLB ExecOps are cached, containers are not. There are also features in one missing in the other (e.g. the stdio control present in containers isn't in execop, containers are missing just secret envs for some reason, most likely due to an oversight). So it would be hard to hide that all from users

However it probably is possible to use a dagger container (aka buildkit exec op) as the base for a dagger service (aka buildkit container). Like there could be a field that takes all the mounts and other configuration of the dagger container and then returns a service with those settings applied? Maybe?

[tepid nova]

Yeah having a separate exec and start each returning a different type, makes total sense for that reason, completely different underlying implementation.

But it also makes sense that both fields start in Container because if I setup mounts, env etc in my container, I expect them to be used in both. It would be really surprising otherwise

from there it boils down to: what type does Container { start } return: Container or Service ? The former leads to docker engine model; the latter to docker compose model

[civic yacht]

Ah okay, yeah I think it should be Service. There are just fields that would be present in Service that would be invalid in Container (like stop)

[tepid nova]

How would I run redis flushcache in the same namespace as the running redis service?

query startRedis {
 container {
  from(adress: “redis”) {
   start(args: [“redis”]) {
    exec(args: [“redis”, “flushcache”]) {
    id
   }
  }
 }
}

ie a service can do additional execs?

[civic yacht]

Yeah that's what I was imagining

[tepid nova]

compose model it is then 🙂

[wild zephyr]

Follow up thought from what we were

[tepid nova]

I see amazing demos in our future 😁

[civic yacht]

I don't know if there will be time to demo tomorrow given how much awesome other stuff we have lined up, but multiplatform is progressing smoothly so far (though it's still early). Also have multiplat image push working now.

The withPlatform(platform: String!): Query! is a pretty interesting pattern too for creating "ambient" configuration that applies by default to all queries, probably has use outside multiplatform if it continues to pan out.

[tawny flicker]

hasura is a fork of the shurcooL module 🙂

[tawny flicker]

diff shurcooL - hasura: https://github.com/shurcooL/graphql/compare/master...hasura:go-graphql-client:master
diff hasura - shurcooL: https://github.com/hasura/go-graphql-client/compare/master...shurcooL:graphql:master

[obsidian rover]

@civic yacht. The bundling is ugly, but works perfectly 😇 We can start the next steps. I'm not sure to have understood the final consensus

[mellow bolt]

I've just created a Vercel extension : https://github.com/slumbering/dagger-vercel

@tidal spire I get inspired by your README just to keep consistency.

[tidal spire]

Looks great!

[tidal spire]

Using cloak today, if I want to Filesystem.pushImage(), is there any way to set attributes on that image like expose or entrypoint? Or do you have to use a Dockerfile and Filesystem.dockerbuild()? Or does that even keep those attributes?

[tepid nova]

The new core api (not yet implemented) addresses this in a new Container type: https://github.com/dagger/cloak/blob/main/api/container.gql#L71

[tidal spire]

Perfect, that's exactly what I was looking for. Thanks!

[tepid nova]

https://tenor.com/view/this-explains-everything-liz-lemon-30rock-that-makes-sense-makes-total-sense-gif-21894293

[hasty basin]

https://github.com/dagger/cloak/issues/43

[civic yacht]

Erik Sipsma3294 The bundling is ugly but

[cosmic cove]

<@&1003717314862129174> Reminder that we have the community call in 3 minutes. We have a lot of great demos, so looking forward to seeing all of you!

[tawny flicker]

https://discord.gg/vH62H85C?event=1020016841160147044

[tepid nova]

Joel about to give a first demo of Dagger Classic: Dagger 0.2 ported to cloak 😍

[tepid nova]

@tawny flicker showed a Hugo extension and Google Cloud extension, and how to use it to deploy a static website easily

[tepid nova]

Now @mellow bolt is showing his Vercel extension and using it for multi-cloud deploment: Netlify vs Vercel !

[tepid nova]

@wet mason is now melting our brain with long-running processes + attach. Run an interactive shell in dagger, then attach to it

[tepid nova]

The brain melting continues with debug breakpoints: add debug anywhere in your query, then open an interactive shell to inspect the directory at that point in the pipeline

[ancient kettle]

It's cool to see this feature (that we've talked about for a long time) actually working!

[cosmic cove]

@tawny flicker just a heads up that I've avoided using this event feature since it shares the event with everyone on Dagger server vs just everyone in Cloak when you don't pick a specific audio location 😦 Once we move to preview launch then we will use this Discord feature.

[tepid nova]

It's less of a problem now that we're telling our community: "if you want access to cloak we'll give you access right away"

[tepid nova]

@wild zephyr finished us off with a demo of a GraphiQL plugin: not only can you prototype your graphql queries in the browser, you can prototype the client code sending them as well! Especially useful when you need to send multiple queries stitched together

[wet mason]

I don't remember who was asking for "how do you block until a service has completed" -- I just implemented that (select exitCode, it will block until the service exits and give back the exit code)

{
  core {
    service(id: "aa5f08e6d2e6472cac7f1049f09cef74") {
      id
      args
      exitCode
    }
  }
}

{
  "data": {
    "core": {
      "service": {
        "args": [
          "sh"
        ],
        "exitCode": 42,
        "id": "aa5f08e6d2e6472cac7f1049f09cef74"
      }
    }
  }
}

@civic yacht re the cache bust issue in the demo: my theory is it's because we serialize the "parent" into the extension call. Changing a parent field selection busts the cache on sub-resolvers, I think?

unrelated with services, basically:

parent {
  ext
}

parent {
  something
  ext
}

the latter is cache-busting the former

I think

[civic yacht]

We serialize the parent object, but the parent object in theory shouldn't be different just because the fields being selected on it is different. Each field being selected in the object should be resolved in parallel and each resolver should get the same object.

That's not to say the parent object isn't causing the cache bust, but I suspect it isn't just that more sibling fields are being selected

But I'm not 100% sure, it's all really subtle. If you do a println debug of the parent object that would help

[wet mason]

yeah that's literally what i'm doing right now

@civic yacht right, it's completely unrelated, wrong theory

@civic yacht This is the symptom:

{
  core {
    git(remote: "github.com/dagger/dagger") {
      dockerbuild {
        exec(input: {args: ["dagger", "version"]}) {
          stdout
        }
      }
    }
  }
}

cache bust:

{
  core {
    git(remote: "github.com/dagger/dagger") {
      dockerbuild {
        debug {
          session
        }
        exec(input: {args: ["dagger", "version"]}) {
          stdout
        }
      }
    }
  }
}

new theory: it's the mount in debug (mounting the parent FS into a service). Somehow re-Solving the same FS re-computes it

[civic yacht]

What in the cache is busted? the exec?

[wet mason]

i run query #1 first, then query #2, and in query 2 I see dagger being compiled again

[civic yacht]

oh so dockerbuild is busted?

[wet mason]

might be because of dockerbuild (since we're calling a frontend)

yeah

but only once

after that it's from cache

[wet mason]

and debug is mounting parent (dockerbuild) into /mnt

[civic yacht]

Hm yeah was just looking at your PR code, I can't see why the cache would be different for the mount in debug. I'm gonna try to run it too

[hasty basin]

huh, would need to make repro, but I remember in Dagger with CUE that often I'd need two initial runs with certain plans to get everything from cache on the third run. First run was super slow, next run not "fully" cached, and third run was fully cached and instant. Wonder if had docker.#Dockerfile in those 🤔

[civic yacht]

When I do this the cache is not busted, the second query is always fully cached

[civic yacht]

Tried with freshly cleared out cache too.

If you run docker logs dagger-buildkitd and look at the end do you see stuff about pruning?

Also, if you start from an empty cache do you get the same behavior?

[hasty basin]

I can repro this 3rd time behavior like this after killing and removing my buildkit container and cache volumes.

cd cloak
cloak do -f examples/queries/docker_build.graphql

[civic yacht]

What's not cached on the 3rd time? Also, does docker logs dagger-buildkitd show anything about pruning at the end? I just tried this too and am getting the same caching behavior each time, nothing different on the 3rd run

Also, output of docker exec dagger-buildkitd buildctl du might be interesting too

Basically, I'm wondering if it's possible that buildkit's pruning logic could be kicking in and removing stuff from your cache. It's the only plausible explanation I can think of immediately (besides good ol' bugs and such)

Oh and docker exec dagger-buildkitd buildctl debug workers -v (which prints the gc policy as calculated based on your disk size)

[tawny flicker]

You are right!, I forgot about that! Sorry

[cosmic cove]

No worries at all!

[wet mason]

Will have to try again. Happened twice: yesrterday's demo and today's, so never had a chance to live debug

@civic yacht opened a services PR this morning: https://github.com/dagger/cloak/pull/218

Quite a few TODOs

[tawny flicker]

I just saw this: https://twitter.com/kelseyhightower/status/1570478763712086018

[hasty basin]

Has always been transient for me and of course it won't do it when I'm looking hard 😆

The last time it seemed to miss and hang, I noticed the longest pause right here:
#34 3.776 go: downloading github.com/containerd/ttrpc v1.1.0
Though that's right before the compilation output

^[#34 26.83 /out/dagger: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=jKdk-7MKPIkWLg1qSnUh/USb_TfJq7zI5n-QzkDvq/Z28Odr9V-q2t1yyXFvEk/9-QIc8WDNRXA4ILWOHh4, stripped

So maybe something happening in there. Seemed like it ran full compile twice maybe?? Maybe something to do with my laptop: m1 arm/macOS/dockerdesktop for mac

[civic yacht]

Hm yeah if you don't have a ton of free disk space in the vm created by docker desktop then I wouldn't be surprised if it's pretty easy for the pruning to kick in pretty early. If it happens again docker logs dagger-buildkitd should show stuff about pruning which can help prove or disprove this theory

[tepid nova]

I thought during the demos: even without the codegen feature in the Go SDK, we can still ship native Go bindings for the core API: even if they’re generated, it will be by us, as part of developing and releasing the core

won’t apply to extensions of course but that’s ok

it’s simpler to ship a generated lib to our developers than to ship code generation tooling

Maybe that was already obvious and I’m just catching up 😁

[civic yacht]

Yeah I considered that while I was removing the codegen but didn't pursue because:

1. It's not clear to me what we would actually ship once we switch to the new API. The new API is so heavily reliant on chaining, with each chain probably heavily dependent on your use case that it's harder to know what the most common operations would be. I'm sure it's possible, but not immediately obvious what we'll put there.
2. One silver lining of forcing use of raw graphql queries is that it enables you already to take advantage of chaining more (what used to be 3 operations can now just be one sometimes). That's nice in and of itself and it also is nice in that it sets up the future transition to a query builder (you can just take your graphql query string and translate it into method calls).

So I'm not at all opposed to adding a hand-written lib if everyone wants it, but I don't know if we should do it right away. Any effort we spend on that is effort that is not going into everything else (including the query builder itself, once that becomes a priority again)

But if a mob with pitchforks comes up to us asking for it, then we should do it

[cosmic cove]

<@&1003717314862129174> Thank you for all the demos today! It was a very exciting community meeting today. If you missed it, you can watch the recording here (you can find the link in community notes too). I marked the start of the demos as a highlight in the recording, so you can jump right to them. https://dagger-io.zoom.us/rec/share/QYWkbMtVpPdQIusPwoOsltljQbm2-d7oC8h3p_NemCG-PPh6HHVSLERiJKBcnypV.Tv4a-It4HP1p5Gc1?startTime=1663261173000
Passcode: eo3w8?rq

[tepid nova]

Makes sense to me. If someone feels a burning desire for it they can open a PR 🙂

[wet mason]

@civic yacht @tepid nova @still garnet Quick braindump on host networking and services

At first, I was thinking cloak would "proxy" endpoints locally (e.g. http://localhost:1234, assuming cloak runs on localhost)

I think a better approach would be expose networking through the API itself, like so:

[API client] --[ws]--> [cloak API] --> [shim] --> [process]

Basically, cloak acts as some kind of io.Pipe between the API client and the end process, instead of proxying directly with its "local" network

Basically means "host networking" itself is an API construct (works remotely, works in browser, etc)

</braindump>

[tepid nova]

how do I connect to my service from another program on my host?

seems like we could get both, one layered on top of the other

[civic yacht]

I think it could make sense to tunnel everything through websockets (even raw tcp and udp), but yeah I have the same question about how existing applications would work with that if we don't use localhost

[still garnet]

i wonder if we could expose a SSH tunnel interface too? i.e. ssh -L 1234:service:5678 <dagger-ip>

but this is just what springs to mind while not 100% grokking your proposal yet 🙂

[wet mason]

So either the client code itself tunnels (through the SDK), or for low effort stuff, you'd use cloak proxy <some arguments> (built on top of the SDK)

Mirror approach to cloak attach -- you can attach through the SDK but if you're lazy cloak attach exposes that through the CLI

(stolen from kubectl proxy)

[wet mason]

Yeah 🙂 Could be built-in or layered on top ... could also be an extension 😛

[civic yacht]

So basically:

1. Everything is tunneled through websockets
2. If you want to interact with localhost (whether listening or connecting) then you use a proxy between localhost and websockets?

[wet mason]

@still garnet went with websocket for tty because that's over the same endpoint as the API itself, seems like a sane default approach

(and graphql uses websocket for subscriptions, that's why I picked that over http/2 -- doing websockets over that port is already expected)

[tepid nova]

I like the original idea better, engine proxies ports, controled by the api

• give me a direct websocket to the service from the client if I need it

host port forwarding should be decoupled from starting services

[wet mason]

the blurry part is the meaning of "host"

that'll be the host where the engine is running, not localhost

[civic yacht]

I can't tell if I'm actually understanding what everyone is proposing.

As a concrete example, say you want to run a database in a service that knows how to listen on tcp port 1234 on its localhost. Now you want to connect to that db from your localhost (on a different machine).

1. You (as in the client running on your local machine) start the service via the API, which execs the db in a buildkit container. The db starts listening on its localhost on port 1234
2. You request, again through the api, port 1234 of that service. This results in the shim in the db's container to start proxying between its localhost port 1234 and a websocket connected back to the engine (through /dagger.sock)
3. You, on your client machine, get in response an endpoint for that websocket, through which all traffic is being proxied
4. Since you also want that traffic to be proxied on your own localhost, you also start up a proxy between the websocket and your localhost.

Is that what you are imagining Andrea?

[wet mason]

yep

[civic yacht]

Okay cool, sorry to be tedious, I just had to double check. That makes sense to me and is nice in that what the shim does in the container is an exact mirror of what you as a client do on the host (probably the same code). What are the other options? If we don't tunnel everything through websockets like that, then how does the stream get back and forth between client and service?

[wet mason]

The other option would be: no changes on the shim side. however, cloak proxies traffic to its own localhost directly rather than exposing it through websockets

1: same
2: same
3-4: not needed. instead, connect to "<cloak host>:<endpoint port>"

[wild zephyr]

this would be similar to kubectl proxy but we'd be using ws instead of https like k8s.

[wet mason]

🙂

[wild zephyr]

whoops. Missed that

[wet mason]

ETOOMANYMESSAGES 🙂 but yeah the command's inspired by kubectl

[still garnet]

tangent: curious how y'all feel about the shim. i initially felt weird about it with bass and wanted it to go away in favor of changes to upstream buildkit, but eventually learned to stop worrying and love the shim. it's saved my ass so many times. feels relevant here with baking more behavior into it.

[wet mason]

I like having the shim. Chances are we'll end up in situations where it doesn't even make sense for upstream buildkit to implement some of the features. I think it COULD be a problem in some specific scenarios, but at that point we could always have a shim-less option (which disables shim-enabled features)

[civic yacht]

Okay right, yeah I guess the downside to that is that it assumes the client can always connect to the cloak host easily like that. I suppose we could support something like protocol://<cloak host>:<endpoint port> to be a bit more generic (i.e. protocol could be unix-sock:// ), but idk, it does feel like just putting everything through websockets is like a universal transport and leaves the party at each end of it to figure out how to proxy that traffic to the local environment. Thinking out loud here though, low confidence

[wild zephyr]

3 and 4 could be combined and make the proxy automagically when using the cloak CLI to avoid an extra call

[wet mason]

Yeah, that's my thinking as well

[still garnet]

yeah makes sense. it's invaluable as an escape hatch to not become blocked on upstream changes that may be controversial/out of scope. once you get past the packaging it's really not too bad.

[wet mason]

Also works in a bunch of different scenarios, e.g. remote cloak (cloud, playwithcloak, in a VM, whatever), works in browser, also avoids port clashing (the end client is eventually responsible for mapping ports)

@still garnet what other use cases you have found for the shim? right now we use it to report back stdout/err, exit codes. Eventually network proxying (this conversation), injecting stdin

we could in theory also support tty attaching for NON services, idk if that's a good idea (/cc @civic yacht?). Same infrastructure for networking, just stream stdout/err instead

[still garnet]

unpacking OCI tarballs into a flat rootfs mount + image config, so you can run an arbitrary command (e.g. nix build) to build an image for a later llb.Run

also service healthchecks

and yeah, injecting stdin too

[civic yacht]

The reason buildkit avoids that is it introduces non-determinism to ExecOp which has results that are supposed to be cached (afaik anyways, that seems to be the underlying philosophy and why NewContainer isn't cached). I think I agree with that logic. Obviously ExecOp isn't 100% hermetic, but if we added a way to just type whatever you want and then cache the result, a lot of difficult questions get raised. Seems safest to just avoid it and rely on uncached NewContainer for interactivity

[wet mason]

very true. it would work for stdout/stderr, but I question the usefulness

maybe if you just can't wait for the execop to finish and want to stream the stdout of the command back, but that's really an edge case

[tepid nova]

This is missing a feature. I want to be able to tell the engine to proxy for me in the context of my host. It’s additive to the rest. But really important because most of the time I don’t want to implement my own proxy. Same reason I don’t want to implement my own workdir loader: too much work, easier to call host { workdir }

Supporting remote engines is not required since host { workdir } already doesn’t. We baked in the requirement that the engine is local, might as well reap the benefits.

(again it’s everything Andrea described + this extra feature on top)

[civic yacht]

Right right, I was just confirming the low-level steps, agree it makes sense to automagically do the proxying (like @wild zephyr phrased it), users don't have to run each low-level command themselves. The shim and the engine just both are proxying between localhost and websockets, mirror images. And like you said before, doing this doesn't mean clients can't also get direct access to the websocket as needed (i.e. in a hypothetical future where dagger cloud connects to a remote engine).

Honestly, the fact that in this scenario the shim and the engine end up becoming mirrors of each other is making me wonder if the shim and the engine actually should be different. Maybe they are the same binary and just connected with the router. I'm guessing that's probably too clever for its own good, but fun food for thought. I suppose if nothing else the shim could in theory be mushed into the same binary under a separate subcommand from the engine, like we are doing with all the rest of our functionality.

[tepid nova]

on the last point, a thought:

https://tenor.com/view/denaziamigo-doggo-dog-puppy-weird-gif-17908738

(I think rolling it up makes sense)

Welcome @eager yew 🙂

@civic yacht @wet mason is there a go package for generating gql schema ? Just the raw generation capability

(ie not tied to any particular frontend)

[tepid nova]

It looks like Github doesn’t have a reference doc for their graphql API: the graphiql explorer plays that role? Or am I missing something?

https://docs.github.com/en/graphql/guides/using-the-explorer

[still garnet]

they've got a category for it here: https://docs.github.com/en/graphql - with a reference in the sidebar, e.g. https://docs.github.com/en/graphql/reference/queries

[tepid nova]

Thanks! I had indeed missed it

[civic yacht]

The parser I've seen positive comments about is this one: https://github.com/vektah/gqlparser

It seems like it may also have the ability to write out schemas too: https://github.com/vektah/gqlparser/blob/b3be96ff69fa97682c43570dcb6f75d08fdf8586/formatter/formatter.go#L137

But not sure how well the use case is supported there, worth looking around more too

[tepid nova]

@civic yacht @wet mason could I bother one of you for 5mn with a very specific question about the internals of our graphql engine?

I'm trying to implement a small slice of the core API, to make sure I'm familiar with the implementation constraints. Picking up where I left off with https://github.com/dagger/cloak/pull/80 basically

(I'm unstuck, thanks Erik)

[civic yacht]

Also forgot to mention that as part of the multiplatform pr I found a way of using generics that allows us to use specific types when writing resolver methods (including an explicit parent type) and then trivially adapt it to the untyped method you have to provide to the graphql-go library. Should result in all that being much more clear if it continues to pan out

So there's hope

[wild zephyr]

don't forget about shopify's graphql docs, which are the dopest I've seen so far https://shopify.dev/api/admin-graphql

[tepid nova]

cc @hybrid widget 🙂

@wild zephyr would there be a way to have a cloud version of our api playground (so far so good) which can load extensions and, somehow, update that user's workspace's API endpoint accordingly so that the extension appears in the editor, explorer etc. Not sure how this would work in practice since at that point you would not yet have a local engine running

[wild zephyr]

do you mean adding the extension via cloud and then it automatically appears in my local editor? Why not adding the extension in the local editor directly? 😕

[tepid nova]

There is no local editor in this case. I'm in cloud. Maybe I don't have dagger installed locally at all.

But I want to start exploring the API in my browser, like I can do with eg. Github's API explorer

@civic yacht I need SSHAuthSockID in my new gitSchema but there's initialization logic that seems specific to coreSchema. How should I go about that?

Maybe we could move SSHAuthSockID to baseSchema ?

[civic yacht]

Yeah I was just typing out a longer version of essentially suggesting that 🙂

[tepid nova]

lol ok, let me give that a try

Ah I just noticed that execSchema could benefit from this too

any reason I shouldn't remove sshAuthSockID from execSchema while I'm at it?

[civic yacht]

No not at all, I think this just evolved "organically" and is ripe for simplification by putting it in baseSchema

[tepid nova]

My patch is causing this error:

% cloak dev
Error: failed to solve: input:2: Variable "$id" cannot be non-input type "String!".
input:2: Unknown type "String".
input:2: Variable "$id" of type "" used in position expecting type "String!".

Is there any way to get more context, some debugging info?

[civic yacht]

That's an error you can get sometimes when there's a problem with your schema. The graphql library for whatever reason ends up not erroring out but instead just returning garbage where even primitives like String aren't defined 😔 It's been a while since it happened, but I'll go make an issue for it because it's incredibly obnoxious. Basically, my best advice is to look at the changes you made to the schema strings and see if you could be referencing a type that you haven't defined yet

cc @wet mason it's that dumb issue again ^

[tepid nova]

It's definitely a change I made that broke the schema. I was hoping to narrow it down 😉 Here's the diff: https://github.com/dagger/cloak/pull/226

I will go over it again

[civic yacht]

https://github.com/dagger/cloak/pull/226/files#r973272554

[wild zephyr]

I see... well.. we don't really need to have a fully blown engine running to expose the API schema with it's extensions. I guess there's a way we can expose an introspection endpoint for each dagger cloud user and dynamically modify the schema with the extensions that the users add / removes. That way you should be able to explore the graph but running queries will just end up in /dev/null if you don't have any engines attached. cc @wet mason @civic yacht the dynamic schema update without reloading the app is possible, correct?

[hybrid widget]

thanks!

[tepid nova]

Hi everyone, heads up we will soon merge the dagger/cloak repo into dagger/dagger. There will be some mild disruption but we will communicate here in advance, and give specific instructions for dealing with the change

There will be a few weeks where cloak is an "open secret", the branch, discord and issues will be accessible but we will not communicate much about it because of the lack of polish and high rate of breaking changes. Next month we plan on releasing a preview with more polished docs and UX, and communicate a bit more

[wet mason]

@civic yacht @tepid nova First test: https://github.com/dagger/dagger/tree/cloak

Looks reasonable to me (dagger history preserved, cloak history preserved, we should be able to rebase cloak PRs on top of that)

If we're happy, I'll re-do it once all PRs that we want to merge have been merged

[wet mason]

@cosmic cove @hasty basin

PR migration steps:

From cloak repository:

1. Fetch dagger/dagger

git remote add dagger [e-mail]:dagger/dagger.git
git fetch --all

2. [optional but recommended] Duplicate your branch

git checkout your-pr-branch
git checkout -b your-pr-branch-copy

3. Rebase your branch on top of dagger/dagger's cloak branch

NOTE: there might be some commits not authored by you. Delete them.

git rebase --reapply-cherry-picks -i dagger/cloak

4. Re-create PR in dagger/dagger

Make sure you have the GitHub CLI installed (on macOS: brew install gh)

gh pr create -R dagger/dagger --base cloak

(^^^ @civic yacht looks good?)

[hybrid widget]

#18 pushing manifest for docker.io/[MASKED]/testrepo:myapp@sha256:edde4c2f5d0723c0d90e9c61f7e2c11bf0de548d8c3b8a63a12fbecb16642e67

is there a way to get this sha256 hash from pushImage()?

[civic yacht]

Just finished rebasing and merging the only PR from someone outside the team: https://github.com/dagger/cloak/pull/228

So let me run through these instructions to double check, but looks good. And any problems at least will be internal to us only 🙂

[wet mason]

@civic yacht I'm going to go ahead and remove write access to everyone on dagger/cloak, to avoid any divergence from dagger/dagger#maintainers -- sounds good?

[wet mason]

I don't think we've ported that yet, it should definitely be in the new API (we had it in classic dagger)

[ancient kettle]

@wet mason I was going to create a dagger-classic (really not liking that name any more...) issue as a milestone for the port to cloak. Should I put that on dagger/dagger?

[civic yacht]

on git rebase --reapply-cherry-picks -i dagger/cloak I get rebase a conflict.

Also, I have git signing setup and hooked up to secretive on my mac, so it's asking my to scan my finger print 362 times in a row for every commit. That's more of a me problem though...

[wet mason]

@civic yacht ugh.

@civic yacht ok might be because of how i created the branch

@civic yacht I basically created an empty branch, started from dagger main, removed everything. Then checked out dagger/cloak's main into cloak. Then rebased on top of empty. Then pushed as dagger/dagger#maintainers

I don't know a better way to create a non-orphaned branch from a common parent

I got a rebase conflict as well, but that was from a couple of unrelated commits that I removed before git rebase -i

[wet mason]

since I've rebased on top of a different parent, all commit IDs are different

@civic yacht Oh. Try without --reapply-cherry-picks

[civic yacht]

the conflict I get is between files not changed in my pr (it's like README.md and .gitignore), it happens on initial commit

[civic yacht]

this is at the edge of my git knowledge, but is this what merge commits enable? like there one commit that collapses a separate history on top of a chain? there's a 90% chance this makes no sense, please ignore if so

[wet mason]

@civic yacht let me try something with merge commits

[civic yacht]

without it I get a conflict on pick b4c1bf6d build(deps): bump go.opentelemetry.io/otel from 1.8.0 to 1.9.0 . Not sure why there's a conflict, but that's much easier to deal with so not a big deal

[wet mason]

yeah I get the same

[civic yacht]

I think you just want the first commit in the conflict, I can never remember if that's -Xtheirs or -Xours, but maybe one of those

[civic yacht]

opening the PR failed but I guess that's because you are working on the new version of the cloak branch, once that's done let me know and I'll try the PR again. I'll see in the meantime if I can resolve that conflict automatically

[wet mason]

@civic yacht just tried with a merge. let me know if it's better

@civic yacht git refused so I had to use merge --allow-unrelated-histories

I don't know if this is better or not

@civic yacht I think it's worse this way, i don't think github is picking up contributions of the merge

have a call, will resume in a bit

[cosmic cove]

@civic yacht all issues are merged, let @hasty basin and I know if we can help at all with PR stuff. Working on communication doc right now.

[hasty basin]

Yep we've moved all dagger/cloak issues to dagger/dagger as part of slow migration 🙂 Issues are disabled on dagger/cloak repo. Please open any cloak issues on the dagger/dagger repo with a cloak label 🙏

[civic yacht]

It's better in that I didn't have to scan my finger print 400 times 🙂 but again, I can just adjust my setup, don't think that applies to anyone else. Where do you see the contribution not picked up? At least in the history here everything seems correctly attributed https://github.com/dagger/dagger/commits/cloak?after=80c0b61a535c38ab8648f0227d2817e4f7206b38+34&branch=cloak&qualified_name=refs%2Fheads%2Fcloak

[civic yacht]

Merged PR from only non-team member, just working on making sure other PRs can be rebased smoothly

[civic yacht]

remaining issue is that I can't seem to create a PR from sipsma/cloak->dagger/dagger#cloak, so I think I need to also setup sipsma/dagger#cloak first? Trying that

[wet mason]

@civic yacht https://github.com/dagger/dagger/graphs/contributors

you were #1 with the rebase approach

[civic yacht]

Oh, hm okay... as much as I like to be #1 I am okay with that not being preserved if this option is better in every other way 🙂

[civic yacht]

Yeah if I do

git remote set-url origin [e-mail]:sipsma/dagger.git
git push --set-upstream origin your-pr-branch-copy

then opening the PR works

[civic yacht]

According to this contributors should only be calculated from the default branch: https://docs.github.com/en/repositories/viewing-activity-and-data-for-your-repository/viewing-a-projects-contributors#troubleshooting-contributors

So I would guess that when we make that switch we'd get updated calculations there that would include both the previous dagger history and cloak?

[wild zephyr]

what's the ETH merge compared to Cloak's ? 😛

[ancient kettle]

😂

[civic yacht]

Collected adjustments

From cloak repository:

1. Fetch dagger/dagger

git remote add dagger [e-mail]:dagger/dagger.git
git fetch --all

2. [optional but recommended] Duplicate your branch

git checkout your-pr-branch
git checkout -b your-pr-branch-copy

3. Rebase your branch on top of dagger/dagger's cloak branch

NOTE: there might be some commits not authored by you. Delete them.

git rebase -i dagger/cloak

4. Update your local cloak repo to be downstream of your dagger fork

git remote set-url origin [e-mail]:<your_username>/dagger.git
git push --set-upstream origin your-pr-branch-copy

5. Re-create PR in dagger/dagger

Make sure you have the GitHub CLI installed (on macOS: brew install gh)

gh pr create -R dagger/dagger --base cloak

6. Close PR in dagger/cloak

@tepid nova or @still garnet if one of you are online do you mind going through the above instructions with your open PRs in cloak? You two have open PRs that don't need rebasing and are most likely to be online 🙂

Just want to verify that it works for you too.

I'll then wait for @wet mason to get back from the call, confirm we are okay with the merge commit strategy and then we can have everyone else migrate their PRs

[tepid nova]

Good timing I have a PR ready for review right now

The goal is to move my PR to dagger/dagger correct?

[civic yacht]

yes exactly, I tested mine here: https://github.com/dagger/dagger/pull/3119

Want to get confirmation it works for someone else too

It was bumpy my first try but think we ironed it out now

[tepid nova]

looks like it worked

we could probably make it a shell script, might be easier to help people tweak it if it fails too

but not enough PRs to justfy the effort probably

[civic yacht]

For PRs yeah I'm not sure, it's only team members left. However I was just realizing that any other existing external users will need to adjust their remotes. That should only be a few lines, so easier to make a script and include in whatever limited message we send out about the soft launch

Giving that a shot

Also before I forget, should do this sooner than later: https://github.com/dagger/dagger/issues/3122

[wet mason]

I swear it was the case when it was a rebase ... weird 🙂

[civic yacht]

I think if I was #1 in the rebase in dagger that would be bad because it means the original dagger history was gone, so idk

[wet mason]

@civic yacht oh it's only for the past month

[civic yacht]

So you are good with the merge strategy then?

[wet mason]

cloak branch looks good to me. Good to you as well?

[civic yacht]

Yep seems to work

[civic yacht]

I'm writing out commands that external users of cloak can use to rebase their local repo. I think it can include making a fork of dagger if they don't have one already and such

[tepid nova]

btw that PR is ready for review 😇

[wet mason]

git push --set-upstream origin your-pr-branch-copy

we should have people push to their fork

ohh nevermind

i keep origin as upstream personally

[civic yacht]

Yeah I have them change the origin to the dagger fork. I'll add a note about that in the instructions

Didn't mean to enforce my opinion 🙂

[wet mason]

Instructions worked for me: https://github.com/dagger/dagger/pull/3123

[ancient kettle]

One question I’ve got… how will/should I import this into my branch of dagger for dagger-classic?

I’m guessing there is going to be a module name clash.

[wet mason]

how is classic setup right now?

[ancient kettle]

A branch in jlongtine/dagger (which is a fork of upstream dagger)

Because I’m reusing largish chunks of that codebase.

[wet mason]

How were you thinking we'd distribute classic? Separate binary, same binary, ...?

[ancient kettle]

I’m open to suggestions. Keeping it in the same binary might ease transition a bit. But not something I feel super strong about.

Probably worth keeping it in the same codebase, though.

[civic yacht]

@wet mason here's the script I ended up with that external users can run to adjust their locally cloned repo:

#!/bin/sh
set -ex

DAGGER_REMOTE_NAME="upstream"
USER_REMOTE_NAME="origin"

GH_USER="$(gh api user | jq -r '.login')"

gh repo fork dagger/dagger # answer no to "clone the fork?"

git remote set-url ${DAGGER_REMOTE_NAME} [e-mail]:dagger/dagger.git
git remote set-url ${USER_REMOTE_NAME} [e-mail]:${GH_USER}/dagger.git
git fetch --all

# Update your main branch to be cloak (run similar commands for any other local branches you have)
OLD_BRANCH_NAME="main"
NEW_BRANCH_NAME="cloak"
git checkout ${OLD_BRANCH_NAME}
git checkout -b ${NEW_BRANCH_NAME}
git rebase -i ${DAGGER_REMOTE_NAME}/cloak
git push --set-upstream ${USER_REMOTE_NAME} ${NEW_BRANCH_NAME}

It enforces the same opinion on what's upstream and what's origin that I was using before, but can be easily adjusted. Anything you would change here?

[wet mason]

@civic yacht https://github.com/dagger/dagger/pull/3127

[civic yacht]

My immediate thought is that it would be nice if dagger-classic was like any other script/frontend, not a special case, which makes me think dagger-classic should actually be a separate repo ( dagger/classic?) and thus also a separate binary. But not very high confidence

[wet mason]

As long as they don't have changes open, they could just clone dagger and "start over"?

[final walrus]

Team, is cloak made fully public? 🎉

[wet mason]

Not yet 🙂 But preparing

the code itself lives in a dagger/dagger branch now, we're working on sending updates / instructions

[civic yacht]

Yeah that's true, I was just thinking about those with local branches with changes and such, but maybe that's rare enough to not matter

[wet mason]

@civic yacht I'd give those instructions as a fallback "got changes? -->"

otherwise, long term, probably best if they switch to dagger/dagger

[civic yacht]

Yeah exactly, the default is to just switch to dagger

[wet mason]

I'm not sure, classic is a "client" to cloak. They'll share zero code (classic has everything CUE, and needs to call cloak through the API to do stuff), so it probably makes more sense to be a separate codebase. Either WITHIN dagger (e.g. dagger/dagger/classic) or as a separate repo, perhaps

[ancient kettle]

I'm down to separate it. Personally, I'd prefer dagger/cue or similar.

[wet mason]

but good point, let's hold for a sec. If that's what we want, I should've git mv'ed * into classic instead of rm'ing and starting over. @civic yacht ?

[civic yacht]

Was this intentional? Once the PRs are all migrated we should archive it (or maybe we can archive already if that doesn't interfere with pr migration)

[wet mason]

yes, can't mv issues between a private and public repo

@civic yacht we should push an update to the README pointing to dagger/dagger and archive

[civic yacht]

I think it should be a separate repo too: #maintainers message
classic can also have the same history of dagger europa and branch off a different direction

[wet mason]

but, just wanted to triple check dagger/dagger is how we want it to be

[wet mason]

waiting for all PRs to be moved first (can't close them once archived I think?). Migrated all mine already

[civic yacht]

I think it is.

I just remember that we need to update cloak.yaml, go.mod and package.json to all point to the new repo. Those are scattered all over (need to include update instructions in whatever brief message we have here), but I'll update ones I know about (todoapp repo, @tidal spire's extension repos, etc.)

And we can probably get rid of most of the instructions about setting up SSH agent!

(appending to todo list)

[wet mason]

^^^ new readme. good?

@civic yacht I've put your updated instructions on every single cloak PR

I'll give people some time to migrate, then archive cloak

[civic yacht]

Yeah I figure it may take a bit given friday+timezones

@twin crow Just noticed this points to your fork while adjusting dependencies to point to dagger/dagger: https://github.com/dagger/dagger/blob/456c17ede354919030b01b6c641547a40c313520/examples/python/hello/requirements.txt#L1

Is that still correct? Or should I point it to github.com/dagger/dagger.git@cloak now?

[twin crow]

indeed, now the PR is merged, it should point to dagger/dagger

[civic yacht]

Awesome, I'll just include that with my changes then

[still garnet]

I like dagger/cue too from the perspective of "this is how Cue looks on the Cloak (now Dagger) platform." calling it "classic" kind of gives it an expiration date imo. who knows, maybe a bunch of passionate Cueists show up and keep it going?

[ancient kettle]

Yeah, that's how I feel about it, as well.

[wet mason]

👍

due? 🙂

[ancient kettle]

dagger-due

[wet mason]

dagger-due do is a mouthful 🙂

[ancient kettle]

😄

Hopefully not a "mouthful"

[wet mason]

but yeah i think dagger/cue and dagger-cue are good candidates

[hybrid widget]

Hi guys, as I'm building an image with cloak, is there a way to change the CMD or ENTRYPOINT of the final image?

[wet mason]

only with the new API, which is not yet implemented

[cosmic cove]

A few Cloak Repo Updates:

What -

• We migrated the Cloak repo into the Dagger public repo on the cloak branch.
• Please open any Cloak issues on the https://github.com/dagger/dagger/tree/cloak repo with the cloak label
• The #maintainers channel will now be public on the Dagger Discord server

**Why - **
The experiment started in a separate repository, but we are now migrating it into the main Dagger repository. “Cloak” is the codename of our next major release of Dagger, and going forward we will be developing it in the open.

The stable cloak release is still several months away. You can continue using Dagger as usual, and we will offer backwards compatibility in the future.

We still have lots of work to do on onboarding, features, and documentation, so we look forward to having your contributions as part of our next Dagger release!

[civic yacht]

Updated all the guides and other leftover urls, everything works now without SSH agent! https://github.com/dagger/dagger/pull/3129

[tepid nova]

I propose renaming this channel to #maintainers and retiring #dev

To clarify that it’s ok to discuss cloak in all channels, ask for help about it in #help-old-do-not-post etc

FYI my next side project is to try code-first extension development in go. If anyone is interested in that let me know!

[civic yacht]

I am highly interested, hope has been to move to that once done with multiplatform, but there’s probably 2 days of work left on that so don’t let me block you on trying stuff out

[tepid nova]

Giving this a try, we can revert if it feels weird

[civic yacht]

Updates todoapp: https://github.com/dagger/todoapp/pull/7 Also updated to use raw queries in go

[tepid nova]

@civic yacht I wonder if we could also do withArchitectures plural? Can you return an array of Query ? Wouldn’t that give us a multi-arch matrix feature for free?

[tepid nova]

Issue: https://github.com/dagger/dagger/issues/3131

[civic yacht]

Yeah I mention in the pr that we need a type that can bundle different platform versions of the same FS together. We also need the ability to split them out and refer to each platform fs individually to support use cases like local exports of binaries built for different platforms to different local dirs.

The bundled FS idea would create the same end effect as an array of query I think. It also might work better with making multiplatform image pushes easier, but i like the idea of returning an areay of query too. Worth keeping in mind in there future, probably will come in handy in other ambient config use cases

This should all fit in with the new API perfectly I think. A container can be associated with multiple platforms but theres also a field that lets you select an individual or subset of platforms from the current set too.

Good chance I won’t implement all that immediately just to save time, but feel good about the plan now, much less nervous multiplatform is going to be a huge wrench like before.

[tepid nova]

I agree that is a good feeling! After having to put it off for so long in 0.1 and 0.2

If you're still around @civic yacht, I'm wondering what this line is for: var _ router.ExecutableSchema = &MySchema{} it seems required in every *.schema.go but I can't figure out what it does

Some sort of kickstart for type introspection?

[civic yacht]

It’s just a way of doing a type assertion, if you forget to implement a method for the interface then you get a clean error. It also makes it clearer for readers what interface this type is supposed to implement

[tepid nova]

Ah! So simple

Too much python, it twisted my mind to see vodoo monkeypatching everywhere

Thank you

[hybrid widget]

Can someone please help me understand why these two images behave differently? The first one runs apache in its CMD but the second doesn't...

$ docker run  php:8.1-apache
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
[Sat Sep 17 06:11:06.065706 2022] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.54 (Debian) PHP/8.1.9 configured -- resuming normal operations
[Sat Sep 17 06:11:06.065751 2022] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
^C[Sat Sep 17 06:11:09.326321 2022] [mpm_prefork:notice] [pid 1] AH00169: caught SIGTERM, shutting down

query {
  core {
    image(ref: "php:8.1-apache") {
      pushImage(ref:"vikramatdagger/testrepo:latest")
    }
  }
}

$ docker run  vikramatdagger/testrepo
docker: Error response from daemon: No command specified.
See 'docker run --help'.

[tepid nova]

At first glance, this looks like image metadata (in this case entrypoint or cmd) getting lost in the pull, which is a known issue. I could be wrong though.

I believe there is a PR open to fix this

[hybrid widget]

OK, thank you

[tepid nova]

For reference: https://github.com/dagger/cloak/pull/220

Again I may be wrong about the cause. But if I'm right, then it's a high priority item to fix it, and this PR is a first step towards accomplishing that

(note that this PR needs to be migrated to the dagger/dagger repo)

[hybrid widget]

Thanks! This explains why my heroku deployment kept failing with No command or entrypoint specified for process type web.

[tepid nova]

Sorry about that. The short explanation is that this particular part of the buildkit API does not have batteries included, it requires lots of extra plumbing code to work

[hybrid widget]

Np

Discussion on API doc generation

[crude trench]

Just to be clear, does cloak drop cue as the configuration and basically turn it into GraphQL via apis? or are they somehow working together

[tepid nova]

Yes, Cue is dropped from the core in favor of graphql. But there will be a compatibility layer (“dagger classic”) to translate Cue configurations to graphql call. We hope to make that layer completely transparent.

[crude trench]

Great! I haven’t played around with cloak yet but I imagine it’ll solve the biggest pain point I had with the cue based configuration, being able to discover how to get the things I wanted to run! (That graphql playground is genius)

Kinda reminds me of the same way GatsbyJS was mind blowing when it came out

[hasty basin]

updated my "Jenkins agent with Cloak" build (to remove ssh stuff no longer needed in public repo!).

Soon, I'll make a version of this in pure Cloak, but this one is still in CUE: https://github.com/jpadams/helloworld-dagger-jenkins/blob/cloak/build_jenkins/agent/jenkins-agent-cloak.cue

But! It gives my Jenkins agents the ability to run Cloak 🙂
https://github.com/jpadams/helloworld-dagger-jenkins/blob/cloak/Jenkinsfile#L16-L17

[hybrid widget]

Docs versioning question

[copper snow]

If anyone wants to use/install cloak with Nix, look here: https://github.com/sagikazarmark/cloak-flake

[radiant vortex]

Are things like approvals on the roadmap for cloak? or is that perhaps outside the scope of the tool?

[tepid nova]

approvals

[slate wave]

is there a minimal example for the Embedded Go SDK use case referenced in getting-started (or a project that uses the sdk someone here created)?

[tepid nova]

Example of using Go SDK

[hasty basin]

Is yarn deploy working for folks in in https://github.com/dagger/todoapp on branch cloak?

Getting errors with yarn deploy in https://github.com/dagger/todoapp on branch cloak:

ClientError: Cannot query field "netlifyDeploy" on type "Filesystem".: {"response":{"data":null,"errors":[{"message":"Cannot query field \"netlifyDeploy\" on type \"Filesystem\".","locations":[{"line":9,"column":11}]}],"status":200,

name: todoapp
dependencies:
  - git:  
      remote: https://github.com/dagger/dagger.git
      ref: cloak
      path: examples/netlify/ts/cloak.yaml

extend type Filesystem {
  netlifyDeploy(
    subdir: String
    siteName: String
    token: SecretID!
    team: String
  ): SiteURLs!
}

[hasty basin]

hmmm...working for me now...

I might have had a cloak binary symlink somewhere built from dagger/cloak repo instead of dagger/dagger branch cloak...possibly 😄, but that shouldn't affect embedded engine...hmmm

[civic yacht]

Ah no that should affect the embedded engine. For nodejs, it invokes the cloak binary wherever it finds it from PATH

[hasty basin]

Oh, okay. Makes sense

[civic yacht]

I believe there's an issue about switching that to use a cloak binary from npm (will make one if not), so should be simpler once we make that switch

[hasty basin]

How do you use -e aka --secret with cloak do?
I'm running from a bash script:

URL=$(cloak -p ./cloak.yaml --secret "token=$TOKEN" do<<EOF
query Deploy {
  netlify {
    deploy(
      contents: "$BUILT",
      token: ??????,
      siteName: "jeremy-foo",
      subdir: "build",
    ) ...

I was originally running three cloak do s (AddDeployToken, Build, Deploy) where I set the secret in the first invocation, but then I couldn't look up the secret when I needed it in the third invocation:.

[civic yacht]

Should be like this

URL=$(cloak -p ./cloak.yaml --secret "token=$TOKEN" do<<EOF
query Deploy($token: SecretID!) {
  netlify {
    deploy(
      contents: "$BUILT",
      token: $token,
      siteName: "jeremy-foo",
      subdir: "build",
    )

[hasty basin]

Will try now, thank you!

deploy.sh

#!/bin/bash

BUILD=$(cloak -p ./cloak.yaml do<<EOF
query Build {
  core {
    git(remote: "https://github.com/dagger/todoapp", ref: "cloak") {
      yarn(runArgs: "build") {
        id
      }
    }
  }
}
EOF
)
CONTENTS=$(echo -n $BUILD | jq -r '.core.git.yarn.id')

DEPLOY=$(cloak -p ./cloak.yaml --set "contents=$CONTENTS" --secret "token=$TOKEN" do<<'EOF'
query Deploy($contents: FSID!, $token: SecretID!) {
  netlify {
    deploy(
      contents: $contents,
      token: $token,
      siteName: "jeremy-foo",
      subdir: "build",
    ) {
      deployURL
    }
  }
}
EOF
)
URL=$(echo -n $DEPLOY | jq -r '.netlify.deploy.deployURL')
echo $URL

cloak.yaml

name: deploy
dependencies:
  - git:
      remote: https://github.com/dagger/dagger.git
      ref: cloak
      path: examples/alpine/cloak.yaml
  - git:
      remote: https://github.com/dagger/dagger.git
      ref: cloak
      path: examples/yarn/cloak.yaml
  - git:
      remote: https://github.com/dagger/dagger.git
      ref: cloak
      path: examples/netlify/ts/cloak.yaml
  - git:
      remote: https://github.com/dagger/todoapp.git
      ref: cloak
      path: cloak.yaml

Invocation:

$ TOKEN=`cat ~/netlify_token` ./deploy.sh

Works! Thanks @civic yacht ! 🎉

[hybrid widget]

Hi - any idea why this is happening?

node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/vikram/public/examples/node/node_modules/@dagger.io/dagger/sdk/nodejs/dagger/dist/index.js' imported from /home/vikram/public/examples/node/scripts/build.mjs

package.json has

...
  "devDependencies": {
    "@dagger.io/dagger": "git+https://github.com/dagger/dagger.git#cloak"
  },  
  "dependencies": {
    "cookie-parser": "~1.4.4",
    "debug": "~2.6.9",
    "express": "~4.16.1",
    "http-errors": "~1.6.3",
    "jade": "~1.11.0",
    "morgan": "~1.9.1"
  }

[wild zephyr]

👋 added some thoughts about the bask SDK here https://github.com/dagger/dagger/issues/3065

[twin crow]

should we merge this channel with #dev?

[tepid nova]

IMO yes 👆 🙂

[twin crow]

moving the thread about the core api

[tepid nova]

Yeah makes more sense to keep this one and retire the other (more cloak history here)

[hybrid widget]

Please help with a GraphQL query to copy files with exclusions

[hasty basin]

Refined this some more with @tidal spire. We got it down to a single nested query. Looking good in bash 🙂

deploy.sh

#!/bin/bash

DEPLOY=$(cloak -p ./cloak.yaml --secret "token=$TOKEN" do<<'EOF'
query Deploy($token: SecretID!) {
  core {
    git(remote: "https://github.com/dagger/todoapp", ref: "cloak") {
      yarn(runArgs: "build") {
        netlifyDeploy(
          subdir: "build"
          siteName: "jeremy-new"
          token: $token) {
            url
         }
      }
    }
  }
}
EOF
)
URL=$(echo -n $DEPLOY | jq -r '.core.git.yarn.netlifyDeploy.url')
echo $URL

myapp
├── cloak.yaml
└── deploy.sh

[civic yacht]

PR for initial multiplatform support is ready for review. It's basically just enough of the implementation that we can feel confident going forward in how to fill in the rest of the associated features, no more. Docs have more details, let me know if they make sense: https://github.com/dagger/dagger/pull/3119/files#diff-f0054461faf24ccab86b829e7af5012000af602fcfabeb8b5f35dbc91415d035

[tepid nova]

That’s amazing.

Could it support @orchid cosmos ‘s awesome tooling eg. https://github.com/crazy-max/goxx ?

ie could he build these on cloak instead of regular buildkit/buildx ?

[civic yacht]

Yeah based on reading through the README it should work.

In the example there you do FROM --platform=$BUILDPLATFORM to pull the goxx image. In dagger that's now the equivalent of doing withArchitecture(architecture: "host")

Then later they use TARGETPLATFORM to enable cross compilation. We can do that too by just setting that TARGETPLATFORM value as an env var. The docs in the PR have an example of basically this (also have a test case for it): https://github.com/dagger/dagger/pull/3119/files#diff-f0054461faf24ccab86b829e7af5012000af602fcfabeb8b5f35dbc91415d035

That being said, it would be great to confirm that all in practice. The cross-complation test case I have is essentially the same idea, so it won't be hard to give it a try quick tomorrow to switch to pulling his goxx image.

[tepid nova]

One note, $BUILDPLATFORM and $TARGETPLATFORM are set automatically by buildx. We need to think about how to get the equivalent values. The latter could be a regular argument, and the former perhaps host { platform }?

(forgot question mark 🙂

[civic yacht]

The approach I follow in the PR resulted in there being no need to explicitly obtain $BUILDPLATFORM (you can just submit queries that are wrapped in withArchitecture(architecture: "host") but if we wanted to make it accessible via an API call like host {platform} it would be completely trivial

And yeah the value you set when you use withArchitecture gives you the closest equivalent of TARGETPLATFORM.

The approach I went with here is inspired by the buildx one but slightly different, so none of the concepts translate 1-to-1. However, the goal was that the same end abilities buildx gives you should also be available here in dagger, even if they are packaged up slightly differently and more specialized for our use cases.

[scarlet gate]

Is it possible to execute a bash file within my gql statement?

[wild zephyr]

@scarlet gate you mean adding an inline bash script? or running a bash file in your current project?

[scarlet gate]

I mean adding an inline bash script.

[scarlet gate]

I can show you my scenario.

[civic yacht]

Added a commit to the PR to have the test case use goxx, worked pretty seamlessly! https://github.com/dagger/dagger/pull/3119/commits/540b7fe9288280afbad266bfb77450c402cd48f3

[hasty basin]

@twin crow @stray heron @wet mason Would be a good time for me to merge this PR. https://github.com/dagger/dagger/pull/3143

Because that ☝️ wasn't in place, we got two tags on same commit during last successful auto-release:
commit 46adf1fbd36cd8bd1052ed4e19f10f26df39540d (tag: v0.2.35, tag: v0.2.34)

I'm planning to merge the above PR. If we have any issues during auto-release, we'll know where to look. After auto-release, we can re-run the auto-release workflow to test that we don't double tag anymore.

Autorelease is set for 17:06 UTC, in about 11 minutes.

Merged ✅

[tepid nova]

what's the canonical way of running integration tests for cloak core api? should go test be enough or is there some other required incantation?

[scarlet gate]

I would expect go integration-test or int-test

[civic yacht]

I do go test -race ./... from the root of the repo. Or if I want to run just one test something like go test -race -run TestMultiplatformExtension $(pwd)/core/integration/

[final walrus]

When I use an extension in my project setup, can the client side scripts be generated from the extension schema?
E.g. are the files in https://github.com/dagger/todoapp/tree/cloak/scripts/deploy handwritten or generated from the netlify extension schema file?

[tepid nova]

The scripts are not generated. However a client library used by the script could be generated. We started experimenting with this, but it's not quite ready yet.

[final walrus]

Any GH issue I can subscribe to to follow up?

[civic yacht]

Looking for an existing one, will make it and point it your way if not

[civic yacht]

https://github.com/dagger/dagger/issues/3077

[tepid nova]

Any objection to splitting the git integration test into a separate git.schema_test.go file?

[civic yacht]

No not at all

[tepid nova]

should each file have its own init() ?

[civic yacht]

no should only be needed once per package (I guess init should just be in its own file then too)

[still garnet]

fyi I mentioned doing this here but happy to adjust my PR if your change lands first: https://github.com/dagger/dagger/pull/3154

i'm being cautious with moving too many things in that PR so we have a diff that's easy to grok (instead of two whole new files)

[final walrus]

Can I set a local path in dependencies during the development of an extension rather than a git repo?

[tepid nova]

I would definitely be grateful if my puny little PR could be grandfathered into the upcoming sweeping changes 😇

but happy to rebase too if that's easier

I can remove the test too and let @still garnet add it back. Basically just tell me what I should do 🙂

https://tenor.com/view/conker-gif-22582027

[civic yacht]

Yes example here: https://github.com/dagger/dagger/blob/cloak/examples/yarn/cloak.yaml#L6

Just need to be careful that if you are using ../ then you have to run cloak from a directory encompassing all of the dependencies (i.e. cloak do -p examples/yarn/cloak.yaml for that one)

(that's currently a limitation, probably ways of improving it in the future)

[still garnet]

@tepid nova let's merge your PR first, it'll be easy for me to just remove the two git tests from my PR

i just dealt with a merge/rebase (i played myself by merging my image config PR) so I can do it again 😆

[civic yacht]

(taking a look at it now)

Merged it

[still garnet]

sweet

ok, https://github.com/dagger/dagger/pull/3154 is rebased

[civic yacht]

@still garnet saw your comment, I'll take a break from the go dx and update the typed resolver PR so we can get rid of the conflicts asap, just a little bit needed for rebasing and converting the rest of the resolvers

[still garnet]

sounds good, ty!

[final walrus]

I added "@dagger.io/dagger": "git+https://github.com/dagger/dagger.git#cloak" to my package.json, but after installing all the dependencies, I get this on build:

index.ts:1:59 - error TS2307: Cannot find module '@dagger.io/dagger' or its corresponding type declarations.

1 import { client, DaggerServer, gql, FSID, SecretID } from "@dagger.io/dagger";

Looking into my node_modules folder, node_modules/@dagger.io/dagger only contains the files package.json, README.md and LICENSE. The module itself from ./sdk/nodejs/dagger/dist/ isn't there. What am I doing wrong here?

[civic yacht]

@hybrid widget ran into the exact same thing yesterday: #1023972500784287764 message

Ultimately I couldn't reproduce it and he couldn't reproduce it anywhere besides his local env, so we weren't sure what was going on... I'm not sure if he was able to fix his local env eventually

My best advice for the moment is to try updating nodejs versions and see if it helps? Unfortunately that's all we know, but if that manages to fix it let us know

cc @dense dust @mellow bolt in case this sort of issue rings any bells ^

[final walrus]

I'm on latest NodeJS v16.15.0 + NPM v8.5.5, which is (almost) the latest versions available via Homebrew.

[civic yacht]

I wish I had any better short term ideas, it's quite an odd issue. With Vikram we tried clearing the cache (npm cache clean --force) and deleting node_modules. It didn't work for him but worth a quick shot here.

Other than that the only things I can think of immediately are to try upgrading to the actual latest versions of node+npm (even just a minor version upgrade). Or finally maybe try yarn? I'm not a yarn fanatic or anything, I just am not sure what else to try immediately.

Making an issue for it either way

[final walrus]

When I remove the files: section from package.json, it works with NPM v8. You can try with my fork:
https://github.com/ringods/dagger/tree/npm

[civic yacht]

That's awesome, thank you so much for finding that. I need to double check whether we were relying on that (seems like we may be able to use .npmignore instead), but glad you're unblocked for the moment at least

[hybrid widget]

My package.json only has this and it doesn't work for me. The only way I could get it to download the missing sdk was to mount my app dir into a node:18 container, shell and run npm install from the container shell.

{
  "dependencies": {
    "@dagger.io/dagger": "github:dagger/dagger#cloak"
  }
}

[tepid nova]

So... about dagger.gql 🙂 https://github.com/dagger/dagger/issues/3016

[final walrus]

@hybrid widget the problem is not in your package.json, but in the package.json of the Dagger/Cloak project. If you replace the Dagger dependency in your package.json with my forked Dagger ("github:ringods/dagger#npm"), it should work.

[wild zephyr]

Ringo, @hybrid widget just wondering... does the same happen witn yarn?

[mellow bolt]

@final walrus Just tried to do a repro with Node v16.15.0 and npm 8.5.5 and I get the same result as you. Bumping the version to 16.15.1 which use npm 8.11.0seems to fix the problem but I did some research and it seems that the problem was coming from the use of the following syntax:

  "files": [
    "./sdk/nodejs/dagger/dist/"
  ],

instead of:

  "files": [
    "/sdk/nodejs/dagger/dist/"
  ],

Notice that I've removed the dot ./ => /
With this change, it works with node 16.15.0 + npm 8.5.5
I'll create a PR for this one and let you know (cc @hybrid widget @final walrus )

[mellow bolt]

Don't forget to remove your package-lock.json file otherwise it will keep install the previous version of @dagger.io/dagger

[wild zephyr]

package managers... 😩

[civic yacht]

Merged it, thank you so much @final walrus @mellow bolt for looking into it and fixing it! It was one of those true headscratchers...

[cosmic cove]

@here Hi all! Just a reminder to add your topics, questions, and demos for the community call tomorrow to the doc below. We have an exciting demo from @scarlet gate planned, but I know there are more of you that have some exciting things to share 🙂 https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[rancid turret]

Hey, as I understand it, the query builder approach makes it easier to chain but also allows less calls to the API by building a single query if possible. Is one of these a nice side effect of the other from the initial intent, both were, or are there other needs for it?

[civic yacht]

Primary motivations (imo) are:

• Doesn't require pre-defining fixed operations and then creating clients from those. All you need is the schema and then clients build the query on the fly in their code. Related to what you said about making chaining easier.
• Closest thing we can get to a totally natural feeling of coding in each language (you may not even realize you are using graphql) while also holding the above point true

The benefits of less API calls is quite nice, but more secondary since it's ultimately mostly a performance optimization

[rancid turret]

Got it. Curious as to what you mean by "natural feeling of coding" as opposed to an auto-generated client (assume you don't need to create an operations file).

[civic yacht]

I just mean it will look like you are just using a builder pattern essentially, it shouldn't require any sort of graphql specific knowledge. Andrea's early hypothetical example here kind of hints at it: https://github.com/dagger/dagger/blob/9af193d1e239083461fdab27ad3e9a866e499594/sdk/go/dagger/core/core_test.go#L12-L16

To be clear, the query builder will actually include generated code too, it will be generated from the schemas rather than operations files though

[rancid turret]

Yes, I just mean that core.Image(ref="alpine") looks more natural than Query().Select("image").Arg("ref", "alpine"). Query builders, in their dynamic nature, depend a lot on strings that can have typos without the IDE knowing about it. We can reduce that a bit by using auto-generated code for types and query builder for operations, and convert the types to their equivalent strings. But still...

The other downside is that we no longer have existing code generation support (most likely) for that. Perhaps we can create a plugin that generates the operations on the fly and then generate the client from that. But you'll lose that performance optimization. It may be possible to also hook into that generation and add some query building features on that client, I'm just thinking aloud.

I know the query building DX can be much improved, I have lots of experience with Django's ORM which does the same with SQL queries.

[civic yacht]

Oh totally, this won't be a query builder like others that exist. It will require completely custom tooling on our part and have generated methods for each graphql field rather than relying on strings (so you get type safety, autocompletion, all that good stuff). That's the tradeoff here; it require a lot more work on our part but if we can achieve the end vision it will be both incredibly nice to use and incredibly flexible (since its all autogenerated from arbitrary schemas).

[tepid nova]

I love the idea that you could query the Dagger API itself for a generated client in the language of your choice

[civic yacht]

That's essentially how the generated code works today: https://github.com/sipsma/dagger/blob/a1308896202d85adb6077c002ff819c07a2d392f/core/project.schema.go#L60-L60

So it should be natural to do the same with the query builder stuff too thankfully

[tepid nova]

That reminds me that we should include these calls in the core api redesign. Right now it's left undefined what happens to it

(I guess we can leave it private/undocumented for now)

[civic yacht]

Good point; I think updating to use dagger.graphql instead of yaml will be a natural point to think about that

[tepid nova]

@civic yacht would you be OK to show a sneak preview of your multi-platform PR in the community call tomorrow? Doesn't have to be a sophisticated presentation. It's such a frequently requested feature... the fact that we now have it working, even in early state, is huge.

[civic yacht]

Yeah totally, happy to show off some of the basics there. I also seem to be getting close to a very-early-but-functional prototype of code-first go schemas, so if there's time I can show that too

Or save that for next week

[tepid nova]

That sounds great too.

[wise field]

I've got cloak dev running, but besides a UI for graphql I'm not sure what Im seeing or what to do with it. What language is this? { alpine{ build(pkgs:["curl"]) { exec(input: {args:["curl", "https://dagger.io"]}) { stdout(lines: 1) } } } }

[tepid nova]

graphql 🙂

[wise field]

🤦‍♂️ of course

[tepid nova]

see the README in cloak branch (it links to docs)

still work in progress

[wise field]

I did but they werent telling me. at least not plainly enough

[civic yacht]

@wild zephyr I just realized what's happening.... It's related to this issue: https://github.com/dagger/dagger/issues/3072

The end goal for the engine is that when you load extensions you just get access to their schemas only. But right now there is only one global schema. So what happens is you are loading the yarn extension still, which has a dependency on the alpine extension, so you still end up loading that alpine extension and seeing it because it's all the same global schema.

If you remove the yarn extension from cloak.yaml too then you will no longer see alpine. So kind of a silly thing, just very confusing when you see it happen in practice right now. Once we resolve that issue it will all make much more sense.

[wild zephyr]

💪 makes total sense Erik. Thx for investigating 🕵️

[hybrid widget]

@mellow bolt it works for me as well now. Thanks

[tidal spire]

quick little demo of go+mage, files here: https://github.com/kpenfound/dagger-demos/tree/main/go/hello

[civic yacht]

Was just able to invoke alpine where the schema is fully derived from the go object in the extension, no schema.graphql needed: https://github.com/sipsma/dagger/tree/goxdx/examples/alpine

A lot more needed to be robust, handle more types of arguments, returns, etc. But the basic idea is workable! There's hope 🥲

Also @tawny flicker I used the reflection code in your PR to help get going, so greatly appreciate the help there 🙂 https://github.com/dagger/cloak/pull/171/files

[ancient kettle]

That's awesome, @civic yacht!

[tepid nova]

There's a direct connection with @stray berry 's demo just above. I imagine this would make it easy to "bridge" from internal helper functions like his image and workdir to extensions?

[twin crow]

if anyone is looking for some docs to onboard on the next cloak release, @hybrid widget is working on a "getting started" guide that I find already pretty useful:

• Pull request: https://github.com/dagger/dagger/pull/3171
• Preview URL: https://deploy-preview-3171--cloak-docs.netlify.app/bvtz9/get-started

Feedback on the PR is welcome!

[wet mason]

@civic yacht @still garnet So, the services PR still needs a bunch of work/rework (e.g. not scoped in the new API, per process control, testing, ...)

I'm thinking it's less urgent for now. I've rebased the PR (to use the new router.ToResolver and a bunch of other breaking changes) and was thinking to just disable it (e.g. not instantiate it in the router) and hide the cloak attach command for now, so at least we can merge it so the code stays up to date and then we can at some point rework the schema etc once we have time.

Thoughts?

[civic yacht]

Not sure IIUC 100%, but I would say that this code-first approach should make it much easier to just wrap big chunks of functionality into a callable API with little more than copy paste, if that's what you are getting at? There's also a connection here between the whole idea of taking a script and converting it to an extension; this code-first approach doesn't solve every problem there but does likely cut out a big chunk of the work.

[civic yacht]

Yeah I'm totally fine with hiding it for now while we figure out the API. I would love it if there was some secret escape hatch to still open a shell sometimes before we get all the details figured out; there are many times day to day I have to resort to the equivalent of println debugging from containerized code that would be much easier with it. But not a huge deal of course

[wet mason]

@civic yacht Ok. How about I uncomment I but we pinky swear we'll comment it back for preview release if we don't get a few cycles to rework the API?

[civic yacht]

Haha that works for me, or honestly I'm fine locally uncommenting it as needed and pinky swearing that I remember to re-comment before sending out PRs

[wet mason]

At least we get some dogfooding from it, maybe it'll influence the API rework

[tepid nova]

on that note I encourage you to review the api reference doc in @hybrid widget ‘s docs pr

there’s stuff in there we may want to hide, and maybe we use the same thing to hide the service “secret hatch” and project calls?

[wet mason]

@civic yacht The main items were:

• Convert to New Core API
• Restructure to be multi-process (just like buildkit)
• Support non-interactive mode (e.g. for launching a db etc, we don't want tty streaming etc)
• cloak attach is hardcoded to localhost

Each of them are API breaking

[civic yacht]

Sounds great, we should just make an issue for that (if it doesn't exist yet)

[wet mason]

https://github.com/dagger/dagger/issues/3173

[wet mason]

@civic yacht @still garnet @tepid nova I resurrected the old query builder branch (https://github.com/dagger/dagger/pull/3125), to try and write "manual" bindings (in a codegen fashion) on top of the new API (using @still garnet's latest directory/file implementation)

It kinda works:

tree := Git("github.com/dagger/dagger").
    Branch("cloak").
    Tree()

files, err := tree.Contents(ctx)
require.NoError(t, err)
require.Contains(t, files, "README.md")

readme, err := tree.File("README.md").Contents(ctx)
require.NoError(t, err)
require.NotEmpty(t, readme)
require.Contains(t, readme, "Dagger")

I'll continue experimenting a bit (still a lot of clunky stuff), but I think it's workable

[civic yacht]

Awesome, yeah I think that in combination with code-first schemas will easily give us an order of magnitude improvement to the xdx

[wet mason]

Yeah your work on code-first schemas inspired me to continue the query builder shenanigans, it's the last remaining bit

there' still a bunch of weirdness to figure out in there thogh

[tepid nova]

this makes me very happy 🥲

[civic yacht]

Yeah totally, my instinct is that we can find reasonable compromises to get it to the point where it's good enough for at least 95%+ of cases and those remaining few users can always just fallback to raw queries

[wet mason]

@civic yacht yeah, major weirdness is deciding how to slice up queries (e.g. right now I'm doing so at the gql leaves -- scalar fields, the rest is just query building on top). But there's trade offs. I'll open up a thread at some point

[still garnet]

this looks awesome! planning to start on Container tomorrow, excited to see how that could look

[tidal spire]

I'll be able to delete like 80%+ of my demo code when all of this makes it into #cloak 🥲

[obsidian rover]

Is someone available for some Go skills (go kung-fu) ? Using //go:embed I am limited by the fact that no go file is at the root of the project: when embedding our entire codebase at compile time, I need to go: embed from the goroot

Is it possible to create a go package at the root of the project and reference it in any of the other package ? Or show me how to import part of the main package into a subpackage ?

[obsidian rover]

@civic yacht I am discarding embed for now as it conflicts with packages (pattern sourceCode: cannot embed directory sourceCode: in different modulego list). pkger currently seems to be the best library. Keeping you in touch

[still garnet]

@obsidian rover (sorry, missed your message last night) is this the error? https://github.com/golang/go/issues/45197 i remember running into this too, go:embed gets really unhappy when it runs into another Go module

something like this seems doable: https://github.com/golang/go/issues/45197#issuecomment-891192133

[civic yacht]

You can try just moving this file to the root of our repo: https://github.com/dagger/dagger/blob/cloak/cmd/cloak/main.go

That's a fairly normal pattern I've seen in the past

[civic yacht]

I ran into that previously when trying to add go.mod to the shim (and had the luxury in that particular case to "solve" it by just not doing that) but that's good to know there's a workaround, I missed that last time. I don't think it affects us at this time; there should just be one go.mod

[civic yacht]

Sorry, wrote this right after waking up, to be more clear you would move the main package to the root of the repo and then everything else besides the main func that's in cmd could be implemented as a library that gets imported into main (probably under internal). Or you could just move literally everything under cmd/cloak to the root of the repo temporarily to verify the idea

[obsidian rover]

Ok, it turns out I just had to declare a package named dagger at the top level, and I can import it wherever I want 🤯 Thanks @civic yacht Such a love-hate relation with go modules 🤣

[obsidian rover]

Thanks for the alternative, we managed to fix it cleanly

[wild zephyr]

best code is the one that you have to delete 😛

[civic yacht]

#maintainers message <- should probably be a pinned message

[obsidian rover]

Fully "slim" auto-served version of the bundling This was the last blocking part 🙏

[hasty basin]

using the alpine, netlify, and yarn extensions from a new minimal container exec environment and getting this:

env: [{name: \"YARN_CACHE_FOLDER\", value: \"/cache\"}, {name: \"GIT_SSH_COMMAND\", value: \"ssh -o StrictHostKeyChecking=no\"}], cacheMounts: {name: \"yarn\", path: \"/cache\", sharingMode: \"locked\"}, sshAuthSock: \"/ssh-agent\"}\n      ) {\n        mount(path: \"/src\") {\n          id\n        }\n      }\n    }\n  }\n}"}}
#28 1.216     at /src/examples/yarn/node_modules/graphql-request/dist/index.js:395:31
#28 1.216     at step (/src/examples/yarn/node_modules/graphql-request/dist/index.js:63:23)
#28 1.216     at Object.next (/src/examples/yarn/node_modules/graphql-request/dist/index.js:44:53)
#28 1.216     at fulfilled (/src/examples/yarn/node_modules/graphql-request/dist/index.js:35:58)
#28 1.216     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
#28 1.216   response: {
#28 1.216     data: null,
#28 1.216     errors: [
#28 1.216       {
#28 1.216         message: 'NotFound: no ssh handler for id default',
#28 1.216         locations: [ { line: 6, column: 7 } ],
#28 1.216         path: [ 'core', 'filesystem', 'exec' ]
#28 1.216       }

Guessing some minimal ssh config is assumed/required?

[civic yacht]

Ah, we had previously updated yarn to use your SSH agent when present (which was fine because that was a requirement at the time), but now that it's not a requirement we should make it optional in yarn too

We should make an issue for fixing it, but yeah in mean time you can workaround by including an SSH_AUTH_SOCK or by deleting the line in the yarn extension itself that sets up the ssh agent mount

[cosmic cove]

I thought I saw a PR with the list of extensions, but now I can't find it. Does anyone know where the README of extensions is living right now?

[tidal spire]

https://github.com/dagger/dagger/blob/cloak/docs/unxpq-introduction.md#available-extensions

[cosmic cove]

Thank you! I thought it was a different page, so good to know.

@civic yacht are you creating an issue to update the documentation for @scarlet gate 's issue that he found in the community call? Just want to make sure someone makes an issue for it. We can get @hybrid widget 's thoughts on where we should put it for docs too

[hybrid widget]

@cosmic cove I'm capturing frequent Q&A in https://github.com/dagger/dagger/issues/3177 and will transfer these to the FAQ page. Feel free to add them there. I've already added the one you refer to above from the community call

[wet mason]

cloak release process decision tree: https://github.com/dagger/dagger/issues/3176#issuecomment-1262596646

There's open questions for @stray heron, @civic yacht, @ancient kettle, @hybrid widget, @mellow bolt

/cc @cosmic cove

[tidal spire]

Anyone want to pair real quick in #dev-audio on the demo error I had? I think I know why it failed, but I want to figure out how we can surface the error better 🙂

[civic yacht]

I can in one min

[civic yacht]

Just created one here: https://github.com/dagger/dagger/issues/3180 There is a lot of stuff that is going to change soon around that but nonetheless it absolutely needs to be documented whereever we land

@tidal spire issue for bubbling up the errors here: https://github.com/dagger/dagger/issues/3182

[hasty basin]

@tidal spire did you figure out anything. I had similar issues.

Got around ssh ones by launching ssh-agent, then hit something like: https://answers.netlify.com/t/build-command-failed-using-the-netlify-cli-on-gitlab-runner/43543/4

[tidal spire]

nothing around ssh for me, still trying to figure out the netlify part specifically

[hasty basin]

cool. I didn't have any issues from my mac, but did from a fresh debian container: #maintainers message

[obsidian rover]

@twin crow. FYI, your PR https://github.com/dagger/dagger/pull/3185, on the error messages will probably conflict with mine (I'll rebase, not a problem). [I was managing it on my PR, but it might be a faster fix]

[obsidian rover]

@wet mason, are we still relying on this Dockerfile ? https://github.com/dagger/dagger/blob/cloak/Dockerfile.

Not sure to fully understand its use, and I'm getting permission conflicts when running docker build -f Dockerfile.daggerd contextPath, only when running go test, which get fixed when renaming my Dockerfile to Dockerfile

[wet mason]

The only thing using it is playwithcloak AFAIK

[obsidian rover]

Ok thanks, I'll use my workaround then (a little more ugly, but will keep then, for now)

[wet mason]

What's the problem?

[obsidian rover]

To embed buildkit, I statically copy our codebase into a tmpdir, then, inside this tmpdir, I do a docker build on a specific Dockerfile configured to properly run cloak serve. However, as there are 2 Dockerfiles (the one above and mine) at the root level, I need to specify docker build -f myDockerfileName. However, I sometimes have permission issues that don't exist when I name my dockerfile Dockerfile. To be more specific, this issue arises only when I run the docker build with the dockerfile not named Dockerfile, from the go tests

The error I get is:

#1 DONE 0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount585092295/Dockerfile.daggerd: no such file or directory

I verified and the dockerfile exists and works properly when running with cloak dev.

My workaround works. It is just not pretty: erase the above dockerfile with mine, inside the tmpdir

[obsidian rover]

The workaround will be sufficient I think, I'll see with Erik 😇 Thanks

[hasty basin]

todoapp via Dagger and Jenkins
https://www.youtube.com/watch?v=UoIxbiMW_wQ

[wild zephyr]

👋 what's the difference between dependencies and extensions? Aren't those the same? 🤔

https://github.com/dagger/todoapp/blob/cloak/cloak.yaml#L9-L20

[civic yacht]

At this moment, extensions refer to the extensions that your project is newly defining. dependencies refer to other extensions that your project (e.g. scripts or extensions) need to run.

So it's the same as a go repo where you have your module and the other modules you depend on

When we switch to dagger.gql, we are going to make the config only support one extension at a time, so then there's no need to distinguish the two with different keys.

[wild zephyr]

gotcha, thx . I know that we'll be switching to dagger.gql, but dependencies was a bit confusing to me. I'd have imagined that everything could be under extensions with different ways of referencing remote, local or current project definitions.

Dagger+Cloak terms/glossary discussion: https://github.com/dagger/dagger/discussions/3194

[wet mason]

@still garnet Hey -- is there a way to get scratch with query { file } ?

err -- directory

[tepid nova]

get scratch with query { directory } ?

[still garnet]

woops missed this. directory { } should get you an empty/scratch directory, yeah. it doesn't do a Mkdir. note that I tweaked the behavior in https://github.com/dagger/dagger/pull/3190/commits/442b9e11e25b339f9dc78d6f94cf4019e5b1daad slightly since I noticed marshaling a llb.Scratch() to a *pb.Definition and back was causing weird behavior

but that PR is currently blocked on a timeout that's only happening in CI 😭

[wet mason]

ohhh -- my bad, I don't support yet nullable arguments and just realized id is optional

[still garnet]

yeah, it needs to be since there's no other way to bootstrap a directory with content; you always start from scratch and then do withNewFile or whatever

@wet mason maybe you could have .NewDirectory() and .Directory(id)? (assuming you're talking about the query builder)

[wet mason]

@still garnet it's more of a generalized problem with nullable fields, e.g. Directory#Contents() has an optional path (and NewContents here wouldn't make sense)

will probably make them pointers for now, but they're tedious to work with in Go (e.g. you can do Contents(&"/foo"), have to put that in a temp variable)

[still garnet]

true, though I wonder if you could apply a similar fix of just having different named methods whenever it comes up. there's probably a breaking point though. e.g. Directory.List() vs Directory.ListPath()

or ContentsOf, etc

another option would be the ...ContentsOpt pattern (LLB does this)

[wet mason]

True, that's neat. I think it makes sense up to ONE argument though, otherwise there's a matrix of combinations to generate (e.g. for a hypothetical 2 optional path/permissions args it'd be NewFilePath, NewFilePermissions, NewFilePathPermissions)

[still garnet]

yeah totally

[wet mason]

Yep. Was thinking this

[still garnet]

i think LLB has the generalized solution, but it's a lot of plumbing

doesn't feel too bad to use, but it can be hard to discover what all the options actually are

[wet mason]

Yeah, and I don't know how automatable it is

[still garnet]

like for codegen?

[wet mason]

yeah

[still garnet]

true, LLB tends to have semantic names

[ancient kettle]

@wet mason @civic yacht @still garnet I’m trying to make some decisions about which APIs to use for dagger-classic, especially so I don’t waste too much time reimplementing if and as things change (e.g. Filesystem vs Directory vs Container). Do any of you have a few minutes to chat about this in a few?

[still garnet]

I'm out on an errand atm, hopefully back in an hour or so, but if others can cover no need to wait

[ancient kettle]

That could work, as well.

I’ve got things I can do in the meantime.

[civic yacht]

Happy to but it would be better if Alex is there all else being equal, just so we keep the whole hive-mind in sync

[ancient kettle]

Let’s sync up when @still garnet gets back, in that case.

[tepid nova]

Query builder patterns for nullable fields

[still garnet]

@ancient kettle @civic yacht back!

[ancient kettle]

👍🏻 I'm ready whenever!

[still garnet]

hopping in dev-audio

[civic yacht]

@still garnet made this https://github.com/dagger/dagger/issues/3201 but where do I add it to the project/roadmap you were making?

[still garnet]

it's automatic as long as you add the area/core and cloak labels

which you already did 🙂

[hybrid widget]

With cloak, is there a way to build and tag a docker image only locally ie. without pushing to any registry? equivalent to docker build -t mytag

[wild zephyr]

not yet, here's the issue to follow. https://github.com/dagger/dagger/issues/3068

[still garnet]

question for container { withMountedCache }: I see we're not letting the user pass in a cache ID, is that because exposing it directly is a footgun for multi-tenancy? I can have it derive a cache ID from the target path + container ID of the direct withMountedCache parent, the trade-off being that any change to the parent will bust the cache, so you probably want to scope withMountedCache as far out as possible.

[tepid nova]

I think ID is optional in llb?

[tepid nova]

But no clear answer on whether to expose IDs, I just forgot to include it

It definitely does confuse people

[still garnet]

oh interesting, that seems to work but I wonder if it just treats "" as its own empty ID

[tepid nova]

but how much of the confusion was related to the Cue dx is unclear

[still garnet]

the main thing I'd be worried about is scoping those cache IDs, it'd be pretty easy for two people to make up the same ID string and then have weird cache collisions

since it's global to the buildkit engine afaik

[civic yacht]

I believe that the ID defaults to the target mount path if not provided. Also, cache mounts are often a source of confusion for users outside of dagger, so not really specific to us.

[still garnet]

hmm looking through buildkit's code it seems to key on cacheID != "" to determine whether it's a cache mount, and otherwise it's just a regular bind mount

e.g. https://github.com/moby/buildkit/blob/master/client/llb/exec.go#L354

[civic yacht]

Agreed, but that can also be a feature in addition to a bug, so it's kind of hard to say what we should do here. I think ideally we should try to come up with a better interface to all of this than what Buildkit provides, but that will also take some time to design and think about (including whether it's actually possible to implement on top of what buildkit currently provides)

[civic yacht]

Ah okay that default behavior is specific to dockerfiles then

[tepid nova]

Scoping definitely requires some thought, it ended up leaking everywhere in the Cue DX because you had to provide an “app name” or similar to anything wrapping cache mounts, so they could interpolate in the global cache ID

I really don’t know how that transposes to cloak DX

[civic yacht]

coincidentally someone in the community has been making up for buildkit's lack of docs on this subject in the last few days: https://github.com/moby/buildkit/issues/1673#issuecomment-1264502398

Which helps clarify how complicated it can get

[still garnet]

makes sense, i mean at least it's just a matter of chucking more values into the ID to scope it if/when we make it more sophisticated

[tepid nova]

My first instinct is to hide it fwiw

[hasty basin]

@tepid nova how will cloak API cutover from current to new happen?

[tepid nova]

Thread about cache mount IDs 🙂

[tepid nova]

Let’s talk about the dagger CLI 🙂 🧵

[wild zephyr]

👋 question about the core.project query. I start cloak dev in the #cloak branch and I run this query afterwards:

query MyQuery {
  core {
    project(name: "examples") {
      extensions {
        path
        schema
        sdk
      }
    }
  }
}

I was hoping to get all the extensions that the project is using but the extensions field returns empty. Is this expected? cc @still garnet @civic yacht

[tawny flicker]

Is it normal `go test ./core/integration takes ~ 50s to run? Seems a little long.
Can we do better?

[still garnet]

there are more tests now, and they're all integration, so it adds up quickly. they already run in parallel, probably limited to # cores by default; you could try a higher -parallel=N. it takes ~20s on my laptop (16 cores), but I usually just run the test I'm focusing on and do a full run before pushing