maintainers

[obsidian rover]

If you had to find the path of a binary, cross platform, in go. How would you do it ? Is there a library for that ?

[tepid nova]

your own binary? or from name?

[obsidian rover]

from name

[tepid nova]

I think there’s a stdlib call

[obsidian rover]

Oh ok, thanks 🙏

[obsidian rover]

You're right, so dumb from me 🤣. I was looking for something extra complicated, and it's just there: https://pkg.go.dev/os/exec#LookPath

[tepid nova]

Tentatively moved this to preview: https://github.com/dagger/dagger/issues/3218

Crazy idea of the day: since dagger CLI is itself a client embedding the engine, what it the CLI commands themselves were run by the engine?

So dagger would do the bare minimum on the host to bootstrap the engine, then everything else would be client calls

[obsidian rover]

@still garnet, I have lately rebased my bundling buildkit PR. Which was working on a previous version of the repo (Friday's or thursday's version).
It seems that I now fail on core/integration/container_test.go:157.
Does input:6: container.from.exec.stdout.contents failed to load cache key: no match for platform in manifest: not found ring a bell to you ?

[tawny flicker]

Let's dig deeper and make it run its own emulated OS :p
But seriously, what would be the benefit?

[tawny flicker]

I think I've seen something about this in this channel a few days ago. Can't remember, though.

@obsidian rover : #1024046530262483006 message . No solution though.
Do you remember @wet mason ?

[tawny flicker]

I tried this test personally, and it works. But I'm full Linux. Is it some Mac OS->Linux platform issue?

concerning the test duration, could it be that multiple test are pulling the same image in parallel and lose the same time at the same moment?
Should we do some kind of prefetch in some init() to then allow each test to run from the same base? (if not better in term of execution time, at least reduce bandwidth?)

[wet mason]

nope, buildkit will dedup and only do it once

[tawny flicker]

Wasn't sure about that. Cool!

[wet mason]

yep!

[tawny flicker]

I guess it dedupes based on image ref at first?

[wet mason]

@civic yacht any thoughts on what happens when an extension returns a "base" type (e.g. Directory)?

[wet mason]

yeah, it will resolve the image first down to the digest in a blocking way for each one, then dedup, then actually get the image

[wet mason]

Namely: what should alpine build return in a code first world?

[civic yacht]

It just current still returns Filesystem, which I think is fine. It would now be replaced with Container in the new API. I have a test case somewhere (I think in multiplatform PR) where I made an extension that returns Exec, it worked fine too.

I guess I don't think that code-first changes many of the considerations there

[tepid nova]

Definitely Container

[wet mason]

nah, just the combo with codegen

Container is not a scalar, so it just represents the query for the container

[civic yacht]

Yeah I'm imagining that codegen includes models for all the types in the API you depend on, which would always include core. So then you'd just have to run your client codegen once and you can return the type you want (e.g. Container). We can also consider just providing core model types like Container as part of the SDK if we don't want to force an initial codegen run.

[wet mason]

func (r *alpine) build(ctx context.Context, pkgs []string) (*graph.Container, error) {
    api := graph.New()

    // start with Alpine base
    image := api.Container("").From("alpine:3.15")

    // install each of the requested packages
    for _, pkg := range pkgs {
        image = image.Exec([]string{"apk", "add", "-U", "--no-cache", pkg}, graph.ExecOpts{})
    }
    return image, nil
}

so the thing is, nothing was actually executed here. and Container doesn't have an ID field, it has an ID() function

so basically marshals back to empty struct, doesn't run anything

I guess the problem is mostly with built-in types returned by extensions?

not sure how to go about that. non trivial

first thing that comes to mind is that somehow, they implement some kind of interface, server-side we catch that at the return and it'll magically eval to {ID: ...}?

[civic yacht]

Yeah I guess the difference from today is that we currently return a Filesystem where the ID has been filled in, so yeah we would need someway of identify this situation and filling in ID automatically

Or ideally something less magic

[wet mason]

yeah

[tawny flicker]

unrelated, I just execute some "simple" integration test, yet I have some yarn running. Why is that?

#2 copy / /
#2 CACHED

#3 resolve image config for docker.io/library/golang:1.19.1-alpine
#3 DONE 0.5s

#4 docker-image://docker.io/library/golang:1.19.1-alpine
#4 resolve docker.io/library/golang:1.19.1-alpine
#4 resolve docker.io/library/golang:1.19.1-alpine 0.2s done
#4 DONE 0.2s

#5 apk add --no-cache file git openssh-client
#5 CACHED

#6 go build -o /entrypoint -ldflags -s -d -w /src/examples/alpine
#6 CACHED

#7 resolve image config for docker.io/library/node:16-alpine
#7 DONE 0.2s

#8 mkfile /known_hosts
#8 DONE 0.0s

#9 docker-image://docker.io/library/node:16-alpine
#9 resolve docker.io/library/node:16-alpine
#9 resolve docker.io/library/node:16-alpine 0.3s done
#9 DONE 0.3s

#10 apk add --no-cache file git openssh-client
#10 CACHED

#11 copy / /src
#11 CACHED

#12 merge (apk add --no-cache file git openssh-client, copy / /src)
#12 CACHED

#13 sh -c cd /src/examples/yarn && yarn install
#13 CACHED

#14 sh -c cd /src/examples/yarn && yarn build
#14 CACHED

#15 mkfile /entrypoint
#15 CACHED

#3 resolve image config for docker.io/library/golang:1.19.1-alpine
#3 DONE 0.6s

#16 go build -o /entrypoint -ldflags -s -d -w /src/examples/netlify/go
#16 CACHED

[civic yacht]

Integration tests end up using the cloak.yaml in the root of the repo by default which includes the builtin extensions

[wet mason]

because it's recursively looking for a cloak.yaml and it ends up picking the one at the root of the repo

[tawny flicker]

Right, got it.

[civic yacht]

first thing that comes to mind is that

[tawny flicker]

So if we want to speed up the integration test, we could create a cloak.yaml right in the /core/integration, right?

[wet mason]

Yeah. It's the problem we were discussing before.

Either we send the query only at the scalar end, in which case scalars are functions and nothing happens until then

Or we do it at the the object end. But 1) not sure how to go about that? which fields to select and 2) lazyness is gone

but at the scalar level, it means generated models have NO static fields, only functions

[civic yacht]

started a thread here: #1026675009705484358 message

[civic yacht]

That will have other side effects (like requiring a separate go.mod for the integration tests too), and I'm not sure if it would speed the tests up much since it looks like those extensions were mostly cached (I could easily be wrong though, just slightly educated guess)

[tawny flicker]

Ok.
I got 25% improvement just on the integration part on the first run. I'll open a PR and see how it improves on GH CI, just to check.

[tepid nova]

Request for feedback on README improvements: https://github.com/dagger/dagger/pull/3220

• New intro
• Simpler diagrams

[tepid nova]

Update: just pushed more

[tawny flicker]

Thinking about this again, I could feel the link between cloak.yaml and go.mod. They seem unrelated, especially as cloak.yaml is used by other languages than just Go. What am I missing?

[hot smelt]

Hello, just got a working (normally 😅 ) extension for kapp (base on the original cue action from dagger) https://github.com/laupse/dagger-kapp. If you want to check it out
It was pretty fun to write. One of the pain point would be that you have to make sure you response struct match the query every time
I Also had an issue but its more link to my way of testing with kind. the issue is in a fil in help

[hasty basin]

Awesome @hot smelt ! Will take a look at to see if still any blockers for you. By the way, we're working on the new Universe catalog for extensions which will include a showcase of community-built extensions. I'll ping you when we're ready. Would love it if you could list your extension there 😇

For everyone Let us know which extensions you need, want, are planning to build here: https://github.com/dagger/dagger/discussions/3216 🙏

[cosmic cove]

@dense dust are you ok with doing a playground demo at the community call this week?

extension feedback

[dense dust]

Not available on Thursday, but looking forward to doing one next week

[civic yacht]

Thinking about this again I could feel

[ancient kettle]

Is there anywhere that we're tracking the current state of the implementation of the latest GraphQL API?

[tepid nova]

@wet mason @civic yacht where can I find the latest version of your go query builder & go xdx work respectively? Would love to play with it, don’t mind if it’s broken

[civic yacht]

in PR here: https://github.com/dagger/dagger/pull/3191

Only works with Go, other extensions won't even load at the moment. You can see the alpine and netlify/go extensions in that branch for examples

[tepid nova]

oh I see it rips out extension loading in other languages you mean?

[civic yacht]

Well I updated the cloak engine to now ask extensions for their schema (rather than just read the schema.graphql file). The problem is I have only implemented that behavior for Go so far, so other languages just fail. That's just a temporary state; today my goal is to clean up stuff like that so it can be mergeable

It's pretty fun to play with already. Being able to write some code and then invoke its new API right away enhances the "addictive feedback loop". Having the nice client stubs will take it to another level I think.

I had an idea over the weekend of how we could implement "hot-reloading" where you change an extension's code and then client stubs get automatically regenerated for you

[tepid nova]

nice 😁

do you know where @wet mason ‘s query builder code is?

[civic yacht]

https://github.com/dagger/dagger/pull/3125

The ability to generate graphql objects from third party structs is also pretty intriguing in its possibilities too: https://github.com/dagger/dagger/pull/3191#issuecomment-1266205960

Can't wait to see that with the client codegen

[tepid nova]

yeah that's pretty crazy

[wet mason]

not up to date, i'm actively working on that

@civic yacht small one: https://github.com/dagger/dagger/pull/3253

[cosmic cove]

@here reminder to add your topics and demos to the community call agenda -#general message

[still garnet]

https://github.com/orgs/dagger/projects/2/views/67 is the project board

[ancient kettle]

🙏🏻

[tepid nova]

Merged new docs intro, with simpler words and diagrams: https://github.com/dagger/dagger/blob/cloak/docs/unxpq-introduction.md

[cosmic cove]

@hybrid widget good one to review tomorrow

[cosmic cove]

in this intro, we have a section that says "language support", but we always reference them as SDKs internally and probably during user calls. Should we add that term to that section or is overkill?

[tepid nova]

I avoided the term because of "shell scripts". When I use "sdk" and "bash" in the same sentence, it always starts a debate over what a SDK is or isn't, whether there should be such a thing as a bash sdk or not.

I just wanted to avoid that debate so avoided using the word SDK in this case.

(good catch)

@wet mason @civic yacht Now that we have more prototyping under our belt on query builder and xdx... Is there a path to making the graphql server code so simple to write, that it's essentially merged with the client code, and we can drop the distinction altogether? Then we could say "1) you can write a pipeline in any language 2) every pipeline has an API and can be shared and reused". It would make for great tech marketing 🙂 But not use if we can't make it true in the DX

🧵

[wild zephyr]

👋 IIRC there's still some stderr improvements pending, correct? I was trying to find the issue but I wasn't able to. I'm running this:

{
  core {
    image(ref: "node:slim") {
      writeFile(contents: "require('express')", path: "/home/node/script.js") {
        exec(input: {args: ["node", "/home/node/script.js"]}) {
          stderr
          stdout
          exitCode
        }
      }
    }
  }
}

and I'm missing some stderr outputs in the errors.message response that I'm currently seeing in the cloak engine stdout

[cosmic cove]

@wet mason you still need an issue created to track the work needed for updating the typescript/javascript SDK to use the new API, right? Can you send me the one for python that you are working on if it is a single issue? I can't find one that looks relevant.

[civic yacht]

👋 IIRC there s still some stderr

[hasty basin]

If we're using cacheMounts in an extension, do we need to do something like this to avoid collisions down the line: https://github.com/actions/setup-go/blob/main/src/cache-restore.ts#L16-L32 ?

[tepid nova]

Yes @still garnet flagged this issue earlier this week, which led to a partial redesign of that part of the API, to deal with the issues of cache ID conflicts (look for a thread called "cache mount ID" in this channel)

[stray heron]

⚠️ <@&785926252749651978> cloak will become dagger after this PR is merged: https://github.com/dagger/dagger/pull/3261

This is a significant and potentially disruptive change which can be minimised by reading the PR. If there are any comments or concerns, please leave your comment here: https://github.com/dagger/dagger/pull/3261

FWIW, this GitHub comment explains why we are doing this change (tl;dr getting ready to ship dagger v0.3.x pre-releases): https://github.com/dagger/dagger/issues/3176#issuecomment-1267359683

[still garnet]

fyi: GitHub actions seems to be down: https://www.githubstatus.com/

none of our PRs are running any checks 😭

[tidal spire]

I'm seeing a weird thing maybe someone can help me understand. It could be weirdness from doing cross-engine things in bash, I'm not sure. https://github.com/kpenfound/dagger-demos/blob/main/make/cpp/dagger.sh#L72-L78
In this query, it takes in the exec.mount.id from the previous step, but rather than pulling the fsid it actually runs the previous step again

[wet mason]

@tepid nova @civic yacht With everything that's happening, I'm less convinced about dagger.gql. 🧵

• With codegen and xdx, users can use dagger without a single line of GraphQL (neither query nor schema).

dagger.gql brings us in the opposite direction where users have to write GraphQL anyway for the manifest

• GraphQL is hard(er) to programmatically generate. Burden for tooling (e.g. Adding a text editor in playground for dagger.gql -> easy. Adding a UI on top -> hard)

e.g. the "Add to Project" that was just mentioned.

• Unclear to me how to add extra stuff in there, if need be. Hypothetical example: if we wanted to add something about resources (e.g. cpu/memory limit, etc) where would it go? Not immediately obvious how to design that query

• [old concern, still there] About static analysis. Hard to do by imposing limitations (what the query can do, e.g. no containers etc), impossible without limitations

• Removes possibility (or at least, makes it harder) to parameterize things "cli side". By being queries it assumes everything will be handled by the graphql server, the CLI is just a dumb passthrough. If we wanted the cli to understand the manifest, it would require to parse and understand gql. Example: for instance this removes the possibility for codegen being done by the CLI, since it won't be able to understand the "sdk" part of the manifest

[rancid turret]

Can someone help me understand the parent field in /inputs/dagger.json? Example of when it's used.

[civic yacht]

Graphql parent

[tepid nova]

Are there limitations to dagger codegen in vanilla Javascript as opposed to Typescript?

[civic yacht]

I would imagine we implement the codegen in Typescript, which should make it available for use in js too, so no limitations in that sense (unless you meant something else)

[stray heron]

@wet mason please review and merge if it all look good to you. I am going to start on the next PR which builds on top of this one: https://github.com/dagger/dagger/pull/3261

[tawny flicker]

I don't know how many times I will be bitten by the facts that git push doesn't push tags by default…

[wild zephyr]

push.followTags ftw! 🎉

[rancid turret]

@civic yacht correct me if I'm wrong. We no longer extend types from other extensions so the parent value should only come from the same extension?

[wet mason]

I didn't mention but:

• codegen is architectured so that it can support multiple languages
• right now, the goal is to get Go just right, before going to other languages
• Positional/optional arguments are actually not a problem in TypeScript/Javascript/Python since that's a language feature (/cc @tepid nova) -- this is a unique problem for the Go SDK

[stray heron]

Great demos @wet mason & @still garnet ! I'm super excited to try the features out for https://github.com/thechangelog/changelog.com/blob/master/2021/dagger/prod_image/main.cue (still running dagger v0.1.0) . That would make for an awesome Christmas present 😉

[still garnet]

new target date: december 24 👍

[wet mason]

@still garnet @tepid nova @civic yacht

PR opened, with a summary of the optional arguments trade-offs and possibilities: https://github.com/dagger/dagger/pull/3125

need to run now, be back later

[tepid nova]

FYI I updated the "improve shell scripting DX" discussion with the different options that were discussed: https://github.com/dagger/dagger/discussions/3276

[wet mason]

@still garnet @civic yacht @tepid nova

Food for buildkit thoughts: isn’t the root of all our problems the fact that we don’t have filesync support in the SDK?

Like: running cloak in docker (and therefore bundling cloak and buildkit as one, and therefore distribution, weird server-in-server architecture, etc)

Hosted engine: sessions, virtual host etc

Many other things

[tepid nova]

I have a few highly hypothetical questions...

Q. If hypothetically we wanted to ship a stable 0.2.x version of dagger built on cloak. How difficult / painful would that be, and what would the main pain points be?

(I don't actually want to do this, but I find it interesting as a lens to look at other problems)

Q. if we had no extensions, would we still need a cloak.yaml or now dagger.json?

[civic yacht]

don't know what exactly is meant by no extensions, you just mean you only want to use core api? If you are writing a script then yeah I don't think there's any need for dagger.json in that case. If you are writing an extension then dagger.json will still currently configure the sdk. I suspect we could approach that in other ways (autodiscovery for example) in which case dagger.json is not needed either

[wet mason]

@tepid nova I was about to reply to your comment, but I think there's a typo?

Example: dir.WithNewFile("/foo", directory.WithNewFileContents("blah"))

Is this actually needed in practice? ie. is there any instance of WithNewFile meaning 2 different things in 2 different places?

Did you mean WithContents?

Or something else?

[tepid nova]

I think I meant WithNewFile? But maybe just confused by the super long names 😉

[still garnet]

raising a tiny red flag: directory.withNewFile.file.secret.id will give you a SecretID that totally encodes the secret in plaintext 😅 not sure how we avoid that besides docs, really.

❯ wl-paste | base64 -d | jq -r .file | base64 -d | jq -r .llb.def[0] | base64 -d
"U*;
/secret"sometimes i dont write tests first(RZ

raising a tiny red flag `directory

[tepid nova]

Good morning cloakeratis

[still garnet]

@civic yacht fyi: working on the Host API + local dirs has led me to the Project code/schema, which has led me to moving all the schema implementations into core/schema/ to resolve a circular dependency on *core.Directory. I think we'll be close to retiring *filesystem.Filesystem and the old API after all the dust settles

[tidal spire]

New demo! https://www.youtube.com/watch?v=qMA7m4F5xRM

[hasty basin]

@tight socket ☝️

[tawny flicker]

I can't figure out why this errors out: https://github.com/dagger/dagger/actions/runs/3205990266/jobs/5241753814
Anybody faced a similar problem?
(it's dagger 0.2, just bumping deps for security purpose)

[stray heron]

WDYT @civic yacht https://github.com/dagger/dagger/pull/3285#issuecomment-1272111932 ?

[civic yacht]

(responded on the issue)

[civic yacht]

Support for Code-first schemas in Go has been merged (no more schema.graphql when writing Go extensions!), some more details and doc links in this discussion: https://github.com/dagger/dagger/discussions/3291

[mellow bolt]

@tidal spire would it make sens to add your video to our Dagger channel ? We've already have Jeremy demo with Jenkins.
https://www.youtube.com/channel/UCVM5UxpF_nzpOsXHShmaqmw

[obsidian rover]

With the new runners in the CI, the environment is still cleaned after each run ? (just to be sure)

[ancient kettle]

@still garnet @civic yacht Any chance I could get a way to convert a directory to a Container? So I can exec in it? I’m needing something like this for dagger-classic.

[still garnet]

it's planned: https://github.com/dagger/dagger/pull/3151 (container { withFS })

[ancient kettle]

🙏🏻 that’s precisely what I need.

[rancid turret]

Hey @wet mason I'm interested in syncing a bit with you when you can on https://github.com/dagger/dagger/pull/3125, particularly on the "Architectured for multi-language" part. I can see you're simply using templates with schema introspection. Python already has schema introspection and a query builder, but curious to contrast with what you have to gauge what would be easier. You say "adding a language is still a non-trivial amount of work", curious to know what that amounts to.

[tepid nova]

@still garnet @civic yacht @ancient kettle starting a thread to discuss core.#Source in Europa, and how to best port that to Cloak

[wet mason]

Yes! I'm free in an ~hour if that works for you? Or we can do async

[rancid turret]

Yes, I'll wait 🙂

[civic yacht]

I did a braindump on some of the cloak design questions that come up frequently, PR here: https://github.com/dagger/dagger/pull/3305

Fairly certain it needs at least one more pass to be coherent+succinct, please let me know what you think of the current state though @tepid nova @wet mason

[wet mason]

(running late!)

[wet mason]

@rancid turret done! grabbing a quick coffee then i'm ready, ~5 min

@rancid turret in dev audio 🙂

[stray heron]

@twin crow @rancid turret when you have a moment for a quick Python SDK question: https://github.com/dagger/dagger/pull/3310#issuecomment-1275137204

[rancid turret]

Replied. Looks like a subshell issue.

[stray heron]

Ok. The first time I tried manually, without the subshell & it failed for a different reason. Will try again & follow-up from there. Thank you!

[stray heron]

All good, thank you! Please review & merge if it looks good to you: https://github.com/dagger/dagger/pull/3310#issuecomment-1275246627

[still garnet]

@tidal spire @hasty basin signal boosting this question around secrets, since more PRs are becoming dependent on this one 🙂 https://github.com/dagger/dagger/pull/3272#issuecomment-1275021977

[tidal spire]

Thanks for the heads up! No issue from my end. Will have to update a demo or two but that's cool

[still garnet]

👍 - with the new secrets/host API I think the replacement will be to set secrets in env instead and then use host { variable(name: "FOO") { secret { id } } }: https://github.com/dagger/dagger/pull/3304

merged the PR and added that as a comment for posterity

[tepid nova]

@still garnet how are you feeling generally about the secrets API design in its current form? Does it feel right?

[hasty basin]

@still garnet got it. I was using --secret (and --set) in a bash context where I didn't have an engine holding my secrets across multiple calls to the cloak cli.

https://github.com/jpadams/dagger-bash-netlify/blob/main/deploy.sh
or
https://github.com/jpadams/dagger-bash-netlify/blob/main/new.sh

[tepid nova]

There's a lot of inter-connected discussions, so personally I'm not confident I can see the forest for the trees right now (insert correct analogy here)

For example if someone complains to us that we're adding artificial roadblocks to just creating a secret with inline content, do we have a good answer as to why it's not arbitrary?

(Not arguing against it at all, I remember thinking it was the right thing to do, I just don't remember why I thought that - ETOOMANYTHREADS 🙂

[still garnet]

i think there's a ton of nuance to secrets that makes it hard to nail on the first try, but taking secrets from env feels more sound than argv. if used carefully env is safer since it doesn't leak secrets into the process tree. to go beyond that, may I introduce you to my best-named Go package, https://github.com/vito/twentythousandtonnesofcrudeoil

[tepid nova]

what the... 😄

[still garnet]

the tl;dr is to just remove values from the process's environment after they're used, so they don't leak to any child processes 😛

[tepid nova]

nice

I'm afraid to ask about the name

[still garnet]

i'd say the video in the readme explains it, but it really doesn't

[ancient kettle]

I have a version of this working locally, but it's just blowing away parent. Do we want to implement this using llb.Merge?

[tepid nova]

OK I watched the video

I am still confused but glad I watched it 😄

[still garnet]

what I had in mind was to replace the parent's filesystem, but keep everything else (config, mounts, etc)

using Merge would be something else entirely

[ancient kettle]

Ah, gotcha

That's straightforward.

[still garnet]

@tepid nova I think it'l be a while before we can really say we nailed secrets, this was always the thorniest problem space in Concourse and then Bass (there's a reason I left it as the last major milestone and still have very primitive support atm)

there's whether we want to integrate with secret providers, and then there's how do you bootstrap auth to those providers, etc.

so I think we're making the right first step which is to provide the simplest possible way to pass sensitive values in, and having secrets themselves be a recipe for how to fetch them, since that should hopefully keep all the API touch points somewhat stable

[tepid nova]

makes sense!

[stray heron]

@rancid turret looks ready to me: https://github.com/dagger/dagger/pull/3310#issuecomment-1275991153

[hasty basin]

so how do I need to change my code here? https://github.com/jpadams/dagger-bash-netlify/blob/main/deploy.sh#L19

[hasty basin]

Since --secret is removed and with bash approach I make separate calls to dagger (fmrly cloak) engine, then I won't have a way to load a secret and then use a secret across two calls to the engine, right? So I should convert this script to golang? @civic yacht (I ask because it was part of my Jenkins demo that I still need to deliver).

[ancient kettle]

@wet mason What's the current status of the Query Builder API/Go SDK? I'm working on porting dagger-classic from the core APIs to the new API... and I'll probably need to build a bad version of the query builder to create the correct environments, mounts, etc...

I'd rather use your nicer one. 🙂

[tepid nova]

Moved up to top priority and getting fast tracked. Andrea can give more details but short version is that it’s a safe bet to use it

in fact you using it will probably help accelerate it even more

[ancient kettle]

Great! It's gonna make my life + dagger-classic port easier, smoother, and more type-correct. 🙂

Ok! It looks like I can probably use this branch as a decent starting point: https://github.com/dagger/dagger/compare/cloak...aluzzardi:dagger:query-builder-test

[tepid nova]

With the initial launch focus on Go SDK we need to talk about the initial docs contents. Starting with a Go API reference: this becomes more important, and (slightly) more urgent than a graphql API reference. Perhaps it also informs a slight change of structure, to give each SDK its own documentation space? Thoughts? 🧵

[wet mason]

Getting merged today.! Then will rapidly iterate, but I'd say go for it, the feedback will be very useful

[ancient kettle]

Sweet! That'll be awesome. I'm currently working on Container { withFS } and Container { fs } from https://github.com/dagger/dagger/pull/3151, so I can unblock core.#Exec, so the timing should be perfect. 🙂

[wet mason]

Feel free to start using that branch right away. Still addressing some review comments though so will be a bumpy ride

[ancient kettle]

👍🏻 Will do!

[ancient kettle]

@still garnet @civic yacht If you're able, a quick look at this would be awesome! If everything looks decent, I'll add some tests for it. https://github.com/dagger/dagger/pull/3333

[wet mason]

Go SDK update: https://github.com/dagger/dagger/pull/3125#issuecomment-1276598791

[tidal spire]

updated api in basic bash example: https://github.com/kpenfound/dagger-demos/commit/5eeb368d5f09865aa515307417b1530bc9a7849e
will do the one with secrets next

[cosmic cove]

@here reminder to add your demos and/or topics to the community call for tomorrow -https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[tepid nova]

FYI @civic yacht @still garnet I opened an issue for the "project scoping" discussion that emerged from cache ID discussion. https://github.com/dagger/dagger/issues/3345

Since the thread started in a merged PR

[tepid nova]

Placeholder for 0.3 CLI design: https://github.com/dagger/dagger/issues/3347

https://tenor.com/view/the-smurfs-smurf-painting-paint-bucket-painters-brush-ship-deck-gif-21376591

Time to paint that shed!

[still garnet]

@tepid nova hope you brought a lot of paint: https://github.com/dagger/dagger/pull/3348 adds a new API, and raises a side question of whether we should have HostDirectory { write(DirectoryID!) } or Directory { write(HostDirectoryID!) } since it adds another thing that wants to write to the host

[tepid nova]

https://tenor.com/view/paint-brush-wall-painting-color-change-gif-5937896

That one is too painfully accurate actually

[wet mason]

@civic yacht @still garnet @tepid nova Highly, highly WIP: https://github.com/dagger/dagger/pull/3349

Tons of open questions

Also, I might need some help -- engine.Start leaked everywhere, there's a ton of stuff that needs updating

[still garnet]

oh, dang, I figured it wasn't the long term plan but I had just started to use engine.Start in Bass since there's no other way to show progress output (which is 99% of the bass UX). didn't know it'd be gone so soon 😛

ah i guess it's just commented out for now, maybe I can keep using it, that code's gotta go somewhere right

will take a deeper look tomorro!

[wet mason]

Oh, it’s not going to go away, but it’s supposed to be internal

Whereas dagger.Connect through the SDK is here to stay (with the caveat of providing the same interface whether the engine is embedded or not)

[still garnet]

makes sense, I'm just taking a shortcut until we figure out how to stream progress over the API (or something)

[wet mason]

Yep exactly. dagger.Connect(ctx, dagger.WithSomethingSomethingLog(…)) would be the public API for that

[still garnet]

re bass runtime: updated to the new SDK + latest changes, down from 16 to 10 failing tests 🎉 https://github.com/vito/bass/blob/dagger/pkg/runtimes/dagger.go - still finding some rough edges, using this as an API feedback loop. latest rough edge: it'd be nice if I could write a single file to a HostPath, instead of a directory. (but really the host writing is a workaround for lack of a tar stream export API)

[wet mason]

Neat! Yeah, writing a single file would be really useful

[tepid nova]

Amazing that you’re already so close

I guess bass itself is written in Go?

[still garnet]

yeah, and dagger is basically an API for all the buildkit stuff I previously did directly (see buildkit.go)

[tepid nova]

Crazi idea: could Directory and (whatever type represents host file system) have a common Graphql interface?

to avoid unnecessary drift

[still garnet]

oh like Directory and HostDirectory? could be neat. so like HostDirectory { withDirectory } instead of write?

[tepid nova]

right something like that

just a little extra structure to help us stay DRY

[tidal spire]

api question: For Container.withMountedFile(path: String!, source: FileID!), is path a directory where the file will be mounted, or the full path to the file? e.g. /bin/ vs /bin/myfile

[tidal spire]

actually more important question, does Container.Publish include mounts, or does it publish rootfs?

[tidal spire]

It appears that the answers are 1) full path to file 2) does not include mounts. Correct me if I'm wrong 🙂

[hasty basin]

@civic yacht do you have your multi-arch/platform code up anywhere? Using goxx, etc. I remember you showed it off in the Sep 29 community call.

[still garnet]

1. is relative to the container's workdir, which might be / by default, and 2) is correct

[civic yacht]

It's in PR 😞, had to prioritize the stuff for release, now it needs tons of rebasing on the new API https://github.com/dagger/dagger/pull/3119

Been starting everyday hoping to have time to get back to rebasing it and ending everyday with multiple TODOs left on my list including that. On PTO next week but will prioritize getting it rebased ASAP when back. Or maybe by some miracle I'll have time before then

Let me know if I can help with anything in particular related to all that though

[stray heron]

We are super close to producing the first dagger binary alpha release, v0.3.0-alpha.1 https://github.com/dagger/dagger/pull/3356 cc @wet mason @twin crow @tepid nova @cosmic cove

[hasty basin]

@round granite here's what was demoed earlier today to give you a sneak peak of the new Go SDK: https://github.com/kpenfound/dagger-demos/blob/main/go/hello-container/magefiles/main.go

[round granite]

that looks awesome!!

[ancient kettle]

@still garnet @civic yacht I think I'm pretty much done with https://github.com/dagger/dagger/pull/3333.

If there are other ways you think I should test withFS I'd appreciate it. I'm coming up empty thinking of other ideas there.

[still garnet]

awesome

[wet mason]

Wrote down what I was explaining during the community call regarding GH actions:

https://github.com/dagger/dagger/discussions/3358#discussioncomment-3873205

[tepid nova]

Thanks! Bummed that I had to drop before the end

[warm tundra]

Here's the thread I promised to start during the community call! https://github.com/dagger/dagger/discussions/3359

[tepid nova]

From initial feedback on the 0.3 CLI design, I'm starting to think we should perhaps not bundle absolutely everything into the same CLI. Maybe with SDK-centric DX, we don't need to... maybe the CLI should only be the client, and we should only release the engine as an oci image? 🧵

[tidal spire]

@civic yacht I updated to the latest commit in my sdk (yay rm cloak.yaml!), and had to rollback fsutil once again after that. I can open an issue if there isn't one

[civic yacht]

Wait what's the issue with fsutil. I have an incredibly vague memory of seeing something about that but forget now

I may have read it immediately after waking up lol

[tidal spire]

Haha yeah I posted about it early this morning. I'll make an issue

[tidal spire]

https://github.com/dagger/dagger/issues/3367

[ancient kettle]

Damn. I just ported core.#Pull and replaced ~25 lines of GraphQL code with 4 lines of Query Builder code.

[tawny flicker]

That's some seriously snarky package. I love that.

[tawny flicker]

https://tenor.com/view/blinking-eyes-white-guy-blinking-meme-what-huh-gif-26334322

I've checked it.
I'm still very confused.
If it is satire, it is quite good.
I hope dagger's roof won't fall off.

[still garnet]

lol, it's indeed satire, Australian comedians iirc

[hasty basin]

Classic Clarke and Dawe

Gradle example destined for Jenkins: https://github.com/jpadams/dagger-jenkins-gradle

Using bash, cache dir

[cosmic cove]

Release date

@civic yacht just want to make sure that we get your thoughts on this before your PTO next week -https://github.com/dagger/dagger/issues/3271

[rancid turret]

@twin crow (et al), Python's convention is to have method names as lower case but creates a conflict with reserved keywords:

alpine = core.container().from_(ref="alpine")

We can just do it like Go, avoiding the issue:

alpine = core.Container().From(ref="alpine")

But it won't solve if there's an argument like that:

# hypothetical
alpine = core.container(from_="alpine")

We can just document that reserved keywords have to be suffixed with an _. Also the editor will help autocomplete. In that case do we do it for both or just arguments?

WDYT?

/cc @mellow bolt

[wet mason]

<@&946480760016207902> -- @stray heron and I are about to merge cloak into main (within the next 30 mins) -- if you have any concerns let us know

[civic yacht]

samalba0001 et al Python s convention is

[wild zephyr]

@wet mason there are still some PR's opened against the cloak branch. Are we good to merge to main regardless of this?

[wet mason]

Yeah. It's just a matter of clicking edit on the PR and switching the base branch

and if it needs rebase, rebase on top of main rather than cloak and we should be good

[restive igloo]

Sorry if this is the wrong place to ask this, but what exactly is cloak? I've tried looking around but haven't really found any info.

[cosmic cove]

Hi! This post answers the general question, but let us know if you have any other questions about it. #general message

[restive igloo]

Oh neat. Don't know how I missed that post. Thanks!

[cosmic cove]

No worries! Discord search isn't great. We look forward to having an alpha release soon, but you can start to look around in the Cloak branch for now -https://github.com/dagger/dagger/tree/cloak#readme

[wet mason]

It's live!

cloak branch -> now main
old main -> v0.2.x branch

<@&946480760016207902> All open PRs base branch has been updated to main. If you need to rebase your PR: git pull main, then git rebase main the PR (instead of git rebase cloak)

I suggest deleting the old branch git branch -d cloak to avoid using it by mistake

[hasty basin]

Welcome @random peak! 👋

[tepid nova]

Exciting!

[wet mason]

I have a failure on TestHostLocalDirWrite:

host.directory.write path "/private/var/folders/bp/4j_0m6ss20ndzn7h9l4695dw0000gn/T/TestHostLocalDirWrite1987826162/001" is outside workdir

I think it might be only on darwin, CI looks fine

[civic yacht]

IIRC that error message happens when there's a symlink in your path you're trying to export to

[wild zephyr]

👋 getting this error when trying to add a file to a scratch container to push

query Build($file: FileID!, $ref: ContainerAddress!) {
  container {
    withMountedFile(path: "/mnt/dagger", source: $file) {
      exec(args: ["cp", "/mnt/dagger", "/dagger"]) {
        withEntrypoint(args: ["dagger"]) {
          publish(address: $ref)
        }
      }
    }
  }
}```

{
"data": null,
"errors": [
{
"message": "build shim: failed to parse target platform unknown: "unknown": unknown operating system or architecture: invalid argument",
"locations": [
{
"line": 4,
"column": 7
}
],
"path": [
"container",
"withMountedFile",
"exec"
]
}
]
}

[civic yacht]

Guessing that we need to default the platform when pushing llb states that have never been associated with a platform (e.g. scratch)

I just tried to reproduce in a simpler way in a test case with

container {
  publish(address: $ref)
}

and triggered a nil pointer exception in buildkit itself lol

So yeah, a couple bugs there. I'll make an issue (in a meeting in a few minutes tho, will after)

[wise field]

@wet mason now that cloak-dev is in main, whats the plan for v0.2 and related cue-based universe support? I'm just wondering if my efforts on certain packages like poetry for example will be worth it

[wet mason]

v0.2 is still the stable version, main contains a pre-release alpha. @ancient kettle is working on porting CUE into main as an additional language (e.g. Go, TypeScript, Python, CUE). That will be in before main becomes the new "stable" version

[ancient kettle]

Literally working on the compatibility table for dagger-classic as we type: https://github.com/dagger/dagger/issues/3121

We haven't dug into what a full CUE SDK would look like, but I'll spend some cycles on a design for that after I've got the dagger-classic port in a good spot.

[cosmic cove]

@tepid nova or @wet mason to review before Erik is OOO -https://github.com/dagger/dagger/pull/3342#issuecomment-1279500173

[wet mason]

@civic yacht @still garnet @tepid nova Converted alpine to the new SDK (trying to get rid of the usages of the old API so we can rip it out before release): https://github.com/dagger/dagger/pull/3392

To make that happen, I quickly scrubbed together the xdx+codegen thing we were discussing a bit ago (e.g. return a Container directly from alpine) -- still needs work but it is what it is

End result is this: https://github.com/dagger/dagger/blob/bd9903bee3b45125580f871b8a77baa103f5d178/examples/alpine/main.go

[civic yacht]

Awesome, taking a look in a little bit

[wet mason]

@civic yacht haven't implemented it yet in serve(), but now scalars also implement an interface so we can remove the "checking if it comes from go.dagger.io" workaround

{
  alpine {
    build(pkgs: ["jq", "curl"]) {
      stdout {
        contents
      }
    }
  }
}

^^^ this now works (alpine returns a Container now, not Filesystem)

if we like how it goes, there's a few other examples to update, then I think we should be good to go to rip out old API

[tidal spire]

Did somebody say demo? https://www.youtube.com/watch?v=n-woWsu6AQU

[civic yacht]

Looked again and realized it's expected to fail right now because you won't be able to execute cp but it should not fail in the way it is.

I think it's an easy fix, I'll see if I have time this afternoon but if not issue is here: https://github.com/dagger/dagger/issues/3393

Also the separate nil pointer exception I triggered in buildkit that I mentioned was another case of one Alex hit the other day, have a fix out in buildkit for it: https://github.com/dagger/dagger/issues/3337

[civic yacht]

@wet mason have you been converting other stuff over to the new API? I can help out with that too, just don't want to duplicate work. There's also the rest of the extensions under examples. There's test cases too but they can just be deleted now. I'll go convert over yarn if you haven't started already

shipit on alpine btw, I saw there's a test failure but the test case just needs an update to select from container fields instead of filesystem fields 🙂

[hasty basin]

Dude!!! That demo made my week!!

[tidal spire]

It was surprisingly easy 😄 if you check the timestamps on the commits lol

[civic yacht]

Always a good sign the DX is heading in the right direction 🙂

[wet mason]

nope, not started

[hasty basin]

I love that golang style coming through.

https://tenor.com/view/goldy-gopher-row-the-boat-skiumah-fleck-game-day-gif-15692641

[wet mason]

(sorry in a meeting ... fixed! waiting for CI to pass)

[wet mason]

I think the last remaining bits are netlify ... but I don't think the API supports retrieivng a secret back

[wet mason]

Ok last one for the week -- I resurrected the old SDK PR I did a while ago, the only remaining bit was the stdio dialer which is now its own PR: https://github.com/dagger/dagger/pull/3265

[mellow bolt]

So cool ❤️ !!

[wild zephyr]

https://twitter.com/styfle/status/1581004161814233090?t=2_KVD4z5tJPD4bRjaW7afQ&s=19

Relevant for npm / yarn Dagger scripts

Steven ⬢ (@styfle)

If you have scripts in your package.json that call other scripts, make sure they are package agnostic by using "$npm_execpath"

This ensures the package manager you used to start the script is used to run the nested scripts.

This is particularly important for "postinstall"!

Likes

160

Twitter · Oct 14, 2022, 7:29 PM

[tepid nova]

I just accidentally pushed straight to main... 🤦 Thankfully it was just a README change.

Isn't that supposed to be prohibited by branch protection? I checked the repo settings real quick, main does seem to be protected... but perhaps not protected enough?

[icy sapphire]

u may be able to push to main if you are an admin on the repository with bypassing rights

@tepid nova

[tepid nova]

looks like it..

Maybe I should give up my admin rights 😅

[icy sapphire]

if u check the box that says dont allow settings to be bypassed then it should prevent this

[rancid turret]

Has anyone thought about how to manage deserialization of a custom scalar result in an SDK client?

[wet mason]

you mean serialization? or really deserialization? deserialization should just be a json decode no?

oh

go sdk assumes custom scalar is always a string (e.g .ID-like). Non-string scalars are not supported

[rancid turret]

Makes it easier 🙂

[wet mason]

@tepid nova @still garnet @tawny flicker Was about to comment on the logs stuff. Realized @tepid nova proposed pretty much the same over the weekend: https://github.com/dagger/dagger/pull/3339#issuecomment-1281388590

oh and @still garnet commented as wel

*well

[still garnet]

lol we all commented within a few minutes

[wet mason]

In summary, I propose we make the SDK look like it's dumping logs from the API, but this week it won't. When it will, it's transparent to SDK users. And we find a way to make bass dogfooding possible, without exposing it in the public API

[tepid nova]

I think the most important takeaways here is that I commented first.

But yeah everything else too

https://tenor.com/view/finish-line-gif-13301498

[tepid nova]

Is there an issue for the ugly go mod edit -replace needed to install Go SDK?

What feels more natural to you:

🅰️ Go SDK for Dagger
🅱️ Dagger SDK for Go
© Dagger Go SDK

[vital plaza]

Tough. It really is a matter of audience point of view… 😅

[hasty basin]

https://github.com/dagger/dagger/issues/3391

[wet mason]

@still garnet https://github.com/dagger/dagger/pull/3414 <-- does this make sense to you? we had some internal code being used left and right

[still garnet]

no objection, but don't have full context on everywhere that uses it, only really familiar with core/integration

[tepid nova]

@wet mason I took the liberty of editing the "mod edit / replace" issue to set context + assigning to you. https://github.com/dagger/dagger/issues/3391

[obsidian rover]

By removing the init, you make sure that dagger-buildkitd doesn't get respawned between the tests in each package, when doing go test ./... ?

[wet mason]

nope, didn't test that

[obsidian rover]

Because, if you don't have a dagger -buildkitd, it would fail here. It was aiming to setup the environment to run the tests, directly from the tests

[wet mason]

@tepid nova @still garnet (@stray heron @tawny flicker for when you're back): https://github.com/dagger/dagger/pull/3416

Pretty substantial change, opening up for discussions

[wet mason]

the problem is it didn't work for other packages than our own, because it's using internal/ code

so from the SDK, I can't run a single test

looking for an alternative solution, but requiring every test that uses dagger to import an internal package is not possible

[wet mason]

Why would it fail though?

[obsidian rover]

Because, when running go tests, do we always rely on the client ?
I believed those init were here to run this SetupBuildkitd function in case we don't rely on the client in the tests + no buildkitd already set. I might be wrong

[wet mason]

I've changed the code to get the go mod version if the build info is not found

[obsidian rover]

You're right. My bad. As vito said, no objection

[tawny flicker]

I tried to push go.mod and go.sum earlier in my magefile, but it doesn't seem to work as I expected: https://github.com/dagger/dagger/pull/3372/commits/d45ffd4452d0bbaa946311111337ab6e949dbf30
Did I use the SDK wrong?
My initial understanding is that I use the workdir, even if I extract the go.mod and go.sum, the base container would be changed as the workdir might be tainted with whatever modification happened in the workdir. How should I handle that case?

[wet mason]

@tawny flicker that should work as far as I know ...

[tawny flicker]

unfortunately, it doesn't. Just adding an empty line would make the action re download the modules

[wet mason]

btw -- in the v0.2.x branch, there's the go module that tries to be fast

I think it's mounting a persistent go mod cache etc

[tawny flicker]

I mean, try it on your side. I'd be happy to hear it's only a problem on my side

[wet mason]

adding an empty line where?

[tawny flicker]

on my PR, just modify the magefile to add an empty line somewhere, and rerun mage build:dagger, and it redownload all the modules

and even on any other Go code somewhere else.

[wet mason]

have you tried WithCopiedFile instead?

maybe it has something to do with the mount

since I guess we're mounting a relative file from the workdir, maybe the entire workdir is part of the cache tree now?

whereas if you copy those files, then it's the contents of the file that get hash cached

[wet mason]

https://github.com/dagger/dagger/blob/v0.2.x/pkg/universe.dagger.io/go/container.cue

uses some advance tricks like persistent cache mounts for mod and build, so that already built files gets retrieved from cache

[tawny flicker]

Yes, I guess this should do the trick. I didn't know this part of the API

[wet mason]

should be "translatable" into go sdk

[wet mason]

e.g.

    alpine.
        WithMountedCache("go-mod-cache", "/root/.cache/go-mod").
        WithEnvVariable("GOMODCACHE", "/root/.cache/go-mod").
        Exec(...)

[still garnet]

quick sanity check: are there any blockers for removing the old API + schema at this point? 🤔

i'm embarking on a pretty large refactor (tldr: late-bound gateway construction so we can have dynamic LocalDirs), wondering if I should just delete any old things it touches (in a separate PR)

[wet mason]

@still garnet yeah I'd say, go for it. IIRC we're missing docker build in the new API

but at this point no code should be targeting the old API anyway

[still garnet]

just ran into that, I guess it could be converted to return a *Container?

[wet mason]

I think there's some discussion on a PR somewhere regarding the interface of build?

in any case: I think it's good to remove the old API, even if some things are not there yet

[tawny flicker]

That worked! 👍

[still garnet]

ah ok - so I'll just remove dockerbuild too, since it's at least not very useful as long as it's the only thing returning a Filesystem, and it sounds like we're figuring out how the new thing should look

[wet mason]

@still garnet I believe it's here: https://github.com/dagger/dagger/pull/3151

@still garnet @tawny flicker ugh, yeah, I don't think replace directives work outside of the repo

go.dagger.io/dagger imports
    go.dagger.io/dagger/engineconn/embedded imports
    github.com/dagger/dagger/engine: github.com/dagger/dagger@v0.0.0-00010101000000-000000000000: invalid version: unknown revision 000000000000

it ignores the replace from dagger/dagger and just tries to install the vendored version

it could work IF we maintained the correct vendored version in go.mod, in addition to the replace directive

like, use replace for dev

then once we cut a dagger/dagger release, go get -u inside the sdk module

then it's importable everywhere without replace directives

Could be something like:

dagger/dagger/engine: engine module (basically git mv engine core etc etc ./engine)
dagger/dagger/sdk/go: Go SDK module

• replace github.com/dagger/dagger/engine => ../../engine
• AND keep the version of dagger/dagger/engine up to date, whenever we cut an engine release

At this point, the SDK has the "latest" changes while still being importable

However, go get'ing the SDK will always use the "previous" engine version (the one currently in go.mod)

so uhm, a bit finicky I guess

[still garnet]

hrm sounds doable, but having to commit something back to the repo on release can be pretty painful

fyi: merging https://github.com/dagger/dagger/pull/3398 since it removes some usage of the old API and is already double-approved

[wet mason]

Yeah, we'd have to adopt different versions/tags, e.g.

v0.3.0-alpha.1 --> nope

engine/v0.3.0-alpha.1 + sdk/go/v0.1.0 --> yep

[still garnet]

PR up to remove all the old stuff: https://github.com/dagger/dagger/pull/3418

[tepid nova]

https://github.com/dagger/dagger/pull/3151 @still garnet

[ancient kettle]

@wet mason @still garnet Is there any way to create new secrets beyond creating files with plaintext and retrieving their ID? (I don't really want to go down that path... keeping secrets in cache hurts my soul)

[still garnet]

you can create secrets from host env vars: host { variable(name: "FOO") { secret } } iirc. (btw @tawny flicker just remembered we need to rename that too!) I think I'm gonna need some kind of 'static secrets' interface pretty soon too though.

[tepid nova]

@still garnet btw you are totally correct that once we spin out the engine from CLI (aka “the grenade”) that opens several existential questions including: “was it the right call to forbid creating secrets by just passing plaintext in the graphql query? This made way more sense when we designed it for a local engine which now no longer exists “

[ancient kettle]

Yeah, I need that functionality for core.#DecodeSecret and #TrimSecret for dagger-classic. Basically any kind of secret manipulation will require that.

[tepid nova]

True

fyi would love ideas: https://discord.com/channels/707636530424053791/1031787295025668117

[wet mason]

@still garnet @tepid nova Ok so this is possible with a bunch of caveats. It DOES allow independent versioning of the engine and Go SDK in addition to independent import path

However — it’s not reversible. The tag name and the module relative path within the repository are mingled forever. And working with it is weird (see note about relative vendoring for outside importers — creates release order dependencies)

I do wish we had more time to decide as it’s a one way street

I’ll sum it up on an issue but basically:

• it’s either a monorepo, single go module, same versioning, different package paths

OR

• it’s a monorepo, different modules, different import paths/vanity, different versioning — however trickier to maintain / uncommon pattern with unknowns

[tepid nova]

is this to eliminate the need for go mod edit -replace by users? Or to avoid additional problems related to replace?

I’ll just wait for the summary issue 🙂

I wonder which sdk will end up causing the most packaging headaches… Go is building quite a lead, but it’s still early in the game

[hot smelt]

Hello, i have refacto the kapp extension with codegen and go core sdk and it works like a charm. Going from 220 to 80 lines is impressive. Honestly it feels so smooth, you guys did an amazing work 😁
https://github.com/laupse/dagger-kapp

[obsidian rover]

Hi,
I am getting this error when running some host tests go test -count=1 -race -v -run TestHostLocalDirWrite ./core/integration :

input:4: host.directory.write path "/private/var/folders/y6/3xb6hfpn7kv7vf6n9j3bq4c40000gn/T/TestHostLocalDirWrite2182375311/001" is outside workdir

Does that ring a bell to you @still garnet @wet mason ?

[obsidian rover]

Please remove your volume prior running the test, it fails with current state of Dagger. Meaning that our Github runners still have the cache ? I don't see how they would miss it

[still garnet]

hm interesting, maybe it only fails on a Mac

do you mean you found a workaround?

[obsidian rover]

Hmm, whenever I run dagger-buildkitd in a Linux container, I have the same issue

[obsidian rover]

Nope, I can check later, or you can

I'm implementing pipelines to run on native Mac os and windows runners, this might help in the future

[still garnet]

phew, reached the light at the end of the tunnel (also turns out we have exactly 100 tests)

oops, real diffstat is 36 files changed, 584 insertions(+), 730 deletions(-) (had a WIP commit)

[wet mason]

Happened to me once ... yeah I think it was cache related

[wet mason]

Sweet!

https://github.com/dagger/dagger/pull/3418

^^^ @tidal spire @hasty basin FYI we're about to remove "old API", you good with that?

[ancient kettle]

https://giphy.com/gifs/007-MKoyVvDRQvVAZbv4GD

[tidal spire]

Yeah with the go sdk I'm not using any FS stuff 👍

[cosmic cove]

@wet mason @still garnet 👀 -https://github.com/dagger/dagger/issues/3422

[wet mason]

@still garnet I'll have to do some house keeping in the repo for the release process today/tomorrow -- we should get in any big pending PRs to avoid rebase pain

There's #3418, #3402 and #3421 -- AFAIK those are the big 3 right?

[still garnet]

@wet mason yep - I merged 3418, I can rebase + merge 3402 today. 3421 is pretty hefty + has some big implications (API + internal) so I'm happy to redo that if we're not 100% sure about it on that timeline

[tawny flicker]

Hi @wet mason, is there a reason why you didn't defer closer.Close() in the main() of /cmd/dagger?

[wet mason]

Probably forgot 🙂

[wet mason]

sounds good -- worst case scenario it's just git mvs and import path changes, rebase will work, might just eat 30mins or so of your time

I'll keep the commits rebase-happy

<@&946480760016207902> Summary on the upcoming release version management, import paths, and a proposal: https://github.com/dagger/dagger/issues/3424

[tidal spire]

It looks like with the latest api, in order to clone a git repo into a directory, you now have to have a branch or tag where before it would pull the default branch automatically. Is that intended?

[wet mason]

@tepid nova @still garnet ^^^

Yeah I hit the same issue, ref-less git LLB is no longer possible in the new API

[still garnet]

good point, the master -> main shift is probably a good reason to support a default

[wet mason]

@tepid nova @twin crow @still garnet ^^^ any feedback? should I start implementing this or hold off for now? rebasing is not an option with this kind of work so it's a do-it-all or nothing

[tidal spire]

Yeah that's where I ran into it. I have it hardcoded as main but I tried to pull a project still using master

[still garnet]

had it on the side, looking more actively now!

[wet mason]

@still garnet FYI https://github.com/dagger/dagger/pull/3421#issuecomment-1282994074

I remember making a similar suggestion to Erik a while ago, and he made some compelling arguments on why it would trigger undesired side effects. I don't remember the specifics, does it ring a bell to you?

(my suggeston was dumber -- one session for each solve, I don't know if this addresses those side effects)

[still garnet]

hmm I was wondering if it would result in syncing LocalDirs multiple times, but haven't confirmed

[wet mason]

something like no guarantees between sessions that your previously computed refs are still there, or something like that

[still garnet]

yeah, that part's true, already ran into it and it's handled locally (see withRef), but I'm not sure if we have places that rely on them being longer-lived

maybe there are places that we grab a SingleRef and encode that state somewhere?

tests seem to be passing at least 🤔

[still garnet]

there are two places I could see there being a problem, both are in places where we use the dockerfile.v0 frontend. one is the shim, and the other is the dockerifle SDK. let me see if I can get this to break with a buildkit prune call.

[inner vigil]

Just butting in case it's useful. I've been messing around with a monorepo in the last year and found that the 1.19 go work use/sync/init seems to have helped with needing replace and also the gopls server in a larger project with multi-modules. I'm still wrapping my head around it as the I don't think the docs are great, but I recall it's designed to help with the need to even have replace in the module itself, which can break imports. YMMV, if you know more about it than me share away ;-p

[wet mason]

Appreciate the feedback! I've been meaning to try go work but then saw somewhere that you're not even supposed to commit the go.work file to git. But maybe that was a misinformed go forum comment?

[inner vigil]

It depends 😆
For me when doing work with pulumi and a multi-module repo, including it in the repo made sense I think as it allowed gopls to work correctly. Syncing the go work was to help align multi-module minimal version selection I think with updates too. That said... big grain of salt, just pointing it out since I noticed the replace mention. I've actually never ha d to do replace in any go mod due to the approaches I've taken so YMMV. https://go.dev/doc/tutorial/workspaces // https://go.dev/ref/mod#workspaces // https://go.dev/ref/mod#go-work-sync

go.work can be used instead of adding replace directives to work across multiple modules.
Since the two modules are in the same workspace it’s easy to make a change in one module and use it in another.
Note that Minimal Version Selection guarantees that the build list’s version of each module is always the same or higher than that in each workspace module.

YMMV, just thought I'd mention since might be useful. It's not written about much and I still find the docs confusing on this topic. Might help though if dealing with module management to look into it, cheers

[still garnet]

dynamic gateways

[tepid nova]

Request for comments: https://github.com/dagger/dagger/discussions/3432

[obsidian rover]

Since yesterday, the core/integration tests don't seem to work anymore on macOS arm m1? Am I the only one ?
Errors:

runtime.goexit
/opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/asm_arm64.s:1172 failed to solve

And:

host_test.go:190: 
            Error Trace:    /Users/home/Documents/dagger-worktree/macos-runner/core/integration/host_test.go:190
            Error:          Received unexpected error:
                            input:4: host.directory.write path "/private/var/folders/y6/3xb6hfpn7kv7vf6n9j3bq4c40000gn/T/TestHostLocalDirWrite2521125082/001" is outside workdir
            Test:           TestHostLocalDirWrite

In github action, tested against Docker Desktop provisionner: https://github.com/grouville/dagger/actions/runs/3282159658/jobs/5405174854

It also seems to times out / panic on amd64 -> https://github.com/grouville/dagger/actions/runs/3282292757/jobs/5405495771#step:7:455 (not 100% sure about that one, as I cannot reproduce locally. Might be able to try later, but let's assume this one is my fault)
@still garnet

[tidal spire]

I'm on m1 too, I'm seeing the TestHostLocalDirWrite failure but not the failed to solve error

[obsidian rover]

Can you please remove all occurences of dagger-buildkitd and try again ?

docker stop dagger-buildkitd ; docker rm dagger-buildkitd ; docker rmi dagger-buildkitd ; docker volume rm dagger-buildkitd
```then:

go test -count=1 -race -v ./core/integration

[tidal spire]

Yeah still just the one error for me

[obsidian rover]

Thanks Kyle. Just confirmed that error 1 is because I am running the tests from a git worktree, that leads to failing retrieval of VCS status 🙏

[ancient kettle]

Yeah, I get that error sometimes, too. 😢

[hasty basin]

I also found errors due to running from a git clone to my /tmp dir perhaps related to workdir permissions.

#19 go build -o build/1.18/darwin/amd64/
#19 CANCELED

#17 go build -o build/1.18/darwin/arm64/
#17 CANCELED

#21 go build -o build/1.18/linux/amd64/
#21 CANCELED

#20 go build -o build/1.19/darwin/amd64/
#20 CANCELED

#23 go build -o build/1.19/darwin/arm64/
#23 CANCELED

#24 go build -o build/1.19/linux/amd64/
#24 CANCELED
input:1: host.workdir.write path "/private/tmp/new_examples/templates/go/multiarch/build/1.18/linux/arm64" is outside workdir

[still garnet]

hm we might just need to EvalSymlinks the given workdir before doing that comparison

[hasty basin]

Makes sense. I'll create an issue

[still garnet]

looks like there's already one: https://github.com/dagger/dagger/issues/3433

I can push a fix PR but can't test it (no mac)

[obsidian rover]

I have a WIP PR that I can push, to test, on CI, with macos runners / docker desktop ifyw. It's just that I still haven't managed to make it work entirely for now

[still garnet]

https://github.com/dagger/dagger/pull/3437

[hasty basin]

I was trying to test this by pointing at your commit @still garnet , how should I refer to it in go.mod?

[obsidian rover]

Hmm, doesn't seem to be working 😿 I'll be back in ~3hours

[hasty basin]

Go noob success! I did this.

Using my repro here: https://github.com/dagger/dagger/issues/3433#issuecomment-1284247615

I additionally cloned down dagger repo and switched to @still garnet 's PR.

# from /tmp/test_bug_3433
git clone https://github.com/dagger/dagger
cd dagger
gh pr checkout 3437
cd ..

then I edited my go.mod by adding the replace directive line:

require (
    go.dagger.io/dagger v0.2.35-0.20221018121957-69716572d38f
    golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
)
replace go.dagger.io/dagger => ./dagger

then I ran

go build
# ran multiarch binary
./multiarch

worked!!

#22 exporting to client
#22 copying files 10.00MB 0.5s done
#22 DONE 1.9s
build/1.18/darwin/amd64/greetings-api
build/1.18/darwin/arm64/greetings-api
build/1.18/linux/amd64/greetings-api
build/1.18/linux/arm64/greetings-api
build/1.19/darwin/amd64/greetings-api
build/1.19/darwin/arm64/greetings-api
build/1.19/linux/amd64/greetings-api
build/1.19/linux/arm64/greetings-api

[still garnet]

oops sorry for ignoring you, I got distracted by talking about basketball

[hasty basin]

no prob, good to struggle a little to feel user pain 😄
added comments to the PR

@ancient kettle @still garnet regarding Dockerfile support (both in classic and new API/Go SDK), is there an issue y'all are using?

I saw @ancient kettle 's issue #3210, but guessing he might benefit from Dockerfile functionality in cloak.

[cosmic cove]

https://github.com/dagger/dagger/issues/3210 @still garnet could you take this one?

[ancient kettle]

This issue is for dagger-classic. I'll need "cloak" functionality under the covers to implement this... so we should create another issue.

[still garnet]

sure - there's some discussion on how it should look in https://github.com/dagger/dagger/pull/3151#discussion_r981749852 but I'll just put together a PR and we can reach consensus there instead

[cosmic cove]

Thank you! And thank you @hasty basin for pointing it out!

[rancid turret]

@twin crow ping -> https://github.com/dagger/dagger/pull/3301

[still garnet]

PR ready: https://github.com/dagger/dagger/pull/3438

[ancient kettle]

Thanks for the answers, @still garnet! I wanted to make sure I understood the semantics correctly. 🙏🏻

[cosmic cove]

@hasty basin can you take a look?

[hasty basin]

yep commented. I like it, just needs DX and arch approvals, I think.

[still garnet]

@ancient kettle does this seem like a reasonable return value for container.publish?: 127.0.0.1:5000/testimagepush:latest@sha256:542a1ab06061ec0d0bead18640f5cf75371016807a237a896a2a109dbc19760f

I was a bit surprised to see :latest still in there, didn't think that would be a valid ref (thought it was either tag or digest, not both)

seems to work fine if I feed it back into dagger 🤷

[stray heron]

That's a valid container image ref. FWIW https://github.com/gerhard/kubehuddle/commit/8fb550d04da041f90790d6d29e6997689a58ed8e

[still garnet]

sweet, ty

[ancient kettle]

I think that should be ok, yeah. Does seem a bit interesting/weird. But I trust @stray heron . 🙂

[stray heron]

Definitely works with K8s 1.22. Also works with Docker 20.10.18

[cosmic cove]

Can someone merge when you have a moment?https://github.com/dagger/dagger/pull/3438

or do we need to wait for Andrea?

[tepid nova]

I think better to wait

(for this one)

[twin crow]

I did, it's what Andrea is getting for not replying to my messages

[tepid nova]

lol

[cosmic cove]

as he sits in bed sick 😢

[twin crow]

I have his live location on my phone, I know you're not Andrea, I can see you

[tepid nova]

I think his exact words when I last asked him for his opinion on this design was: “bikeshedding overload”

[twin crow]

I'll merge 1 PR on the core api every hour without an answer: 🔫

[ancient kettle]

@still garnet deep breath You ready?

[still garnet]

oh snap, have to rebase now

[ancient kettle]

🙂

[tepid nova]

https://tenor.com/view/nbc-superstore-black-friday-gif-9684879

People looking at my github history next year: “what happened on October 19 2022?”

[tepid nova]

I'm starting to think that each SDK should:

• Have a dedicated directory in the repo, say github.com/dagger/dagger/sdk/<NAME>
• Have a per-SDK examples directory: github.com/dagger/dagger/sdk/<NAME>/examples
• Have a per-SDK docs directory: github.com/dagger/dagger/sdk/<NAME>/docs

This would match the common definition of SDK as "all the resources you need to develop for X"

Consequences would be:

1. top-level docs directory would be lighter (most stuff moved to sdk docs)
2. no more need for a separate dagger/examples repo

It feels more natural that way, and less surprising to someone browsing the repo. Look for a SDK, boom it's all there.

Thoughts?

[hasty basin]

Container add build by vito · Pull Re...

[tawny flicker]

I m starting to think that each SDK

[tepid nova]

Can anyone spare a quick review? 🙂 https://github.com/dagger/dagger/pull/3451

[rancid turret]

Both of these have POCs for the codegen to output... published at the same time 😆

[wet mason]

https://tenor.com/view/fast-street-racing-muscle-car-race-car-cars-gif-4550935

[wet mason]

Been playing a bit around this ... I don't know if it works for our use case.

I think it definitely helps removing replace directives, but as far as I understand, only parent->child

We're in a weird situation where the child (Go SDK) depends on the parent (Dagger Engine) (so we have a replace that goes "up" a few directories -- only when working locally though)

Definitely need to play with it a bit more though

/cc @tawny flicker

[tepid nova]

@wet mason would it be worth removing the special case, and just having the Go SDK fork-exec like the others?

actually forget that, go package manager is a PITA enough already, no need to throw in the bundling of a binary release

[strong coral]

Are there binary distributions yet or is all the 0.3 functionality still build-it-yourself?

[wet mason]

We could ... let's see how the first non-go SDK goes and how we're going to distribute the binary

Might make sense to get everything under the same umbrella then

[tepid nova]

@wet mason but we're keeping the plan you wrote this morning, right? 🙂

[wet mason]

@tepid nova There's the whole version mismatch chapter that we haven't figured out yet.

Take the last change we just did: support for docker build. If you yarn upgrade the SDK but don't update the CLI and attempt to do a container.Build(), it'll fail.

Right now we DON'T have that problem because the Go SDK imports the engine, so it's version locked (no mismatches possible)

Once we go to the fork-exec model, we have a small matrix from hell which needs to be solved

(Just saying -- at least by not fork exec'ing, we don't need to figure this out today)

[tepid nova]

Yeah agree. Let's not fork exec from Go, we need the extra time buffer to figure out the CLI part

also: your plan works, ship it 😄

[wet mason]

Yes

1. Plan from this morning

2. Deal with fork-exec SDKs

3. Re-assess the Go SDK: go the same route? Invisible to the user so we've got time

Implementation of the plan: https://github.com/dagger/dagger/pull/3425 <-- @tepid nova @still garnet @tawny flicker @stray heron

@tepid nova I finished assembling the bike shed and there's an extra piece left 🤔

Are we ok for the SDK to take-over dagger.io/dagger from any other (go) usage? Would sdk.dagger.io/dagger or similar be safer, or over engineered?

[tepid nova]

Yeah I think it's 100% safe.

[wet mason]

Great!

[tepid nova]

Since by definition the Go SDK is "everything you need to develop for Dagger in Go"

[wet mason]

Ok then 🙂 The PR is ready then

[tepid nova]

https://tenor.com/view/me-calling-all-gods-before-jumping-getting-ready-preparing-to-jump-diving-board-viral-hog-gif-14370898

[tepid nova]

• No binary distribution of the 0.3 dagger CLI yet. That will come soon
• As of next week the Go SDK will be released separately. It gives you a library which takes care of everything, including running and connecting to an engine
• The engine itself (API server and runner) does not have a binary release. That will take some time, but it will happen. For now you need to go through a SDK to provision it

Are you asking in the context of developing your own custom SDK?

[strong coral]

Yeah. What's a good way to get a server to develop against?

[tepid nova]

The short version is: bundle the CLI, which we will make sure is easy to instrument for this purpose

The CLI will take care of A) setting up the engine, and B) proxying your queries to it

That's how the Python and NodeJS SDKs do it

Since we haven't finalized our packaging and release plan for the new CLI, those SDKs currently use the dev version of the CLI, and are waiting for the final CLI design and release plan to drop, to adjust their dependency accordingly.

[strong coral]

I see

    def _spawn_args(self) -> list[str]:
        return [
            "dagger",
            "dev",
            "--workdir",
            self.workdir,
            "--port",
            str(self.port),
            "-p",
            self.config_path,
        ]

[wet mason]

@hybrid widget @stray heron @tepid nova is https://cloak-docs.netlify.app retired now that website-0.3 is gone?

[tepid nova]

I didn't even know those were related

Yes I believe so

[wet mason]

Yeah, cloak-docs --> github.com/dagger/cloak/website that got grandfathered into github.com/dagger/dagger/website-0.3

[tepid nova]

Exactly. This command will almost certainly change. But the basic mechanism will remain the same. Our reference SDK is Go. Other SDKs must either A) reimplement it 100% (bad idea at this stage, can be done later), or B) use the "fork-exec the CLI" cheat code

[tepid nova]

Yeah we can retire that

[wet mason]

Ok, nuking

CI is red because of that (netlify is trying to deploy it on every PR)

[wild zephyr]

@wet mason do we still need this replace here? https://github.com/dagger/dagger/blob/main/go.mod#L112

This doesn't allow me to install dagger with go install -v github.com/dagger/dagger/cmd/dagger@main which would be awesome

[wet mason]

https://github.com/dagger/dagger/issues/3391

it's because docker and go aren't happy with each other. Breaks any project that imports docker/docker, even many levels down (like us)

[wild zephyr]

is this only required for the Go SDK as per the issue? or it also breaks dagger/dagger? I'm asking since the Go SDK will be it's only module maybe we could move the replace there?

[wet mason]

No, completely cascading. docker -> buildkit -> dagger -> go sdk -> go sdk users -> users importing a lib using our go sdk -> [...]

go does MVS so even if a grand parent imports docker, then it becomes an indirect dependency which needs replacing

dagger's codebase does NOT import docker/docker -- we're getting grandfathered by moby/buildkit which is used by dagger/dagger (and the SDK uses dagger/dagger)

^^^ @tepid nova actually good point -- since we moved the SDK into its own module, if we were to spawn dagger as a binary, we wouldn't need the replace directive as we have 2 distinct go.mod's

[tepid nova]

!!!

[wet mason]

maybe that's how we solve this mess short term

[tepid nova]

I swear, I am trying so hard not to make any last minute changes to the plan, but it’s proving to be very hard! 😅

That replace directive is a major usability problem…

ah you mean post-launch right?

like short term as in “maybe next week or week after that”?

[wet mason]

yeah

upstream issue is 2 years old, so "week after that" is short term 😛

[tepid nova]

we’re movingso fast my definition of “short term” is warped to mean “before the next meal”

[tepid nova]

right 🙂

[wet mason]

There's another trick up our sleeve though, using the same technique (github.com/dagger/dagger needs buildkit, dagger.io/dagger is getting grandfathered by the engine dependency)

[still garnet]

out of curiosity, is there a root issue/discussion tracking the go<=>docker disgreement?

[wet mason]

https://github.com/moby/moby/issues/41503#issuecomment-701476992

Not really a disagreement -- more like go decided that it's semver or nothing, and docker has been tagging with non-semver friendly tags (e.g. v19.03.13 for the March 2019 release, which is not valid)

[still garnet]

ah gotcha. ty

[wet mason]

(03 is not semver)

and then the -ce for community edition which are not semver either

so basically go get picks the wrong version (one that looks like valid semver, which is a 2017 version, and of course it doesn't work)

[still garnet]

what a mess

an understandable mess, but a mess

[wet mason]

either go goes back to accepting non-semver, or docker needs to rewrite the tag history

I think that package (docker/docker) is not that much used beside internal things at docker, so there's no pitch forks. However buildkit does use it, so it affects all buildkit users

[still garnet]

yeah, it might be an easier path to "just" extract the parts that are needed into proper packages

(but that package would then have to be adopted everywhere)

[tepid nova]

using podman would solve this

[still garnet]

ill open an issue

[still garnet]

ok "just" was doing a lot of work here: https://gist.github.com/vito/e783f8ea43c8ca15897025f2540cda24

[tepid nova]

So who’s going to port Earthly to Dagger? 😁

[wet mason]

so ... what about the old school ./vendor?

it sucks to go back there, but it is what it is

[tepid nova]

but we need tooling to search replace imports no?

to keep up with upstream changes

[wet mason]

no no, go mod has a vendor mode

i've never used it

[tepid nova]

oh! I thought they killed that because it was too convenient

[wet mason]

but it was provided as a migration path during the vendor big bang

[still garnet]

does it remove the need for the replace? i'm not sure how the version resolution works with vendor mode

but i'd assume that's orthogonal to fetching (which is vendor's main focus?)

[wild zephyr]

yes since the go toolchain uses the vendor package if it finds it over anything else

[still garnet]

what if your go.mod says version 1.x but you've vendered version 2.x?

or 0.1.x vs 0.2.x to avoid the whole /v2 issue 😛

[wet mason]

looks like so 😐

[wild zephyr]

well. it should use 0,2 since they "should be" compatible per the semver spec

buildkit has a vendor folder as well (https://github.com/moby/buildkit/tree/master/vendor)

[still garnet]

hm yeah seems like a bad sign that we have this issue despite them vendoring

[wet mason]

Yeah. Looks like it's decoupled

it'll do module resolution no matter what. Then check vendor before gomodcache

[tidal spire]

I thought vendoring was no longer the way

[still garnet]

it's pretty rare to find these days, shouldn't be needed in a utopia where everyone uses semver

[tidal spire]

off the top of my head 1.16 started the move away from go mod vendor (maybe +/- a version)

[wet mason]

I'm going to (nervously) carry on https://github.com/dagger/dagger/pull/3425#pullrequestreview-1149818146

Unless I hear strong objections in the next few mins

[still garnet]

:shipitwithrecklessabandon:

I think separate go modules, despite the quirks, are still worth it for reducing the dependency footprint of the SDK (once we stop embedding the engine)

[wet mason]

https://tenor.com/view/space-cowboy-stickmen-toys-stickmen-toys-space-cowboy-stickmen-toys-stickmen-toys-lfg-stickmen-toys-to-the-moon-gif-26259582

[wet mason]

Yeah. As soon as the sdk communicates with an external engine rather than embedding, it doesn't make sense to import the whole code base just to get a client

[tepid nova]

So: fork-exec in the "short term"?

[wet mason]

I think this still makes sense

the go mod replace just added a HUGE pro in the balance for step 3

[tepid nova]

Next step for me: draft CLI packaging proposal

I'll incorporate this

[wet mason]

Going ahead with the merge -- happy to make changes post-merge, but the rebase is just too painful to wait

[tidal spire]

Is that import path change effective now or are there some other redirect bits that still need to happen?

[wet mason]

Within the repo, yes. From the outside, need to cut a release first

I'll cut a test release soon

@still garnet any PRs to get in?

[still garnet]

just the big scary one 🙃

(so no)

[wet mason]

ok if we wait monday for Erik for that one?

[still garnet]

yeah no big deal

[tawny flicker]

@wet mason I have the host envVariable

small

[wet mason]

https://github.com/dagger/dagger/pull/3461
https://github.com/dagger/dagger/pull/3348
https://github.com/dagger/dagger/pull/3435

i see these 3

[tawny flicker]

I just need to rebase and make the test pass (forgot to re go:generate the api)

@cosmic cove Is there a place where we can find all the recordings of the community calls?

[still garnet]

3348 is based on the big scary one, so that's on hold

oh, it'd make sense to merge 3461 since that's an API inconsistency atm

looks like it needs a SDK regen oh tanguy said that already 🙂

[wet mason]

@tidal spire Another FYI on #3445

The SDK will stop taking over the output of your script. To get the same behavior, use dagger.WithLogOutput(os.Stderr) on dagger.Connect()

[tawny flicker]

isn't it dagger.WithOutputWriter(os.Stderr)?

let me check my pr :p

[tidal spire]

On #3425? That's interesting, I'll play around with it. Maybe some cool stuff to do there

[wet mason]

sorry -- #3435

[tawny flicker]

3461: rebased
3435: rebase in progress

got 0 conflicts. Life is good.

3435: rebased

[wet mason]

@tepid nova @still garnet @tidal spire @tawny flicker I shouldn't even open this so close to the release, but here we go anyway: https://github.com/dagger/dagger/pull/3468

Trading a cooler SDK now for future extension pain

[wet mason]

broken :/

[tawny flicker]

https://tenor.com/view/kid-crying-cry-sad-ugly-cry-gif-25016984

but it worked on MY computer 😢

wait, at least this: https://github.com/dagger/dagger/pull/3435 worked 🙂

[tidal spire]

If you don't want to worry about this before the release, I can update the example code from

client, err := dagger.Connect(ctx)
if err != nil {
  return err
}
defer client.Close()

to

client, err := dagger.Connect(ctx)
if err != nil {
  return err
}
defer client.Close()
core := client.Core()
...
core. ...

and then the dx mostly wont change after any changes are made. Aside from the 1 line that could be removed

[tawny flicker]

https://tenor.com/view/think-smart-gif-22411928

[wet mason]

@tawny flicker good to on with https://github.com/dagger/dagger/pull/3461?

woops, already done

@tawny flicker also this can go in whenever you want, but no rush: https://github.com/dagger/dagger/pull/3372

[wet mason]

Looks like we're going to go ahead with https://github.com/dagger/dagger/pull/3468

I'll tag the repo shortly

[tidal spire]

Sounds good! Thanks for the heads up!

[wet mason]

Shouldn't be much of a change and the result should be less code/import

If you have any feedback on usage, let me know!

combining this and the import path change, basically we go from this:

import (
  "go.dagger.io/dagger/sdk/go/dagger"
  "go.dagger.io/dagger/sdk/go/dagger/api"
)

func main() {
  c, _ := dagger.Connect()
  c.Core().Directory().WithNewFile(api.DirectoryWithNewFileOpts{...})
}

to this:

import (
  "dagger.io/dagger"
)

func main() {
  c, _ := dagger.Connect()
  c.Directory().WithNewFile(dagger.DirectoryWithNewFileOpts{...})
}

[tepid nova]

Last call for giving your opinion on what to call the new "classic" cue CLI: https://github.com/dagger/dagger/discussions/3432#discussioncomment-3927435

By default the name will be dagger-cue, and it will be bundled as part of the new CUE SDK

@ancient kettle just to confirm you're OK with dagger-cue as opposed to your initial suggestion of cue-dagger?

[ancient kettle]

Yeah, dagger-cue is my preference!

[tepid nova]

OK 🙂

[wet mason]

Can I get a quick lgtm here? https://github.com/dagger/dagger/pull/3469

[tepid nova]

done

[wet mason]

🙏

go: module dagger.io/dagger: git ls-remote -q origin in /Users/al/go/pkg/mod/cache/vcs/a50c891df610f8f5426773184f3ecbd8723af7eddeff873315689b36a16ce91e: exit status 128:
    remote: Not Found
    fatal: repository 'https://github.com/dagger/dagger/sdk/go/' not found

ugh. vanity URL is not liking the sub-module

[tidal spire]

Yeah I just tried the same thing 😞

[still garnet]

uh oh?

[twin crow]

turn your computer off and on, and try again

[wet mason]

didn't work

so. close.

[tepid nova]

So plan B is to move the root gomod to ./engine so there are no submodules involved?

[wet mason]

No, plan B is much worse than that. We'd have to go back to the previous state

[tepid nova]

https://tenor.com/view/i-cant-go-back-man-martin-sheen-robert-hanson-grace-and-frankie-head-shake-gif-17586247

well that sucks

anything we can do to help?

[wet mason]

I'm going to take a break, I'm running in circles. I'll try something else

[tepid nova]

let’s just port everything to rust

[wet mason]

ok so, first, I'm going to revert this -- it'll be dagger.io/dagger/sdk/go/dagger, same go.mod, same versioning. Just so I un-screw-up all the demos etc since now it's all broken

and I'll look into something else tomorrow

the pain: https://github.com/dagger/dagger/pull/3473 https://github.com/dagger/dagger.io/pull/1018

I have to fork the repo and use my own subdomain to experiment with this and find a better alternative

[wet mason]

it's green -- @tepid nova mind doing a quick lgtm?

[tepid nova]

which one? both?

done

[wet mason]

Thanks!

[tidal spire]

This isn’t helpful now, but here’s this anyway https://github.com/golang/go/issues/34055

Vanity urls for sub modules someday in the future. Maybe there’s a workaround somewhere in there that I missed

[tepid nova]

maybe having one repo per SDK wouldn’t be so bad, compared to this? multi-repo management can be repetitive and error prone… but that can be fixed with automation. No amount of automation will fix Go sub modules …

Also we can automate multi-repo workflows with dagger, could be a sweet dogfood opportunity 🙂

other arguments:

• CUE SDK needs a separate repo anyway, for now. Would save us the effort of merging it back in later

• Github issues would be easier to file and navigate: right now lots of Cue issues, soon mixed with python and go issues…

https://tenor.com/view/rest-my-case-i-rest-my-case-lawyer-courtroom-gif-14813556

[still garnet]

github releases will also be much clearer with multiple repos, assuming we're planning to do individual releases for each SDK

[wet mason]

Everything should be working now

[tawny flicker]

Yeah, the TL;DR is that it is a valid use case, but noone is working on it for now.
I guess contribution welcome.

Also, even if our import path would be: dagger.io/dagger, the version wouldn't be v0.1.0, but it would use the subdir prefix: sdk/go/v0.1.0 (https://github.com/golang/go/issues/34055#issuecomment-785279844)
So go get dagger.io/dagger@sdk/go/v0.1.0 if we want it pinned. I guess @latest should be the most common used, though.

[inner vigil]

I did a year with a monorepo and wow did it take a lot of care. So much tooling is driven odd one project is one repo. Go has only started down multimodule support with go work and i still don't fully grok it 😆. Seperate sounds a lot easier for multi language for sure. If you want one other example of a SDK that does multi language in one place pulumi does. You can see their GitHub provider repo for an pattern. I feel like they are one of the best at SDK generation.

[tepid nova]

FYI @vital plaza and @plush pond are having issues importing the Go SDK today.

Actually I will post in #help

Looking at the situation in #help ... I'm feeling like a separate repo per SDK would help with support (issues)... With all SDKs mixed together, I think it's less likely that developers will find the answer to their question (basically help will be write-only)

One alternative could be to steer people towards GH discussions rather than issues - and we can use categories for SDKs.

[wild zephyr]

do you mean that Discord labels don't seem to be working? I don't see it mutually exclusive. In fact, the current Go SDK situation should be an issue in dagger/dagger with the propler go sdk labels.

I'm not sure about this support case being a compelling example of vouching for multi-repo.

[tepid nova]

Yeah discord labels are working. I just think as we grow, they may not be enough. Maybe I'm wrong

For example I noticed posters often don't set the tags. I just added tags after the fact to a few posts.

[wild zephyr]

May be we can improve this message to highly recommend to use labels?

[tepid nova]

Sure. Not a big deal right now. Let's see how it evolves.

(done)

[wet mason]

Update on packaging/vanity URLs:

Unfortunately, the path within the repository cannot be hidden using vanity URLs, so if we remain on a monorepo the import path has to be:

import “dagger.io/dagger/sdk/go/dagger”

OR

import dagger “dagger.io/dagger/sdk/go” (e.g. we the import doesn't end with dagger, we'd have to alias it to something anyhow)

The only way to go around that limitation is splitting up repositories. I would much prefer we keep a monorepo, at least in the foreseeable future

Barring the import path, we still have options:

Option X:

• Single go.mod
• Single versioning
• Life is easy, but we have to release everything at once.

Option Z:

• The original plan is still viable -- aside from the import path
• Involves creating a sub-module (e.g. sdk/go)
• Allows to have independent versioning
• Life is a bit harder, but we can independently release. We just lost the custom vanity URL to soften the blow of having to deal with sub-modules

/cc @tepid nova @twin crow @still garnet @wild zephyr @tawny flicker

[still garnet]

Playing devil's advocate, but secretly I've already sold my soul: what if we just did multiple repos? I see a lot of pros to it, and the cons feel like they can be mitigated with automation, but I'd like to hear more about it

[wet mason]

Any go wizards that can help me find the right filter-branch incantation to create a new repo that only contains sdk/go?

[still garnet]

https://docs.github.com/en/get-started/using-git/splitting-a-subfolder-out-into-a-new-repository turns out github has docs on it!

[wet mason]

@still garnet Yeah, saw that

I'm now wondering if we should do that on all commits or just tags

The thing with everything+tags, I'm not sure how to mirror tags since I don't know what upstream tags point to (the ref doesn't match the fork)

Also -- it's probably a bit more involved since we need to remove the replace statement from go.mod ...

[still garnet]

is there a possibility to only tag the sdk repo? it makes it less of a dummy, but I think it'd be closer to the theoretical world of true multi-repo anyway

[wet mason]

yeah, however you can't point to main from the outside, is that ok?

or maybe you could with a replace statement pointing to the monorepo?

[still garnet]

do you mean the sdk => engine dependency path? (for embedding?)

or do you mean client => sdk@main

[wet mason]

the latter yeah

like "I just pushed WithNewDirectory" --> let me try --> oh no you have to wait for a release next week

[still garnet]

wouldn't you just get the latest commit synced from sdk/go?

[wet mason]

well if we do tags only, then no it'll only happen once we tag, not everytime we push to main

rephrasing the original question: should the sync workflow kick off when we push to main or only when we create a new tag on dagger/dagger?

[still garnet]

oh i see, yeah that's the disconnect

I was thinking every push to dagger/dagger main will sync commits (if any) to the external repo, and the tags are unrelated between the repos

we only tag dagger/dagger for engine releases, we only tag dagger/go-sdk (or whatever) for sdk releases

[wet mason]

oh I see

[still garnet]

unsure if we even tag dagger/dagger engine releases at this point, but yeah

and for local dev we would always just have a replace rule to make sure the tests/etc use the local SDK

[wet mason]

I imagined we'd still tag the main repo with sdk/go/v0.1.0 (like we'd do without proxy) and use that to sync tags

Otherwise we'd have to "blindly" tag the proxy repo since no commit id matches

[still garnet]

yeah, different perspectives on the source of truth i guess

[wet mason]

(e.g. otherwise git pull the proxy repo and hope HEAD matches whatever you want to tag?)

like, if you merge a PR and git tag the proxy repo ... maybe the GH action finished syncing up, maybe not?

and since all the commit IDs are different, you don't really know what it is that you're tagging

(well you can match the commit message)

(but that's harder to automate)

[still garnet]

grr I know how I'd do this with concourse but don't really want to 😂

hmm I guess it's up to how we know what to tag

I assumed that'd be a human for now pushing a "shipit" button so we'd have a rough idea of what we're shipping

[still garnet]

I'm good with this too, seems simpler, we don't really need it to be continuously synced since it's just our release process

local dev should be using replace anyway

also: it's possible to continuously/incrementally do the filter-repo thing, right? 🤔 hopefully we get the exact same commit hashes and everything pushes cleanly...?

[wet mason]

Yep, looks like it

[still garnet]

phew

[wet mason]

I'm going to end up in git jail

[still garnet]

😂 it's ok we'll get you out with git reflog

[wet mason]

not yet fully working but:

git filter-branch -f --prune-empty \
    --subdirectory-filter sdk/go \
    --tree-filter  "go mod edit -dropreplace github.com/dagger/dagger" \
    -- tmp-branch

not only it rewrites the history and promotes sdk/go as the root folder, but it also drops the local rewrite to ../..

(since the sdk has a rewrite to ../.. for the engine it embeds -- otherwise the mirror repo will be broken)

not fully working yet

[still garnet]

nice

[wet mason]

by that I mean fully broken

I'll continue on Monday, too much git

[tepid nova]

observation: we could totally have a dagger extension that bundles those cryptic git filter tree commands into a neat reusable pipeline

[tepid nova]

query gitFilter($dir: DirectoryID!, $command: [String!]!) {
 gitFilter(dir: $dir, command: $command) { id }
}

[tepid nova]

I kind of wish the Go SDK provided an optional "runner" like Mage, where you only need to write a function for each pipeline, and the main is already written for you. Does that make sense to anyone else?

[still garnet]

https://github.blog/changelog/2022-10-21-explicitly-set-the-latest-release/ useful, helps mitigate one monorepo pain point I mentioned

[wild zephyr]

dang.. now I need to change bin to support that 😄 (https://github.com/marcosnils/bin)

[still garnet]

neat!

[tepid nova]

Hi @wet mason @cosmic cove @tidal spire @civic yacht @still garnet

We might need some help getting the Go SDK 0.1.0 release done, we have a few loose ends. Starting a thread.

[tepid nova]

@civic yacht about that mysitfying cache issue... It seems like probably a very very basic but very very hidden problem, like not talking to the version of buildkit you think you're talking to? Or running a build from the wrong branch or something?

[civic yacht]

I've been trying to rule things like that out... Been verifying I'm running the code I think I'm running by adding printlns and such

[civic yacht]

Figured it out 😞 #wandb-ai message

[tepid nova]

@wet mason are imports at dagger.io/dagger supposed to work now? Or not yet

[wet mason]

not yet

[tidal spire]

is there an import path that should be working right now? I've been locked on an old commit

[wet mason]

dagger.io/dagger/sdk/go/dagger, just have to go get with @latestcommit

[tidal spire]

Yeah for me that gives:

go get dagger.io/dagger/sdk/go/dagger@main
go: dagger.io/dagger@v0.0.0-00010101000000-000000000000: invalid version: unknown revision 000000000000

see also #1033028060615819304 message

(same with @69419c0)

[hasty basin]

same

[tepid nova]

FYI I merged @civic yacht 's PR, no need to confuse everyone with a temporary import path in the docs (mentioning it here on discord is fine of course)

[tepid nova]

Can anyone spare a LGTM on these tiny docs edits?

https://github.com/dagger/dagger/pull/3499

[wet mason]

my laptop is dying on me

[tepid nova]

If anyone is still awake, are you able to rebase your branches on main and push them to your fork?

I'm getting a weird git error:

$ git push
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 20 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 298 bytes | 298.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To github.com:shykes/dagger.git
 ! [remote rejected]     README -> README (shallow update not allowed)
error: failed to push some refs to 'github.com:shykes/dagger.git'

Never seen that one before..

[fair ermine]

I just created a branch and yes I can push on my fork 😮

$ git push tom
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 977 bytes | 977.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:TomChv/dagger.git
   c6e549bd..5c36443d  feat/test-push -> feat/test-push

[tepid nova]

Ok cool , thanks for checking 🙂 Not sure what I did to mess up my repo.. All I'm doing is very basic docs & README edits...

Anyway I bypassed the issue for now by pushing to a branch on upstream

[fair ermine]

Same here, I just added a \n

[tepid nova]

@fair ermine while you're here could you spare a lgtm? 🙂 https://github.com/dagger/dagger/pull/3529

@fair ermine and if you're interested, we just shipped brand new docs and website, feel free to take a look and share feedback, questions..

[fair ermine]

With pleasure! Reading Cloak doc on GH was not the best haha

[tepid nova]

Thanks 🙂 Yeah the docs have come a long way

(but only one new language for now)

Also this one 🙏: https://github.com/dagger/dagger/pull/3530

[fair ermine]

Approved, I left comments but you can ignore them if you wanna merge directly, it's minor things.

[fair ermine]

Is there a discussion in #1030538312508776540 or an issue to share feedback?

[tepid nova]

@fair ermine I was thinking about creating a separate #feedback channel structured like #help, but specifically to share notes and feedback. WDYT?

btw welcome back to the channel, it's good to see you around again 🙂

[fair ermine]

What's the advantage since we have GH discussion?

[tepid nova]

Probably lower friction? People generally seem more inclined to share notes on discord first, since it's often unstructured, and interactive

[fair ermine]

So it's a good idea, it may lead to more spontaneous feedbacks!

[tepid nova]

Ok how about I create it now, and you try using it for your feedback. If it turns out to be a bad idea tomorrow, I'll remove it

[fair ermine]

However, I think it's important to also use a GH discussion to keep one source of truth at the end

[fair ermine]

I let you create the post "website" or shall I do it?

[tepid nova]

@final nymph go for it 🙂

don't forget https://github.com/dagger/dagger/pull/3530 🙏

[fair ermine]

Done!

[tepid nova]

many thanks

[fair ermine]

It feels good to be on the same Timezone 🤣

[tepid nova]

Stay up another hour and you'll be right back in France

[fair ermine]

I'm part of the late-night team so yeah definitely 🤣
@obsidian rover 👋

[obsidian rover]

Following @fair ermine's feedback. Fixes a typo: https://github.com/dagger/dagger/pull/3531

[hasty basin]

For @warm tundra: looking for state of withCopiedFiles and include/exclude on Host workdir and local dirs.

[ancient kettle]

https://github.com/dagger/dagger/issues/3463
https://github.com/dagger/dagger/issues/3264#issuecomment-1287230360

I think that both of those are still in development.

[still garnet]

it's implemented, but based on the new host API w/ dynamic sessions, so it's a bit logjammed at the moment. it could probably be redone for the current host API if we want it sooner

[bronze hollow]

Wow. I get busy for a few weeks and you drop a Go SDK that looks sweet. Great job! Excited to catch up and play with it. Is TS far behind?

[tepid nova]

Thanks! Typescript not far behind 😁

[bronze hollow]

AHH. From a previous comment, NodeJS is about a month away. Is there a fork or branch I can catch up with it?

I might just use Go. It looks like fun

[tepid nova]

there’s a dev version of the nodejs sdk, but it will change heavily. Final version will look at lot more like the Go SDK. In particular it will have fully generated bindings without leaking graphql

[bronze hollow]

Awesome

Now to go find out what extensions look like with the new SDK 🙂

[tepid nova]

Extensions are not shipped yet.. Wanted to finish the design without rushing it

It’s still there in dev, but hidden. Roughly same as what you saw

major benefit is that the community can start adopting the Go SDK with ou having to also grok 1) graphql and 2) extensions with a codegen workflow. That’s a lot to digest… We’re spacing out the launches for better digestion 🙂

[bronze hollow]

Got it. Thanks for the updates 🙂

[wet mason]

pkg.go.dev links are broken, probably something about sub modules, trying this: https://github.com/dagger/dagger.io/pull/1082

[hasty basin]

This works for me, for example, something different? https://pkg.go.dev/dagger.io/dagger#Container.Build

[wet mason]

@hasty basin if you click on "Build" you're redirected to github, and the link is broken

[ancient kettle]

Hmm... I'm getting a weird set of errors.

container.withFS.withMountedDirectory.withUser.withWorkdir.exec.exitCode open /tmp/buildkit-mount3069500825/meta/exitCode: no such file or directory
container.withFS.withMountedDirectory.withUser.withWorkdir.exec.stdout.contents open /tmp/buildkit-mount3255851064/meta/stdout: no such file or directory

That reads to me as the shim mounts are getting broken or something...

dagger.io/dagger v0.3.1, by the way.

[civic yacht]

Oh are you mounting anything at /dagger by any chance? I think I saw @hasty basin make an issue the other day about that

We should really just change the mount, it can be named /......jfdklajfdksla or anything really

[ancient kettle]

I don’t think so… but I’ll check.

        git: {
            repo: core.#GitPull & {
                remote: "https://github.com/dagger/dagger"
                ref:    "main"
            }

            image: core.#Pull & {
                source: "alpine:3.15.0"
            }

            verify: core.#Exec & {
                input: image.output
                // args: ["ls", "/"]
                // args: ["cat", "/etc/alpine-release"]
                args: ["cat", "/dagger/README.md"]
                mounts: {
                    a: {dest: "/dagger", contents: repo.output}
                }
            }
        }

I am. 🙂

Thanks, @civic yacht!

[civic yacht]

No problem, but you're also not doing anything wrong, I'll send out a fix, I think it's literally one line 🤞

[ancient kettle]

🤘🏻

I agree… and it’s nice to know why it was breaking. 🙂

[civic yacht]

https://github.com/dagger/dagger/pull/3548

[oak sandal]

Really excited to see the Go SDK. Wondering if it's possible to integrate it with Buildpacks SDK? Basically, build images with pack client pointing to dagger engine/buildkit and then use the output with dagger SDK.

[oak sandal]

In general, am I right in viewing dagger as a user-friendly way to interact with buildkit?

[tepid nova]

Yes that’s a good summary 🙂 There is more to it but it’s accurate.

EDIT: actually it depends what you mean exactly by “interact”

[tepid nova]

What is pack client?

Buildpack integration

[oak sandal]

That's great. Oh, I meant Pack client as buildpacks sdk. Buildpack sdk has a method to create a client to a docker daemon(not sure if there are others) and it then lets us create images using buildpacks.

[bronze hollow]

Is there a SourceDir() with the new Go SDK?

I guess I can plumb this myself

[obsidian rover]

https://pkg.go.dev/dagger.io/dagger#Host

[bronze hollow]

Yeah, in a mono repo environment it's going to look a little different as I support global and service level builds

I'm building a quick spike now

[ornate rover]

@bronze hollow I was playing with that just before figuring that out:

src, _ := client.Host().Workdir().Read().Directory(".").ID(ctx)

And that doc link above.

There may be a better way - but that worked.

[bronze hollow]

Is there an example of working with secrets with the Go SDK?

[bronze hollow]

Nice, thanks!

[ornate rover]

If you find something for secrets I'm keen - couldn't seem to auth to my private GH repos, but build java/mvn for my OSS stuff now. Might even have to script up a screencast or something on this.

It's definitely going to be a topic for the next podcast episode if we ever kick off again.

maven = maven.WithMountedCache(cacheID, "/root/.m2/repository")
maven = maven.WithMountedFile("/root/.m2/settings.xml", mavenSettingsId)

That inconsistency in the ordering of string and FileId/CacheId irks me. I see mounted directory/secret are all string first - wonder if WithMountedCache should be switched.

[obsidian rover]

Hi, can you please add this in the feedback section. It's great insight

[ornate rover]

Posted.

[bronze hollow]

OK. I think I got blocked trying to do something that should be an extension but I was trying to do as code

[still garnet]

@civic yacht so I copy-pasted Buildkit's filesync implementation and just made it so llb.Local("/foo/bar") uses that directory as the path, and it just works, no upstream changes needed. sneaky!

❯ go test ./core/integration/ -run Host -v
=== RUN   TestHostWorkdir
=== PAUSE TestHostWorkdir
=== RUN   TestHostDirectoryRelative
=== RUN   TestHostDirectoryRelative/._is_same_as_workdir
=== RUN   TestHostDirectoryRelative/./foo_is_relative_to_workdir
=== RUN   TestHostDirectoryRelative/../_does_not_allow_escaping
--- PASS: TestHostDirectoryRelative (0.40s)
    --- PASS: TestHostDirectoryRelative/._is_same_as_workdir (0.00s)
    --- PASS: TestHostDirectoryRelative/./foo_is_relative_to_workdir (0.04s)
    --- PASS: TestHostDirectoryRelative/../_does_not_allow_escaping (0.00s)
=== RUN   TestHostDirectoryReadWrite
=== PAUSE TestHostDirectoryReadWrite
=== RUN   TestHostVariable
=== PAUSE TestHostVariable
=== CONT  TestHostWorkdir
=== CONT  TestHostVariable
=== CONT  TestHostDirectoryReadWrite
--- PASS: TestHostDirectoryReadWrite (0.47s)
=== RUN   TestHostWorkdir/contains_the_workdir's_content
=== RUN   TestHostWorkdir/does_NOT_re-sync_on_each_call
--- PASS: TestHostVariable (1.32s)
--- PASS: TestHostWorkdir (1.64s)
    --- PASS: TestHostWorkdir/contains_the_workdir's_content (0.79s)
    --- PASS: TestHostWorkdir/does_NOT_re-sync_on_each_call (0.32s)
PASS
ok      github.com/dagger/dagger/core/integration       2.055s

just had to remove invert the test that says the workdir is synced on each call, since we no longer want that

i'll prepare a new PR for this and leave the dynamic sessions one around for now

i'll add a readme + copyright notice to the copypasta too

[ancient kettle]

Really curious to see this...

[still garnet]

@tepid nova we might need to start thinking about how to safeguard this really soon; once it hits the playground for example people will be able to read from the entire host machine

you also mentioned preventing extensions from using it

[civic yacht]

related issue, added a comment: https://github.com/dagger/dagger/issues/3072

[civic yacht]

That's amazing, I'm really glad it actually worked out in practice too. I've also wondered if going this route could unlock other interesting possibilities in the long run. Like, would it be possible to now sync files that aren't actually in the local filesystem? Like if we wanted to "stream" data in could we reuse this mechanism? If we can, should we? Has science gone too far? etc.

I have getting back to your comment on that PR in my queue too btw, from my initial skim I think we are on the same page for the most part actually.

[still garnet]

🙂 I'm glad, I spent way too long on that comment and didn't want it to feel exhausting/stubborn

[still garnet]

yeah, I think this could open the door to streaming through the API layer and directly into Buildkit somehow, for a remote engine 🤔 like some kind of "hook" from a streaming endpoint into the attachable object and a bit of synchronization

but I haven't looked too deeply into the code yet, because I really didn't have to change much (I thought this would be a grander adventure...)

[civic yacht]

Yeah that would be through services+websockets probably, which honestly may make more sense and is overall more dogfoody, I was just intrigued by the possibilities of hooking into the filesync api deeper, but that doesn't mean it's a good idea or worth pursuing at all

[still garnet]

https://github.com/dagger/dagger/pull/3560

[tepid nova]

Good point. I have ideas, what's the best issue for me to participate & follow?

[still garnet]

probably here: https://github.com/dagger/dagger/issues/3072#issuecomment-1292400376

well, actually that's for extensions I guess, for Playground we need something more general

maybe just the PR then? (https://github.com/dagger/dagger/pull/3560) - still trying to think about whether this has implications beyond the Playground

(lunch, brb)

[civic yacht]

I think it could be a somewhat similar idea though, we can just not expose those APIs in the playground. I guess the general idea being that there shouldn't be a single "global" schema, there has to be a more configurable one

[still garnet]

yeah true, we really don't need to expose the host at all eh

I was thinking about restricting to certain paths, but that doesn't seem necessary

[ancient kettle]

<@&946480760016207902> I'm planning to add a draft PR for my dagger-cue work, so we can have something solid to start discussing. I was planning to submit it against the current cue-sdk branch? Is there somewhere better?

[tepid nova]

Exciting!

I see 2 options for the PR:

1. cue-sdk branch, and we move to main later
2. main branch directly

I am leaning towards option 2, for simplicity (that's where we want the code to live in the end). But curious what others think

[ancient kettle]

I'm flexible. One argument for option 1 is that we'll be able to see the diff against the old version, which might make what I'm doing with the Go SDK clearer.

[tepid nova]

Yeah that's true

@wet mason ?

[wet mason]

How much of the old codebase is being reused vs replaced?

[ancient kettle]

Much of the old structure is being maintained, I've largely just replaced the task implementations thus far + added some utility functions around Directories, Secrets, etc (because I'm storing them in the old #FS/#Secret ids.

I still have a fair bit of code to clear out, that hasn't been my priority thus far.

[wet mason]

No strong opinion, but probably easier to do a PR against cue-sdk (so we can see the diff), then working on clearing out stuff (e.g. nuke the old buildkit code, etc -- there's tons of stuff to delete) and once it's a reasonable size we can look into moving that into main

[ancient kettle]

I like that approach! Thanks for the feedback.

[wet mason]

There's some stuff not supported yet in the Go SDK like logs (and the "cue sdk" has a bunch of code to handle logs), we probably need to figure something out

Also -- extensions / universe: we should probably "nuke" the old universe and have codegen for CUE to get the extensions? idk, it's a fuzzy area, and cue-sdk without universe is not very useful I guess

[ancient kettle]

Yeah, I definitely have open questions around that stuff. I've been working on documenting what is/isn't available (here: https://github.com/dagger/dagger/issues/3121)

[ancient kettle]

I'm not sure about nuking universe, but codegen for extensions and such makes sense to me.

[tepid nova]

That's for a future, unrelated PR though right? Right now we're talking about compat at the core action layer, which preserves universe

[wet mason]

Me neither, but it's probably going to be confusing to have two universes with the CUE SDK having its own separate universe

[tepid nova]

at this point "universe" is just a collection of CUE packages, specific to the CUE SDK

It's just an artifact of the language siloes in our ecosystem, until we unify it with extensions

[wet mason]

Yeah, but CUE universe brings its own huge CI -- I think it's probably cleaner to start on a branch/repo and bring to main eventually

[ancient kettle]

Totally get that. I was thinking codegen + a translation layer for #FS <-> Directory and #Image <-> Container

[tepid nova]

Ah I see. I guess we may have the same question for all the other SDKs soon? CUE just has a head start because it's more mature than the others

but I guess we won't allow the language-specific ecosystem to grow as larger as it did for CUE, before unifying with extensions. So their CI won't grow as huge

@wet mason so, option 1 for now (PR against cue-sdk) and we figure out the rest as we go?

[wet mason]

Yeah, easy

[tepid nova]

There you go @ancient kettle 🙂

[ancient kettle]

Thanks, @tepid nova + @wet mason. 🙂 I appreciate the help!

[still garnet]

Thinking about this more, I think we still need a safeguard since you could theoretically craft a directoryid that references a host path without using the API

E.g. copy paste from local dev

[ancient kettle]

https://tenor.com/view/red-team-just-won-we-won-cobra-kai-youtube-originals-gif-13997130

@still garnet I like your devious mind.

[civic yacht]

Good point! Hm I guess the only thing I can think of is we'll have to introspect all the LLB we send and find and local:// sources. There's also a whole capability system in buildkit, not sure if it can help here

[still garnet]

I think we could change our filesync impl to have some kind of check, and for playground we just have it always return false. might even be able to fold that into an upstream PR. basically change it from taking a static list of local dirs to taking a lookup function that returns (SyncedDir, bool)

[civic yacht]

I was wondering that too, there's also possible nastiness where an evily crafted ID could specify a session ID to use for the local, but I guess if all the connected clients are guaranteed to be the playground client that never responds with any actual local dirs, that would be fine

Should probably run the playground clients in locked down containers where even if somehow they are tricked into responding w/ localdirs there's nothing of interest to load

[still garnet]

would that be assuming the attacker knows the session ID of an active connection that has local dirs configured? or alternatively one that's using our own filesync that allows everything?

[civic yacht]

Yeah they'd need to know the session ID, which is random, so it's probably not exploitable on it's own, but still seems like it could be chained with other bugs to do bad things

I think working at AWS made me paranoid because the threat model there is "literally anything you could conceive of plus more" 🙂

[tepid nova]

It would help to know the architecture of the playground. Since the engine itself is "in between architectures"... playground is kind of in a weird in between as well. Hard to discuss safeguards without knowing the architectural context

[civic yacht]

Agreed, I'm making millions of assumptions right now

[still garnet]

makes me wonder if we're exposing a bit of a live wire by putting LLB directly in our IDs... but it sure is convenient!

[wild zephyr]

the current playground architecture is just standard dagger dev API started from an empty workdir. We still haven't reached the point where we started fiddling with advanced sandboxing techniques like user namespaces and other things. Next step is exactly to start trying that out

[tepid nova]

Is it a long-running dagger dev for all users?

[wild zephyr]

correct

[tepid nova]

That reminds me we need to rename dagger to cloak in the main branch as we discussed @wet mason , or people will get very confused (I saw a few already)

[wild zephyr]

@tepid nova you mean the CLI ? I can open a PR for that

[tepid nova]

Yes, to avoid confusion since we are planning a new dagger CLI and it will almost certainly break the syntax

[wild zephyr]

👍 I'll send a PR in a minute

[tepid nova]

(great if you can open the PR, thanks!)

[wet mason]

Can someone do a quick review? CI is broken

https://github.com/dagger/dagger/pull/3564

@still garnet Is there a way to use include/exclude in copy?

[still garnet]

not yet, I had a branch for it but it was dependent on dynamic-hostdirs for no good reason

it could either be redone on top of the current host API or redone on top of https://github.com/dagger/dagger/pull/3560

[wet mason]

and btw -- should we call WithDirectory() WithCopiedDirectory()? Was confused by WithCopiedFile

not sure what the diff is between the two

(e.g. WithDirectory is copying right?)

[still garnet]

we talked about that before too 😄 do you mean on Container?

#1033044388831567943 message

tl;dr yea, WithCopiedDirectory seems more clear alongside WithCopiedFile

[wet mason]

sorry, rambling 😄 dogfooding a bit

@still garnet also -- should "copy" stuff do a mkdir (e.g. lazy create destination)?

I think we are doing that in the CUE SDK

[still garnet]

in my mind the low hanging fruit is: new host API, include/exclude, Container.withCopiedDirectory, and renaming Directory.with{,Copied}Directory

[still garnet]

i think it does already

[wet mason]

I don't think so, had to hack around with

        Exec(dagger.ContainerExecOpts{
            Args: []string{"mkdir", "-p", "/app/sdk/go"},
        })

[still garnet]

ah, yep, it's set for withDirectory but not withCopiedFile!

[wet mason]

same with Workdir.Write

Error: input:1: host.workdir.write lstat /Users/al/work/dagger/bin: no such file or directory

[still garnet]

just braindumping from Roam... (not all of these have issues):

• {{[[TODO]]}} finish new host API
• {{[[TODO]]}} file include/exclude
• {{[[TODO]]}} rename Directory.withDirectory
• {{[[TODO]]}} Directory.withCopiedFile should mkdir dest
• {{[[TODO]]}} add Container.withCopiedDirectory
• {{[[TODO]]}} add Container.withCopiedFile
• {{[[TODO]]}} Directory.export should mkdir dest

last one is analogous to workdir.write (that's the old API)

good thing we launched with a compiled language first 😛

[wet mason]

also WithoutEntrypoint? I thought it was already done but I don't see it

[still garnet]

oh, yeah, we have an issue for it. right now you just do WithEntrypoint()

https://github.com/dagger/dagger/issues/3284

[ancient kettle]

@tepid nova @wet mason https://github.com/dagger/dagger/pull/3565

Plenty more to do, but the bones are there for pretty much all of stuff I can accomplish with the current Dagger GraphQL library.

[wild zephyr]

That reminds me we need to rename

[ancient kettle]

@wet mason @still garnet @civic yacht Any thoughts on https://github.com/dagger/dagger/issues/3442 ? It's a pretty large blocker for a number of issues for dagger-cue.

[still garnet]

haven't really had time to think about this much, but I think there should be a way to pass your own secret provider in somehow. I don't think we want a secret creation API, but I'm not against it if someone has a good idea (mainly want to avoid state and/or putting sensitive values directly in a SecretID)

e.g. dagger.Connect(ctx, dagger.WithSecretProvider(dagger.StaticSecrets{"foo": "bar"})) - with StaticSecrets conforming to some kind of interface

should be somewhat easy, at least until we have remote engine (but that makes everything harder so nothing new there)

relevant: https://github.com/dagger/dagger/pull/3124

[ancient kettle]

I like this! Happy to manage the secrets myself for now. A simple map (or similar) would work great.

[ancient kettle]

Thanks for the link! I'll check this out.

[tepid nova]

Should we make a list of "easy contributions" and maybe label the issues accordingly?

When I have a little extra time between other tasks, that's always what I look for, something useful but not too big. I know I'm not the only one

[wet mason]

@still garnet @civic yacht switched the markdown linter to use dagger (works locally now). Nice to see the API in action

https://github.com/dagger/dagger/pull/3568/files

[wet mason]

(/cc @hybrid widget -- you can now run the docs linter locally)

[tepid nova]

That's awesome! What's the command to lint locally?

[wet mason]

mage lint:markdown -- or mage lint:all to run all the linters

[wet mason]

if someone can review that PR btw 🙂

[tepid nova]

Wondering why args aren't a positional argument in Exec?

[rancid turret]

I guess because it's an optional argument and in the Go SDK those go into an opts struct.

[wet mason]

this. args are optional because of WithEntrypoint etc

[tepid nova]

ah! Makes sense. Probably a slightly better tradeoff would have been to make it positional anyway, and if you really want no arg, you say nil - not weird since exec is universally understood to usually take arguments.

But I'm guessing the above 👆is trickier to do with codegen

[still garnet]

yeah, starts to feel like we need more hints from the schema

like parameter order, maybe even variadic params so it's Exec() or Exec("go", "build", "./...")

[tepid nova]

variadic kind of screws up the opts though

[still garnet]

oh do we have other opts for exec? drat

ah right stdio redirects

[tepid nova]

We could go all the way and make this With.. too

[civic yacht]

Thinking about it again, I honestly wonder if we even need args to exec in the schema. They could just be fields on Container (withStdin, withRedirectStdout, etc.). In the past I thought that would be confusing, but I'm less sure now.

[tepid nova]

even args?

feels like I'm doing assembly again, pushing args to the stack (or registers, don't want to give away what I was programming on) 😄

[civic yacht]

Container().From(blah).Args([]string{"echo","hi"}).Exec().Args([]string{"echo", "bye"}).Exec() doesn't look that bad? Not totally convinced but maybe...

[tepid nova]

I think it might be a bridge too far.. Someone used the words "polish calculator notation" yesterday already 😄

Seems kind of arbitrary to put even stream redirects in a With but not args... but feels "right" for some reason

[still garnet]

i think what feels odd is redirects are a property of an individual process, which we can visualize as foo < in > out 2> err. i mean, you could say the same for env BAR=1 foo but technically that's two exec()s

[tepid nova]

yeah that's the thing we've muddied the waters already, BAR=1 foo is one exec

but you're right it does feel more odd

part of it is the (also arbitrary) convention of what you can persist in a dockerfile

[still garnet]

true, but that's because of your shell, so technically still two processes involved 🤓

I may still be treading on thin ice with this argument though

[tepid nova]

@still garnet before I share this link let me say it is completely unnecessary and purely for the joy of continuing this debate 😉

https://man7.org/linux/man-pages/man2/execve.2.html

[civic yacht]

Also, if someone disagrees with our choices, in theory they should someday be able to write an extension that provides a type called DifferentlyOpinionatedContainer that provides an API they like more. So at least there will be an escape hatch someday

[still garnet]

noooo my argument

[tepid nova]

https://tenor.com/view/sml-black-yoshi-oooh-you-broke-it-man-you-broke-it-broken-gif-26098263

[still garnet]

i counter your argument with another: should exec have an env param? right now there's no way to e.g. run a one-off command with env but not persist that env to the container config

i guess you could just run env, FOO=1, ...

just thinking back, pretty sure I've seen RUN FOO=1 ... in Dockerfiles, but maybe it had env in front

[tepid nova]

I don't think it needs to, personally. Given the nature of chaining, if you want a one-off env for one command in the pipeline but not the others - you just branch off the pipeline before setting that env and calling that exec

[still garnet]

but if you want to use the result of that exec to export/publish, you'll need to unset the env first

well, don't need to, but it might just be cruft

[tepid nova]

I was seeing something like:

base := client.Container().From("alpine")
cmd1 := base.WithEnv("DEBUG", "1").Exec(...)
pipeline := base.Exec().Exec().Exec()...

(in this example cmd1 is "branching off")

[still garnet]

right, but take this example: base.WithEnv("CGO_ENABLED", "0").Exec("go", "install", "./cmd/go-releaser").Publish("vito/goreleaser")

that image will have CGO_ENABLED=0 set not only for the single build command, but all future build commands - so the branching off doesn't help if the purpose of the branch is to build+publish the image

the fix there would be to add WithoutEnv("CGO_ENABLED") before the Publish

[tepid nova]

yeah, same problem as with dockerfile I guess

[still garnet]

yup - the same workaround in Dockerfiles is at least possible for us

[tepid nova]

in dockerfile you’d solve that with a multi-stage build & copy probably

[still garnet]

oh, yeah true

or do base.Exec("env", "CGO_ENABLED=0", "go", "install", "./cmd/go-releaser").Publish("vito/goreleaser")

[tepid nova]

I think the problem of "I don't want to inherit build-time env variables at runtime" generalizes well to "I don't want to inherit build-time <stuff> at runtime", which is what multi-stage builds solve

[still garnet]

so in your multi-pipeline example you could do something like base.WithCopiedFile(cmd1.File("./go-releaser"), "/usr/bin/go-releaser"))

(not quite that succinct since you need an ID, but yeah)

[tepid nova]

Hopefully soon the ID part can be hidden 🙂

@still garnet do you think new host api is enough to solve monorepo juggling, or need something more fundamental?

I guess doing everything at monorepo root + include/exclude should improve things

[still garnet]

host { directory("foo") } will sync the foo subdir individually, do they need anything more than that?

you'd want to avoid workdir.directory("foo") since that'd still require the full workdir sync

[tepid nova]

@wet mason is it an option to split query building from client? Wondering if it would make for cooler reusable functions that just spit out the query without having to pass a client? but maybe too much complication and not worth it?

[tepid nova]

yeah I think so. definitely simpler to grok.

assuming I can pass include/exclude, that’s clearly a must have for big repos

[still garnet]

I think host { workdir(include:["foo"]).directory("foo") } and host { directory("foo") } should be roughly the same, for example

and you'd be able to do host { directory("foo", exclude: ["node_modules"]) } too

i'll rebase my include/exclude branch on this PR soon, but there's some bikeshedding to do: https://github.com/dagger/dagger/pull/3560#discussion_r1006067948

[wet mason]

Kinda, yeah. We could provide a way to spit out the underlying query. Client and query building do however become intermixed when interpreting the result back (as in, before codegen, you had to build out a struct that matched the response format, whereas codegen takes care of that automatically by "unpacking" fields)

[wet mason]

that'd be awesome. hitting that problem with dogfooding (e.g. .git busting the cache)

[wet mason]

also awesome to optimize

re: markdown linter mage target, I could just include *.md files, which means the entire linting step will be cached if changes are made to non-md files

(replaces the need to use github action filters, and works locally)

[still garnet]

nice, makes me want to have paths like foo/*.md "just work" in Bass (i.e. expand to include: ["*.md"])

[civic yacht]

@wet mason @still garnet @tepid nova PTAL at comment here on the multiplatform PR and let me know if you have any thoughts: https://github.com/dagger/dagger/pull/3119#issuecomment-1292860015

It's pretty light on detail for sake of time (see first paragraph), so also feel free to let me know if it makes no sense and needs more details. I'm going to start working on the implementation concurrently though

[civic yacht]

BTW, please don't block merging the PR on that, that's one of those suggestions where I'm throwing it out in case everyone is enthusiastically agrees, but if it's meh or controversial I don't care, just leave it as is 🙂

[still garnet]

i think it's worth considering, since otherwise there's no proposal for container.import (maybe someone has a use for that?)

though i'm not sure if buildkit even makes it possible to load from an OCI image tarball (I don't think so)

could always have the API layer double as a registry

anyway normally I wouldn't block, it'd just be nice to figure that out rather than have folks migrate to a new host API twice

[civic yacht]

I believe it is now (not in official release yet): https://github.com/moby/buildkit/pull/2827

[still garnet]

omg

i can delete so much code

[still garnet]

oh right - I have a use for that

[ornate rover]

doh, just did a go get -u and now hit:

 github.com/moby/buildkit/session/filesync
../../../golang/pkg/mod/github.com/moby/buildkit@v0.10.5/session/filesync/filesync.go:112:20: cannot use dir.Map (variable of type func(string, *types.Stat) bool) as type fsutil.MapFunc in struct literal

Which I had yesterday as well. darn - I knew I should have git init'd this test.

And fixed - and commited. Now lets see what actually broke it.

oh - I see @civic yacht mentioning a moby update above - I should just wait maybe - no need to update really anyway yet 🙂

[civic yacht]

Yeah go get -u will update all transitive dependencies too, which can cause breakage like that if there's a dep that doesn't have official releases such as fsutil. We need to update our buildkit dependency in the very near future, which will take care of this very particular problem for now at least.

If you do happen to run with -u, you can patch it after the fact with go get github.com/tonistiigi/fsutil@v0.0.0-20220115021204-b19f7f9cb274.

Like I said this is most likely going to go away in the very near future but it'll take probably 5 minutes to add a blurb to our docs in the meantime so I'll just do that quickly too in case it continues being a problem.

EDIT: https://github.com/dagger/dagger/pull/3575

[civic yacht]

i can delete so much code

[hasty basin]

@ornate rover are you in the Community Call?

We're going to show your demo 🙂

Looked like you were online.

[wild zephyr]

@wet mason @civic yacht @still garnet wouldn't just vendoring github.com/docker/docker inside dagger would allow us to forget about the replace for the moment? Why would we need to maintain different forks?

[civic yacht]

I don't remember enough about how vendor/ works to say right away, we'd need it so that anyone depending on our go SDK also picks up all transitive dependencies from our vendor/ dir I think

Probably worth a shot though

[still garnet]

@wild zephyr i think we talked about that before, the problem is vendor only solves the fetching part, not the version override part, which comes before that

[ornate rover]

@hasty basin Yus! Good call/conversation.

[wild zephyr]

not sure I follow Alex, since we can vendor whatever commit we want, why wouldn't the version be fixed?

buildkit currenly does this: (https://github.com/moby/buildkit/tree/master/vendor) it vendors specific modules at specific versions

[still garnet]

right but even with them vendoring it, we still need our replace, so I don't think that fixes it

[wild zephyr]

true..now that I'm thinking about it, it's probably because the proxy doesn't use /vendor folders

[still garnet]

yeah, i think it's because the two things are decoupled. replace is version A => version B, vendor/ is version => source code