#daggernauts

1 messages · Page 2 of 1

sharp zealot
#

Thanks for the zenith mod walkthrough! Quick question, the go mod init and go mod replace steps feel like they could/should be wrapped into dagger mod, so that I don't need to install any native Go tooling on my machine 🙂

thorn moat
# sharp zealot Thanks for the zenith mod walkthrough! Quick question, the `go mod init` and `go...

go mod replace definitely needs to go, and yeah go mod init could maybe be automated for you; it makes sense for your Dagger Go code to have its own module as a sane default, so you don't pollute your project's dependencies with Dagger dependencies. I just had to fix that recently in Booklit: https://github.com/vito/booklit/issues/55

Aside from that though, I think it's a safe assumption that you'll have the Go tooling around if you're planning to write Go code, at least until we wrap the full dev environment (like doing dev out of dagger shell).

#

In general we need to figure out how to handle dependencies of the generated code, which is what the go mod replace thing is revealing. Same is true of github.com/Khan/genqlient/graphql, it just happens to be a stable published one so it works, but we're not pinning its version it at all.

sharp zealot
#

What's the story behind the go mod replace? I see the comment but don't fully understand it

#

Side quest: I'm going to try taking all those go mod commands and shove them into dagger mod init

#

In a very clean and elegant way (just kidding it will be fugly)

sharp zealot
thorn moat
thorn moat
#

fyi: starting on a somewhat hefty merge now that https://github.com/dagger/dagger/pull/5764 has landed. Going with a merge instead of a rebase for now, don't want to hide away botched merge conflict resolutions in the history.

wintry prism
#

@warped canyon not sure if you already have one, but next step for me is a "main" module with two "utility" module depedendecies. One in github and one in gitlab! 😉

bleak nest
#

Trying out a module, need some help

thorn moat
#

Do we have a preference on what to call module repos? I'm writing a Git utility module (theoretically Daggerverse itself will use it, but right now it's just so I can test it at all). Should we have a convention like gitutil-dagger? dagger-gitutil? dag-gitutil? If we're using repo URIs as our canonical name for modules, length will matter

sharp zealot
#

A low-hanging fruit for the first module-a-thon (better name needed?). I think it would be helpful to ship a binary release ahead of time, so that writing a module doesn't require building zenith, or even checking out the dagger repo at all.

warped canyon
#

dagger mod extend 🧵

sharp zealot
#

pro-tip for testing your module: dagger query supports a --doc flag

#

@warped canyon I'm trying to extend the container type but getting weird errors when querying my module

#

Got my first module to work!

#

Feature request: dagger --mod <URL> so that I can do dagger -m git://.... query without having to git clone anything

#

It extends the Container core type with a ttlpush function, which publishes it to ttl.sh, a throaway registry without any credentials required

#

Example query:

query test {
        container {
                from(address: "alpine") {
                        ttlpush(repo: "solotest", tag: "latest")
                }
        }
}
#
  • Development time: 30mn, including first setup.
  • Line count: 20 lines (will be 15 soon)
sharp zealot
#

@thorn moat since bug reports and feature requests are starting to pile up, how do you recommend we go about organizing them? We could just do github, but feels a bit heavyweight given the pace. Linear is closed... Maybe a special discord forum for zenith help? wdyt?

thorn moat
wintry prism
sharp zealot
#

ha ha 🙂

#

by the way @wintry prism you can do dagger query --doc test.gql test

wintry prism
#

what's the difference? @sharp zealot

#

oh, does that create the test.gql?

sharp zealot
#

The graphql document can contain multiple named queries. Then you can specify the one you want to run (in this case: "test")

#

I think when there's only one query in the doc, you can omit the name. So: dagger query --doc test.gql

wintry prism
wintry prism
wintry prism
#

that works! 🤯

sharp zealot
#

oh wow

#

missed it in the usage output

wintry prism
thorn moat
#

Should we .gitignore dagger.gen.go?

thorn moat
#

NodeJS isn't happy with us using the word Function 😅

/sdk/nodejs/sdk/nodejs/api/client.gen.ts
  3851:30  error  Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
sharp zealot
#

Fn? DaggerFunction?

thorn moat
#

Func is probably fine too, I think that's mostly a Go thing and it's OK with it as long as it's capitalized

#

Fun 😄

wintry prism
thorn moat
#

i am extremely impressed

#

bikeshed - I'm renaming dagger mod extend to dagger mod use but lemme know if anyone disagrees. use feels a lot more obvious to me; extend feels like a holdover from the 'extensions' terminology. Also down for import or something but it sounds a bit dry, and possibly has more language keyword conflicts.

sharp zealot
#

I agree with ditching extend. i vote for whatever word scores most familiarity points in focus groups 😁

thorn moat
#

works for me!

sharp zealot
#

use is fine with me

thorn moat
#

PSA: I've gone ahead and changed module refs to look like Go module refs: github.com/foo/bar/subpath@version. dagger mod use now accepts any git ref as the version and will resolve it to a commit before placing it in dagger.json. There are many reasons this might not be the final format we land on, but for now it looks pretty and at least leans on an existing familiar syntax.

Some open questions:

  • Right now it only supports github.com/. This is because it's impossible to know where the repo URL ends and the subpath begins. Most repo URLs are username/reponame, but some of them can be arbitrarily nested (e.g. GitLab). Go went to great lengths to solve this problem, but it works by requiring the vendor to co-operate. (If you view the source of https://github.com/vito/booklit you'll see a special <meta name="go-import" ...> tag)
  • Do we want to support HTTP source code tarball module references or any other format? Or do we want to be "pure git"? There's kind of a beauty in making this open-ended, since anything that can produce a dagger.Directory can be a valid module ref, and theoretically Daggerverse can proxy any of that just as easily as it can proxy Git.
  • Right now all the Git stuff happens by running the alpine/git image in as part of dagger mod use, which is a hard dependency on an external party. We should rein that in somehow.

Challenges not unique to this format that might be nice to resolve:

  • It's impossible to know whether the user has specified a commit, a tag, or a branch. Right now it just blindly resolves whatever ref you gave it to a commit, but it would be nice to preserve tags. I suppose it could parse semver, with an optional prefix, to detect that. Have to keep in mind the difference between github.com/kpenfound/daggerverse/foo/bar@v1.2.3 and github.com/kpenfound/daggerverse/foo/bar@foo/bar/v1.2.3 though (assuming path-prefixed tags in a monorepo scheme like the one we use in dagger/dagger for SDKs).
thorn moat
#

Bikeshedding module ref URLs

warped canyon
#

@thorn moat I have dagger mod sync running infinitely on latest right now trying to codegen github.com/kpenfound/dagger-modules/golang

warped canyon
#

in my greetings-api project I have the zenith module as a subdirectory of the project it does ci for. With the earlier build of zenith I would use root: "../". Is that still the way today, or is there another way to load the parent dir when I call Host().Directory(".")

sharp zealot
warped canyon
#

so my two modules are working together nicely now (until I have to codegen again)

  • github.com/kpenfound/dagger-modules/golang: generic golang module to provide a base image, build/test/lint helpers
  • github.com/kpenfound/greetings-api/ci: my greetings-api ci module. Combines the golang module with my project specific settings
Screen_Shot_2023-09-14_at_2.01.52_PM.png
sharp zealot
#

Nice, I will try using one of your modules in one of mine today, see what happens

sharp zealot
#

@warped canyon Are you OK to run the module dev "workshop" tomorrow? I'm thinking there are a few details that are worth preparing today, like pre-building the zenith binary and adding that to the README so we all save setup time tomorrow. I'm available to work on it with you this afternoon. Should free up dev time for @thorn moat

sharp zealot
#

Any requests for a zenith module? 🙂

#

I'm making a basic "dagger-dev" one now

#

@warped canyon once I get a basic standalone version to work, I'll try swapping in your golang module

sharp zealot
#

I'm stuck on a cryptic error while developing my 2nd module, if anyone is around to help me out?

sharp zealot
#

Here's a fun one: I'm trying to daggerize ./hack/make engine:build (a dagger function to build dagger from source), but that command itself wraps dagger... 😛

warped canyon
sharp zealot
#

At some point I'll hand you the mic and we'll all follow your instructions.

#

If those are "checkout the repo, follow the readme, raise your hand if you're stuck" that's perfectly fine

#

OK, I might have bitten off more than I can chew with this dagger-dev module...

warped canyon
#

Cool. If we're providing a zenith binary (and an engine image??) we should be able to get people started without checking anything out, and just run 100% from scratch

warped canyon
sharp zealot
#

It's OK, I can use it as my own project for the workshop tomorrow 🙂

sharp zealot
warped canyon
#

the main thing I'd update is instructions to create a module in your own repo rather than as a subdirectory in zenith, and maybe touch on dagger mod use with the golang module?

sharp zealot
sharp zealot
#

Maybe also mention the cold fusion thing? 🙂

sharp zealot
#

OK nevermind. It does successfully run dagger-in-dagger, but then I get weird errors about missing files

warped canyon
sharp zealot
#

So is my module, to be honest 😄

#

Let me share that with you

#

(would be cool if somehow you could go from a dagger cloud run page, to the daggerverse link of the corresponding module... but I digress 🙂

#

Oh wait let me try and share a copy-pasteable query 🙂

warped canyon
#

ohh I see

sharp zealot
# 
echo '{ dagger_dev { source(version: "0.8.4") { daggerHack(version: "0.8.4", args: ["docs:lint"]) { id } } } }' | dagger query -m 'git://github.com/shykes/daggerverse?ref=main&subpath=dagger-dev'
#

uuuuuh.... @thorn moat 🙂

#

Am I allowed to share the link to my module on your RIDICULOUS daggerverse demo?

#

Or would that spoil the surprise?

#

because it works FYI

thorn moat
#

eh go for it

sharp zealot
thorn moat
#

whee our little index is growing

sharp zealot
#

In case it's easy to scrape: I added comments to the module type, to describe the module itself

thorn moat
#

the web UI supposedly grabs the module description and shows it, but I didn't know where it would be even coming from. Module struct type makes sense! I'm guessing it's just not being collected by the codegen yet

sharp zealot
#

@wintry prism right now 👆

warped canyon
#

@thorn moat on latest should I be using

git://github.com/shykes/daggerverse?ref=main&subpath=dagger-dev

or

github.com/shykes/daggerverse/dagger-dev

with -m

thorn moat
#

the latter

#

tried that though and it still doesn't work

warped canyon
#

I tried daggerDev too, no luck

wintry prism
#

It's funny, I ran ./hack/dev on the zenith-functions branch an got a running dagger-engine.dev container, but my dagger CLI is broken and quite a bit smaller than if I build the binary with go build 🤔

image.png
#

the one I built with go build actually runs fine, but he other one spits out

image.png
warped canyon
wintry prism
#

It's like the go build is getting cut off mid stream in the ./hack/dev case

#

or something

warped canyon
#

oh I didn't catch that

#

would make sense

#

also if you're using goenv, that is temporarily unsupported

deft rain
#

Out of curiosity, does the zenith call get recorded? I can't make it unfortunately, but would love to catch-up later 🙂

wintry prism
#

I think audio is pretty solved on Discord, but video...

#

Everyone says, get an extra machine with OBS istalled 😆

deft rain
#

Oh yeah I suppose it's not super trivial 🤔 yeah I don't have any particular knowledge of it...

thorn moat
#

found the issue with -m <ref>, fixing!

wintry prism
thorn moat
#

@warped canyon @sharp zealot pushed, this should ... well it doesn't work, but it does something now: echo '{ dagger_dev { source(version: "0.8.4") { daggerHack(version: "0.8.4", args: ["docs:lint"]) { id } } } }' | dagger query -m github.com/shykes/daggerverse/dagger-dev

warped canyon
#

Yup, now I can see the same thing cloud showed

sharp zealot
warped canyon
#

/tmp/buildkit-mount2577139352/.markdownlint.yaml: no such file or directory

sharp zealot
#

Yup that's the same one I get

#

But separately, can't get dagger query -m github.com/shykes/daggerverse/dagger-dev to work (getting the 'Cannot query field' error)

warped canyon
#

here's my full command with dagger built from the latest commit echo '{ dagger_dev { source(version: "0.8.4") { daggerHack(version: "0.8.4", args: ["docs:lint"]) { id } } } }' | dagger query -m 'github.com/shykes/daggerverse/dagger-dev'

thorn moat
sharp zealot
#

Yeah, I pulled + ran hack/dev but might have messed up. Can dagger version give me the build commit or something?

warped canyon
# 
dagger version
dagger devel (b39ac77b8) darwin/arm64
# 
which dagger
/Users/kylepenfound/scratch/zenith/bin/dagger
thorn moat
#

interesting, I get dagger devel () linux/amd64, maybe because that's the ./hack/dev version?

sharp zealot
#

same

warped canyon
#

oh right, I'm doing my own go build

sharp zealot
#

I think maybe it's my module build logic that causes the error

thorn moat
#

quick sidebar: dagger_dev should probably be normalized to daggerDev right? not sure where that isn't happening. is what what you set as the module name?

#

i noticed it also accepts foo-bar which just detonates the GraphQL schema

sharp zealot
#

Basically I'm trying to decouple our magefile code from the target source code. Which is annoyingly hard to do

warped canyon
#

I'm curious about the multiple git clones, since source in DaggerHack isn't used

sharp zealot
#

they're both used normally

#

I decouple the magefile version from the dagger source version. So you can build version X with magefile version Y

warped canyon
#

ah I see, so you'd probably want env := hackEnv(version, source)

sharp zealot
#

My first approach was to go build internal/mage/ into a binary, but go won't allow it because it's a non-main package

sharp zealot
#

So now I'm trying to mount both in the same container, and go run one against the other. But I may be in a situation where my workdir needs to be two things at once

warped canyon
#

this would be super messy, but you could have

sourceA(blahOpts{ Include: []string{"internal/mage"})
sourceB(blahOpts{ Exclude: []string{"internal/mage"})

and drop them on top of eachother

#

and then less reliant on mage to do the workdir thing properly

thorn moat
#

doing some fly deploys

warped canyon
warped canyon
sharp zealot
#

You know I feel like it's easier to shift gears, instead of wrapping hack/make etc, I might just implement a simple go build for now, and gradually move the low-level functions right into the module

#

basically port hack/make to zenith, instead of wrapping it

sharp zealot
#

I kind of want to take arguments to my module’s top-level field… is that possible?

#

eg. { dagger(version: “0.8.5”) { cli { export(path: “./dagger”) } } }

thorn moat
#

failing that, you could always add a constructor + type of your own

sharp zealot
thorn moat
warped canyon
#

just extend it right off of Query 🧌

sharp zealot
thorn moat
#

i tried it once on mobile and the module page looked pretty bad too

thorn moat
#

i mean, theoretically...

#

we just don't have a Query type (it's fused with Client)

sharp zealot
sharp zealot
# thorn moat you made me go check if that was possible 😂

I think we should consider making that the default behavior of the top-level struct (merge it into query instead of namespacing it). We’ll need to handle conflicts, but we already need to do that anyway with core type extensions

wdyt @warped canyon ?

#

I can already tell I’m going to want that kind of control

warped canyon
thorn moat
whole maple
#

is there any way to use dagger as service?

sharp zealot
whole maple
# sharp zealot We are developing a cloud service fittingly called Dagger Cloud. May I ask what ...

it's simple...we dont want to setup docker for 10 machine and docker swarm on it to connet and drive devops workflow to build, test and deploy code...we just need dagger as service to directly connect it and use it...this is one of the use case...and which can be auto scaled also...and we should be able to connect our machines with it also to save money like github self hosted runner...this can be win for startups...

#

we just need to connect machines with that service and rest of the things will be handled by dagger service to use mac machine's CPU & GPU

ember walrus
#

That makes a lot of sense, @whole maple. While we don't have a hosted service for the compute, we do integrate with compute hosted by CIs, such as runners managed by GitLab, GitHub, CircleCI, and others. Is that something that could work for you?

obtuse lion
#

sorry, I'm a complete idiot when it comes to discord. Is 9am call happening right now somehow, somewhere?

sharp zealot
#

Starting a thread about how type extensions, since that was a big topic in the call today

ember walrus
thorn moat
#

@sharp zealot btw, I don't know if you noticed this "fine print" easter egg 😁 - it's handy if you want to load a module up into the Apollo explorer, I used it for debugging things while building Daggerverse

image.png
sharp zealot
#

@thorn moat resuming zenith hacking for my last few hours before the week-end...

#

to answer your earlier question on dagger query -m: I am on the latest version of the branch (I think)

thorn moat
#

do ya have a command I can try to run?

sharp zealot
#

echo '{host{directory(path:"."){goTest(args:["./..."]) {stdout }}}}' | dagger query -m 'github.com/kpenfound/dagger-modules/golang'

#

What's the easiest way to setup a zenith nightly build?

wintry prism
#

as a starting point...taking a look

sharp zealot
#

I mean... if it could be using a dagger module, that would feel extra chef_kiss

wintry prism
#

noticing Mage syntax

func (Dagger) Publish(ctx context.Context, version string) error {
    err := Engine{}.Publish(ctx, version)
    if err != nil {
        return err
    }

    err = Cli{}.Publish(ctx, version)
#

related to module namespacing

sharp zealot
#

Have a great week-end everyone. Fantastic progress this week, I am very excited for what we'll build next week 🙂

#

🙂

muted plank
#

Where is the list of dagger types like Container that the gen can generate ?

sharp zealot
#

I think all core types? 🤷🏻‍♂️

thorn moat
#

@muted plank Yeah I don't know of any limitation, one bug though is that you have to refer to any 'extended' types from your module's entrypoint type, otherwise the codegen won't pick it up. (It sounds confusing because it is confusing. 😅)

So a workaround is like this:

type MyModule struct {}

// this won't be picked up on its own
func (m *Container) MyContainerMethod(ctx context.Context) (string, error) {
  return "xyz", nil
}

// workaround - note the Container return type:
func (m *MyModule) Foo(ctx context.Context) (*Container, error) {
  return nil, fmt.Errorf("ignore me")
}
muted plank
#

I see. I was wondering what are types that cannot be translated to some kind of "CLI syntax". I realize that graphQL must have the same problem. So any types that is not graphql serializable must be a problem. The obvious examples i am thinking of are "chan", or io.Reader, etc. these were types libchan could serialize in the old days (and I have been punting on a libchan v2 using HTTP3 forever but that's another story). Note that I cannot think of a good usecase now.

spiral whale
#

🎉 just pushed the first module https://github.com/aweris/daggerverse/blob/main/gh. echo '{gh{run(version: "v2.34.0", args:["repo", "view", "dagger/dagger", "--json", "id,name,nameWithOwner,url,defaultBranchRef"], token: "'"$GITHUB_TOKEN"'")}}' | dagger query . I needed this while working on gale module.

thorn moat
spiral whale
#

Sure, I'll make some update later today, than I can add daggerverse

wintry prism
#

Working on getting this one in shape. Similar to what you built @spiral whale in that it installs a CLI, the AWS Copilot CLI for deploying to ECS (not sure how many folks use it, but looked interesting).
https://github.com/jpadams/dagger-module-awscopilot

The thing is, once you actually run it, it wants to do a docker build of a Dockefile and such to deploy to ECS...I hacked this together really quick last night and was following a getting started example for inspiration, but maybe I can deploy an image from a registry vs doing a dind build (which I couldn't quite get to work). Will take a look in the light of day 😄

I used passed AWS creds as env vars like this to my test function as plain strings that I then turnded into secrets. Better way?

echo "{awscopilot{test(awsid: \"$AWS_ACCESS_KEY_ID\", awssecret: \"$AWS_SECRET_ACCESS_KEY\", awsregion: \"us-west-2\")}}" | dagger query

func (m *Awscopilot) Test(ctx context.Context, awsid string, awssecret string, awsregion string) (string, error) {
...
WithSecretVariable("AWS_ACCESS_KEY_ID", dag.SetSecret("awsid", awsid)).
        WithSecretVariable("AWS_SECRET_ACCESS_KEY", dag.SetSecret("awssecret", awssecret)).
        WithEnvVariable("AWS_REGION", awsregion).
...
warped canyon
#

Working on getting this one in shape

wintry prism
#

@thorn moat We weren't able to use *Secret for our secrets and send them through the dagger cli, so both Ali and I used string and then set the secret within. Any idea how hard it would be to enable secrets from the start?

I also ran into wanting content from a directory outside of my project root.

I also ran into trouble trying to run a dind service container (might be doing that wrong).

Also wanted dagger shell a lot to try my new CLI in a container with my creds and project loaded 🙂

sharp zealot
#

Otherwise high probability it will get lost in the noise

rustic river
#

is there a page explaining what zenith is? or a github repo or something with an explaintion of what the project is about?

#

I found this video, from the first few mins it seems like the opposite of what I was asking about: https://www.youtube.com/watch?v=Pt_AsbQilAM. I want to write an internal tool with the SDK not a DSL. I just want to use it for generic cli tool use cases not just CI/CD

YouTube
Dagger
Introduction to Project Zenith

In this demo, Solomon Hykes, Erik Sipsma, and Alex Suraci introduce Project Zenith. They discuss the concept of reusable environments and demonstrate its implementation in real-world scenarios. With Project Zenith, developers can leverage standardized entrypoints, rich data-driven environments, and language-agnostic APIs, enabling seamless colla...

▶ Play video
sharp zealot
#

Hi @rustic river , i answered in that forum post. Short version is that Zenith is relevant to your use case, but it is not a requirement. It will just be an additional improvement when it ships. Embedding Dagger in your custom CLI is supported today and will continue to be supported after Zenith ships

sharp zealot
wintry prism
#

echo "{container{from(address: \"alpine\"){trivyScan}}}" | dagger query -m github.com/jpadams/dagger-module-trivy

wintry prism
#

@spiral whale leveled up! daggerfire

image.png
sharp zealot
#

“modulator”? 😁

#

“guardian of the daggerverse”?

wintry prism
#

By the many swords of Zenith!

sharp zealot
# 
echo '{ dagger { engine(version: "0.8.7") { cli(operatingSystem: "darwin", arch: "amd64") { export(path: "/tmp/dagger-darwin") } } } }'  | dagger query -m github.com/shykes/daggerverse/dagger
#

🙂

#

If you have a zenith build of dagger, this should work out of the box

#

My dream would be to add enough to this module, to be able to build and release a working binary release of zenith, using zenith - and for everyone here to be able to get the latest version of zenith, using zenith 😛

deft rain
#

So I've been playing about making a module - I found a kind of interesting thing with overriding native types to allow chaining.
It's kinda weird to pass parameters to the overridden methods of core types - e.g. if I want to pick a version of the software (e.g. a version of hugo, for the module I'm working on).
I guess the way to do it might be something like the Container.DockerBuild implementation (https://pkg.go.dev/dagger.io/dagger#Directory.DockerBuild), and pass some config struct to each call in that module. But it does feel a bit bulky if you were to call multiple module functions in a chain - e.g. HugoBuild, HugoLint, you'd have to pass that same config everywhere.

The chaining feels so nice though. Without it, you'd create a HugoBuilder object, and then use that, which gives you a nice place to chuck various configuration parameters. Obviously this breaks chaining though.

Out of curiosity - would it be possible to do something similar to With to allow keeping chaining without needing to directly modify methods on Container? You could potentially have something like this: https://gist.github.com/jedevc/ec3e37b0afffa472f8a7570fd6ef139e (I have no idea how the typing of something like this would shake out though). As a nice bonus, you then don't end up having potential clashes when extending core types.

sharp zealot
#

As a user though, I still like regular chaining better 🙂

#

@deft rain in your example, Sample embeds Directory. If that's the crucial part, should we consider one day making struct embed a feature of our type system?

#

eg. dir.AsSample(version: "latest").DoThing(param: "foo").WithExec([]string{"echo", "foo"}).Stdout(ctx) ?

deft rain
#

Ah, that wasn't really the essential part to me - it was more about wondering how we could do the typing of something like that, e.g. how does the implementation of With in my example know that Sample can be chained onto Directory properly (as opposed to Container, or any other core type)

Buuut, I really like that example. Seems to solve scoping and parameterisation as well.

sharp zealot
#

Yeah I appreciate the practicality of With as a stopgap, but compared to chaining, it just feels harder - like I have to translate it in my head, whereas chaining just makes sense immediately

#

With feels like "meta-chaining" if that makes sense

deft rain
#

Agreed, I think I prefer the more explicit struct embedding you suggested, I think that's a better way of trying to work out my pre-coffee module problems.

agile saddle
# sharp zealot ```console echo '{ dagger { engine(version: "0.8.7") { cli(operatingSystem: "dar...

I was really excited for a moment here, but this is just using Go’s built-in cross compilation support. Unfortunately the “real deal” is probably contrary to Apple’s terms of service.

https://github.com/shykes/daggerverse/blob/adbe451b8307402153183224a26ffd02ad795d0b/dagger/main.go#L52C5-L52C5

GitHub
daggerverse/dagger/main.go at adbe451b8307402153183224a26ffd02ad795...

My personal collection of Dagger modules. Contribute to shykes/daggerverse development by creating an account on GitHub.

thorn moat
thorn moat
#

Update: as of the latest push (https://github.com/dagger/dagger/pull/5757/commits/73aff3c57263a505f5548d51a1f3e25614bf612b) functions no longer require a ctx argument and error return value.
So these should all work:

  • func() X <- return a value with no error, like Container
  • func() <- weird to do, but still works, returns Void
  • func() error <- less weird (side effectful), returns Void
  • func() (X, error) <- same as today but without ctx
  • func(context.Context) error <- etc...
thorn moat
#

Update as of the latest push https

warped canyon
#

I forgot I ran this 😂

Screen_Shot_2023-09-19_at_3.44.57_PM.png
#

somehow got back on goenv

sharp zealot
sharp zealot
thorn moat
#

somehow got back on goenv

thorn moat
sharp zealot
#

I passed on to @jedevc the advice you gave me which is: just push to the branch as-is, with a heads up, long-running dev branches are an exception (and we should try to merge soon-ish)

wintry prism
#

You're right @sharp zealot, your helloWorld is :chef's kiss:

{
┃   │ │     "helloWorld": {
┃   │ │         "withGreeting": {
┃   │ │             "withName": {
┃   │ │                 "message": "you're the best!, jeremy!"
┃   │ │             }
┃   │ │         }
┃   │ │     }
┃   │ │ }
sharp zealot
wintry prism
#

Some "genera" or "families" of modules are starting to emerge.

Uses container image with CLI already inside. Provides interface to that CLI.
closely related to
Installs a CLI in a container. Provides interface to that CLI.
optionally using
Takes in one or more secrets to drive a CLI or API calls.

etc

#

I figure if we have really solid examples for the major kinds, it will be easy for folks to make more.

#

Mixing and matching best practices

#

which we're still discovering

sharp zealot
#

@thorn moat I'm trying to make my 'dagger' module into something actually useful... I started lifting chunks of internal/mage but there's so much... The most useful thing I can think of, is getting us to a binary release of zenith itself... in other words building & publishing a CLI+engine container combo. Does that feel feasible to you? Any pointers on where to look in the dagger repo? cc @chrome pilot

thorn moat
sharp zealot
#

OK maybe not that long

spiral whale
#

finally, the initial gale module is up (https://github.com/aweris/daggerverse/blob/main/gale). It needs to work on module and gale site but it's good enough for now. 🎉

export GITHUB_TOKEN=$(gh auth token)

echo '{
    gale {
        repo(name: "aweris/gale") {
            withWorkflowsDir(dir: "ci/workflows"){
                list(token: "'"$GITHUB_TOKEN"'")
                
                workflow(name: "ci/workflows/lint.yaml") {
                    withJob(name: "golangci-lint") {
                        withDebug{
                            run(token: "'"$GITHUB_TOKEN"'")
                        }
                    }
                }
            }
        }
    }
}' | dagger query -m github.com/aweris/daggerverse/gale@main
deft rain
sharp zealot
#

@kind carbon I’ll be home in 10mn, want to catch up on zenith before your day ends?

#

@thorn moat @chrome pilot @deft rain you’re welcome to join of course

kind carbon
thorn moat
#

Will join in a moment! Coming back from errand

#

back

sharp zealot
#

joining now

wintry prism
#

@deft rain leveled up!! daggerfire

image.png
sharp zealot
#

@mossy hazel 🙂

wintry prism
mossy hazel
sharp zealot
#

Feature request: dagger query -q for a quiet mode, that prints only the result of my query. It can get pretty hard to spot with all the TUI output

thorn moat
#

used to be the default, but found it was hiding errors too often, needs fine-tuning

sharp zealot
#

if everything disappeared at the end, it would be OK - I think that's how it worked before?

#

can someone try this?

echo '{myip{ip}}' | dagger query -m github.com/shykes/daggerverse/myip
wintry prism
#

I have an "empty" dir called sandbox with just a .envrc in it to run these 🙂

# 
{
┃   │     "myip": {
┃   │         "ip": "97.115.90.190\n"
┃   │     }
┃   │ }
#

@sharp zealot 👆

sharp zealot
#

yay 🙂

wintry prism
sharp zealot
#

Imagine once we have dagger do 🙂

#

dagger do -m github.com/shykes/daggerverse/myip ip ?

#

maybe do is not the right verb

wintry prism
#

maybe just dagger -m like python -m

sharp zealot
#

coukd be dagger run MODULE FUNCTION ...

#

dagger run github.com/shykes/daggerverse/myip ip

wintry prism
# 
dagger trust github.com/shykes/daggerverse
dagger run myip ip
dagger run helloworld ...
#

if no conflicts can make shorter

sharp zealot
#

this would not work well with single-module repos

wintry prism
sharp zealot
#

But I would love to find a way to shorten things eventually

#

I think for now having the URL in there every time is fine, if we get something consistent and reliable

wintry prism
#

shorter and well-trusted seem important, so we don't get a curl | bash type problem with untrusted content

sharp zealot
#

shorter and well-trusted are separate problems though

# 
curl --trust hack.ru
curl index.html

--> shorter but doesn't actually help with trust

mossy hazel
wintry prism
#

When you publish after that PR is merged, you'll get a new entry in the daggerverse, right?

sharp zealot
#

Nice 🙂 thanks, merged

wintry prism
#

All by commit

sharp zealot
#

Yes correct

#

re-publishing now

#

@thorn moat I think soon we'll be ready for a "latest events" feed, and a separate list of modules

wintry prism
#

oooh the README.md has gotten nicer over the last two releases

#

(of that module)

sharp zealot
#

Slowly but surely, frequently updated modules are drowning out "older" modules

#

Oh wait daggerverse renders the README? love it

wintry prism
#

@mossy hazel has joined the modulators! daggerfire

image.png
thorn moat
#

There's a whole separate rabbit hole around how to identify newer/older modules since they're all git commits, which aren't statically comparable. Would be nice if the main page just showed the latest version of each. I have an idea, but it'll take some schema tweaks

wintry prism
thorn moat
#

The README will probably compete with docs attached to the module's entrypoint type, once that's implemented. Could make sense to keep both, since the README often has higher-level info

sharp zealot
#

@thorn moat BTW my function comments don't show in daggerverse module docs anymore

wintry prism
#

@thorn moat where would I implement dropping a README.md file during dagger mod init?

sharp zealot
#

Feature request @mossy hazel any chance you could take the python code as a File ?

#

that would remove 3 lines of error checking from my code 🙂

# 
func (m *Myip) IP(ctx context.Context) (string, error) {
    code, err := m.Code().Contents(ctx)
    if err != nil {
        return code, err
    }
    return dag.InlinePython().WithPackage("requests").Code(code).Stdout(ctx)
}

func (m *Myip) Code() *File {
    return dag.Host().File("myip.py")
}
mossy hazel
#

I guess we can simply add a function with a different signature and support both

thorn moat
thorn moat
#

also that ip function is a good example of something we'll want cache busting for!

sharp zealot
sharp zealot
#

but how?

wintry prism
#

I think I found it: ./cmd/dagger/module.go

sharp zealot
#

I can’t inject cache busting in @mossy hazel ‘s function without forking it

thorn moat
# sharp zealot but how?

putting on my v2 IDs hat, I think we'll need a way to declare resolvers/functions as 'impure' / 'tainted', and those ones won't be cached, and any objects returned by them will be 'tainted'. not 100% sure though. Erik mentioned something similar (some way to mark a function as 'no cache')

sharp zealot
thorn moat
sharp zealot
#

Re: dagger do vs other verb. What if there were several verbs to make a query, and each verb corresponds to a UI paradigm?

--> dagger shell [-m MOD] FUNCTION [opts] ...: function must return a container, spawn an interactive shell

--> dagger print [-m MOD] FUNCTION [opts]: call function and simply print the output, with various formatting options. Minimize other output (TUI etc)

--> dagger download [-m MOD] FUNCTION [opts]: call function and download the result to the local filesystem - might be a directory, file, container...

--> dagger do dagger run [-m MOD] FUNCTION [opts]... call function and focus on what it actually does: stream logs, emphasize errors etc. (btw I agree dagger run would be better than do for this...)

thorn moat
#

👍 - that's roughly what I had in mind too. What's nice is the same function can have different verbs applied to it. Container can be debugged (shell) or exported (download) etc.

sharp zealot
#

yeah feels elegant to me

#

btw shell could work with Directory by mounting it into a default container 🙂

#

I’m going to call it now, this is a Docker moment in the making

#

fyi @chrome pilot 👆

sharp zealot
# 
Error: failed to run codegen: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleMainSrc: json: unsupported value: encountered a cycle via *dagger.TypeDefInput

😭

thorn moat
sharp zealot
#

it might actually be a genuine cycle

#

or at least something stupid on my end, made hard to debug by confusing codegen error

#

This happened in the middle of splitting one custom type into two, so there might be some entanglement

#

@thorn moat I ripped off parts of your ci/ directory hope you don't mind

#

Trying to get us closer to a zenith module that can ship zenith itself

thorn moat
#

Erik wrote all that, but I'm sure he doesn't mind 😛

sharp zealot
#

@thorn moat what do you think will be faster:

  1. Setting up a bootleg release from the branch
  2. Merging xrc2 with an experimental flag and doing a real release
thorn moat
#

I think 1. There's still some things up in the air (extending core types, some schema changes), and at least 1 is guaranteed dogfooding experience

sharp zealot
#

Bootleg it is

kind carbon
#

@thorn moat, I'm looking at the new TypeDef (input) way of adding the functions. I think you mentioned something yesterday that you want to refactor something around this. Can you clarify what's your backlog of core changes? I'm looking especially for anything that would affect the Python implementation.

thorn moat
#

refactoring away from complex input types

wintry prism
#

For all the folks that were hounding me in DMs about this 😉

dagger query -m github.com/jpadams/dagger-module-trivy << EOF
{ container { from(address: "alpine") { trivyScan }}}
EOF
#

We use Trivy (via a GitHub action) to scan in our CI today, so I'll be excited to get it into our future version that uses your Dagger module @sharp zealot 🙌 (once the features are more fleshed out. Will ping the nice folks at Aqua to see if they can direct me or want to lead on that 🙂 )

hoary geyser
#

@warped canyon , that demo was awesome. I have Argo mostly working at https://argocd.inlets.tutes.ai but I think I have to edit my configmap because of the TLS redirects....

#

testing in production

grizzled basin
#

Look forward to watching the video when available! Darn these 4am streams 🙂

sharp zealot
#

@grizzled basin may I ask where in the world you're based?

warped canyon
wind coyote
#

A dagger module for Vault would be super cool

grizzled basin
sharp zealot
#

@thorn moat just in case you're there, we're trying to pick the safest route through the release spaghetti 🙂

#
  • Option 1: do a "regular" release to the real registry.dagger.io (probably involves pushing a semver tag to the main repo?)
  • Option 2: parametrize the engine registry, and push a release on a different registry (may require forking the code to make registry.dagger.io parametrizable
  • Option 3: somehow ship the output of ./hack/dev
thorn moat
#

I can hop on in a bit, walking the 🐶

sharp zealot
plucky ermine
grizzled basin
#

Nice!

sharp zealot
#

We successfully shipped a working zenith build with zenith! Awesome collab session with @wintry prism @ember walrus @thorn moat and special guest @latent trellis ❤️

latent trellis
#

Do you plan to publish binaries and the image somewhere?

plucky ermine
sharp zealot
hoary geyser
#

I'm excited about inline python. I think @mossy hazel is my new best friend.

sharp zealot
hoary geyser
#

One thing I haven't seen discussed here is Gerrit/Zuul-CI it seems to me that dagger would be killer for toolchains like that, where people typically do trunk-based development on projects with cross-repo dependencies.
Does anyone know what I'm talking about?

sharp zealot
grizzled basin
#

@hoary geyser speaking of Gerrit - when learning Go years ago I wrote a simple, specialised Gerrit CI tool - just follows the even stream and then builds the (Apache Maven) projects it sees - I've since changed that to switch that to using Dagger to build the maven project, using a hard-coded/standard Dagger pipeline which is all inside the CI tool - which, hopefully with Zenith - I can just replace that with "detech if theres dagger, run it"

#

@hoary geyser have been meaning to setup a simple local gerrit-in-docker instance to use to make a small demo video, I just havn't had the time yet.

hoary geyser
grizzled basin
# hoary geyser Interesting. I was thinking particularly about the code review aspect of Gerrit,...

That's what I'm now using it for - we used to have jenkins doing builds, but we moved all our stuff to Azure Devops and kinda lost the gerrit connection/builds - so adapted my old project to use gerrit - really solves one of the issues I had with poisoned repos, sort of. Using the persisent cache is wonders. https://github.com/talios/gowest is my repo - tho I've not pushed the dagger stuff yet

GitHub
GitHub - talios/gowest: Experiment Golang based Gerrit watching tool

Experiment Golang based Gerrit watching tool. Contribute to talios/gowest development by creating an account on GitHub.

#

I watch for the patchset updates, or new revs and clone, then call the pipeline build - really need to add some docs to that repo. It really was something to learn Go with.

hoary geyser
sharp zealot
#

getting a bunch of errors on the latest build... 😭

thorn moat
#

You'll also need to dagger mod sync everywhere

sharp zealot
#

actually at least one of the problems seems to be introduced by the last commit

#

(but another appears when I rollback by 1-2 commits)

#

On the latest commit I get unknown object against the latest shykes/daggerverse/dagger

#

dagger mod sync returns:

Error: failed to run codegen: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleMainSrc: expected named type, got *types.Slice
#

(I guess that one is the real error 👆 )

kind carbon
# 
from typing import Annotated

from dagger.ext import function


@function
def hello(name: Annotated[str, "The name of the person to say hello to."]) -> str:
    """Say hello to someone."""
    return f"Hello, {name}!"

$ echo '{ helloWorld { hello(name: "from Python") }}' | dagger query -m zenith/hello --focus
✔ query [1.20s]
┃ {
┃     "helloWorld": {
┃         "hello": "Hello, from Python!"
┃     }
┃ }
WARNING: Using development engine; skipping version compatibility check.
WARNING: Using development engine; skipping version compatibility check.
• Engine: afebe977b8ed (version devel ())
⧗ 2.52s ✔ 22 ∅ 7
Screenshot_2023-09-22_at_13.52.54.png
thorn moat
sharp zealot
wintry prism
# thorn moat fix pushed!

hmmm....still getting errors with latest on

echo '{container{from(address: "alpine"){trivyScan}}}' | dagger query -m github.com/jpadams/dagger-module-trivy
# 
▽ init
│ █ [0.21s] connect
│ ┣ [0.00s] starting engine
│ ┣ [0.21s] starting session
│ ┃ !!! STARTING SESSION
│ ┃ !!! STARTING SESSION sha256:94a50fccc5bd0d0f980dd7657c6545315605c52e5f175a77ea53656381
│ ┃ 41426e
│ ┃ Failed to connect; retrying... name:"error" value:"make request: Post \"http://dagger/
│ ┃ query\": rpc error: code = Unknown desc = failed to verify client: client ID \"a418ku2
│ ┃ mhv2j9ts8it2a06i5m\" not registered"
│ ┃ OK!

[1.96s] ERROR dagger query -m github.com/jpadams/dagger-module-trivy
┣ [1.96s] loading module
█ [0.44s] ERROR exec /runtime
┃ unknown object

Error: failed to load module: failed to install module: input:1: git.commit.tree.directory.asModule failed to call module to get functions: failed to get function output directory: process "/runtime" did not complete successfully: exit code: 2
sharp zealot
wintry prism
#

oh, but I need to release that new mod sync'd to daggerverse

#

let me try again locally

#

I get more errors locally including a dump on the schema or something that I've seen in the past.

thorn moat
wintry prism
thorn moat
#

taking a look!

#

can repro locally, likely to do with extending core types

wintry prism
thorn moat
#

it's possible for now, still debating the topic

wintry prism
#

The chaining feels nice for sure. If I didn't extend Container with the trivyScan, then I'd just make a scan function that took the container as an arg?

thorn moat
#

yep!

#

@wintry prism fix pushed

sharp zealot
#

Personally I think a convenience for converting core types to custom specialized types (Container.AsX) would replace the need for full blown monkeypatching

#

also makes discovery easier. as<autocomplete>

deft rain
#

When building a module, i was curious - has anyone thought about writing tests? Writing ci pipeline tests would be so cool imo (especially for writing reusable components).

thorn moat
#

@sharp zealot also make sure you pull again, there was another bug affecting shykes/dagger 😛

#

speaking of tests, it would be nice if every module had a test entrypoint, so I could just try running every module in the daggerverse as I'm making changes

#

I can do part of that now, but knowing what to run as a smoke-test requires module-specific knowledge

deft rain
thorn moat
wintry prism
deft rain
thorn moat
warped canyon
thorn moat
#

I'm operating under the assumption that all developers are good and kind

#

and if they're not, no one should use their modules 😠

#

(I need caffeine...)

deft rain
#

@thorn moat you're much more trusting than me 😄

thorn moat
#

there's a layer of sarcasm there. 😛 My only point is that there's a balance to strike between what's enforced by patterns/convention and what's enforced by distrustful mechanisms

#

sometimes the arrow does go in the other direction though, e.g. if the mechanism prefixes all types/methods with the module name, the convention would become that people do short module names like Hugo or Go

wintry prism
#

yeah that pattern seemed intuitive for

deft rain
#

I briefly did an experiment where a

thorn moat
kind carbon
#

@thorn moat, heads up, I'm going to push py mvp support.

thorn moat
#

looking forward to having more than one language in our cross-language module ecosystem 🤩

spiral whale
kind carbon
#

No codegen yet. Just tested a simple function, will add more support as I go along.

#

No codegen means that it won't pick up new stuff from dependencies. You'd have to reach for the lower lever query builder for that.

#

It also means there's no dagger mod sync pre-req after changes.

#

I'm going to try publishing a simple python module to daggerverse to act as a template for others also.

thorn moat
#

it'll probably just fail until then

kind carbon
#

Oh, ok 🙂 I'll prep it to test locally in the meantime.

#

mkdir daggerverse 😁

kind carbon
warped canyon
#

it's not quite published yet, but this is what I had in mind for general secrets interface:

func (v *Vault) Auth *Vault
func (v *Vault) GetSecret (string, error)
func (v *Vault) PutSecret (*Vault, error)
func (c *Container) WithVaultSecret (*Container, error)

(same in the AWS Secrets Manager module)

latent trellis
#

People seem to put their dagger modules into a single daggerverse repo at this point. Is that considered the best practice right now? Or it's just until Zenith is released?

I'd imagine testing and releasing modules is a bit more challenging this way.

warped canyon
wintry prism
latent trellis
wintry prism
wintry prism
thorn moat
#

@sharp zealot re: running codegen on-the-fly... One option: when ExperimentalPrivilegedNesting is true, we also mount a dagger CLI in to the container at /bin/dagger (bikeshed) . That way the runtime can literally just run dagger mod sync before building. Have we thought about something like this before? Having an automatically guaranteed-compatible dagger binary around for nesting seems like it could be useful, but I don't have a concrete in mind. The other option I was considering was to add codegen logic to the shim, which is doable but will take some elbow grease / hackery to tell the shim to do its thing.

warped canyon
#

Definitely seems useful in general. Like if you can use registry.dagger.io/dagger:v0.9.0 as your CI executor in the future

#

short of having a cli-only image

#

i'm sure there's some other nice things that come with the combo

thorn moat
#

Oh you mean just having the CLI image available in the engine image?

warped canyon
#

yeah, is that not what you were saying 😅

#

oh nvm I think I get it

thorn moat
#

Different topic, it's a prerequisite though. I'm talking about having Dagger automatically mount the CLI in for any WIthExecs that enable netsing

warped canyon
#

Yup, that also seems useful ➕

#

(invisible heavy plus sign emoji)

thorn moat
#

visible on light mode 😄

#

I think we even used to have the dagger CLI baked into the image and download it from the container in the SDK, but maybe we stopped doing that because it'd be awkward supporting all the different host-side platforms

#

But in this case it just has to be the same platform as the image

sharp zealot
#

Since eventually it will be SDK-specific anyway

#

Re: shipping dagger binary:

  • Love the idea of shipping a dagger binary by default, in general
  • In the context of using that for codegen: assume that CLI syntax will change (dagger mod sync might not be a valid command in the future) so not sure if you want to depend on that CLI syntax for lower-level features
thorn moat
thorn moat
sharp zealot
#

I find it helpful to imagine everything Go-specific (or Python-specific etc) eventually bundled together in a module - ie. SDK-as-module

thorn moat
#

I could do the dagger mod sync thing as a shortcut for now (it took approx. 1 minute to implement), and we can move it into the runtime once we figure out what that looks like? Just trying to get Daggerverse updated ASAP without losing all the past stuff

#

("once we figure out what that looks like" could mean Monday)

#

It works! Just ran dagger mod use github.com/shykes/daggerverse/datetime@d16ecd28b133b2b0b0a8ed866fa0bc196fcc7427 🎉 - transitive dependencies also worked. Nice to confirm we can move this responsibility to the runtime.

sharp zealot
sharp zealot
thorn moat
sharp zealot
thorn moat
#

done

kind carbon
plucky ermine
#

Did we reach a general recommendation from this morning for how to handle the case where caching swallows the output after first run for things like Solomons python datetime demo?

sharp zealot
#

For example, for my datetime module, instead of calling code(code: "import datetime; blbla") I would want to call code(cache: false, code: "import datetime; ")

#

PS that is a feature request @mossy hazel 😛

#

Under the hood, the way you disable cache is that you pass an input to WithExec (usually an environment variable) that changes each time you want caching disabled. We call that pattern "cache busting"

#

If you want caching to be disabled every time, you pass a random value

#

If you want it to be disabled once a week, you pass the current time, truncated to the beginning of the week

#

That's the pattern @thorn moat was alluding to

#

Then eventually we'll figure out how to incorporate this into the core API (but that will take more time)

plucky ermine
sharp zealot
thorn moat
sharp zealot
#

Done with my meetings for the day, I get 20mn of Daggerverse fun before school pickup 🙂

#

@warped canyon I removed ./bin before running ./hack/dev, didn't run into the "killed" issue so far

#

(this is on the latest commit)

sharp zealot
#

My mistake

#

So more like:

code(code: "....", cacheKey: "XXXX")

thorn moat
#

Yep. Tempting to put cache key generation into a module but ... it'd be cached

#

I guess you could always give it the current time + an interval to truncate

sharp zealot
#

Does this mean there's no way for a function to decide that its own exec will never be cached?

#

I think I now understand what @kind carbon was saying 🙂

thorn moat
#

Yeah - that's why we need some sort of explicit opt-out

sharp zealot
#

(sorry for pinging you helder, please go back to bed 😛

#

You can sleep soundly knowing that you were right, as always

#

@thorn moat what if we added a core random call?

#

Then engine could trace back any dependencies on those values, and invalidate cache

#

... but if it's my function calling that random value dynamically, I guess it's still the same problem - I'll never get the chance to call it, I get cached before that

#

@thorn moat I have the latest commit, and dagger mod init still creates dagger.gen.go and querybuilder/, normal?

thorn moat
#

yeah - I haven't changed codegen yet, and it might still generate all that anyway just for local editing

#

so the main difference would be just adding them to a .gitignore

mossy hazel
#

then I don't see a workaround currently for bypassing the cache given functions are cached, maybe we could implement a "noCache" like in my example, but as a builtin, that bypasses the caching of the function and inject a llb.IgnoreCache (not sure of the name) so the cache would be disabled for all underlying operations

sharp zealot
#

Better than the alternative which is for me to inject a random comment in my python code 😛

mossy hazel
#

yes but good luck setting a random key from gql

sharp zealot
#

yeah, I personally don't care about that because I'm calling your module from another module 🙂 But you're right that's not great

#

Ooooohhhh my own module will be cached too....

#

So I have to take a random cache key also don't I.... all the way back to the graphql client ...

mossy hazel
#

at least it would work from the client side (not from another module), as soon as we have the query builder from native languages

#

btw, what if functions were not cached at all (by design)? The module output: go binary binary (or python pyc files) can be cached, but why the run? I think it causes more issues that it should. I guess it'd be slightly slower because of the grpc roundtrips, but given the buidlkit ops remain cached, I guess it would be more than acceptable without the side effects.

thorn moat
#

to cache or not to cache [by default], that is the question

warped canyon
#

What if the cache key is a Secret thinkies

hoary geyser
#

I was thinking about this morning's discussion of the Python SDK size wrt inline Python
There is a tiny Python runtime called Micropython which is popular in the IoT scene. Well, the PyScript project has recently looked into it because they run Python in the browser:
https://pyscript.net/tech-preview/micropython/about.html
I wonder if this may be a tactic for "bootstrapping" the python runtime for modules?
(possibly @mossy hazel or @kind carbon have already considered this...)

Another angle is Nuitka
https://www.nuitka.net/
the gentleman's python compiler.

Nuitka the Python Compiler — Nuitka the Python Compiler documentat...

With the Python compiler Nuitka you create protected binaries out of your Python source code.

hoary geyser
grizzled basin
#

How are modules intended to work/plugin with existing pipelines? watching the recent demo I didn't really see that mentioned, I suppose that falls into the dagger mod get calls? which somehow installs the module into my local pipeline project?

kind carbon
kind carbon
# grizzled basin How are modules intended to work/plugin with existing pipelines? watching the r...

I think the idea is that everything becomes a module. so you'd migrate your existing pipelines to a module, or multiple ones. If you want to mix anyway, you just need to start the Dagger Engine session with dagger listen but you'll still need to run codegen to get the extended API from the module you're depending on in the SDK's client, otherwise you'll need to drop to lower level API (GraphQL) calls. Using the underlying query builder is easier.

But you make a good point, we probably need to make this interop easy for folks that already have big pipelines and need to transition/experiment more gradually. However, it's good to remember that the existing pipeline code should remain mostly the same, just needs to be wrapped in a module function, while removing some boilerplate that's no longer needed for bootstrapping.

thorn moat
sharp zealot
#

@ember walrus I figured out why that Dagger Cloud visualization looked funny on the Zenith runs. It was a dagger listen session, and we made a bunch of runs from the browser, all in the same session 🙂

ember walrus
#

If we want to trigger runs or re-runs from a GUI, we have to figure out that flow for visualization based on one session. That is, what is the event?

grizzled basin
wintry prism
#

@kind carbon has joined the league! daggerfire

image.png
sharp zealot
#

@ocean escarp 👋😁

ocean escarp
#

Hello everyone !

#

I just cloned the Zenith version of Dagger, I'll try to port some FluentCI pipelines (which are mainly written in TypeScript) into a Dagger module, but what if I want to write a module in Javascript/Typescript?

sharp zealot
kind carbon
#

The reliance on code gen somewhat

drowsy basin
#

I wanted to play with the zenith build but my company man-in-the-middle proxy striked again 😭

█ [0.58s] ERROR exec go build -o /runtime -ldflags -s -d -w .
┃ go: downloading github.com/Khan/genqlient v0.6.0
┃ go: downloading github.com/99designs/gqlgen v0.17.31
┃ go: downloading golang.org/x/sync v0.3.0
┃ dagger.gen.go:15:2: github.com/Khan/genqlient@v0.6.0: Get "https://proxy.golang.org/github.com/%21khan/genqlie
┃ nt/@v/v0.6.0.zip": tls: failed to verify certificate: x509: certificate signed by unknown authority
┃ internal/querybuilder/marshal.go:11:2: github.com/99designs/gqlgen@v0.17.31: Get "https://proxy.golang.org/git
┃ hub.com/99designs/gqlgen/@v/v0.17.31.zip": tls: failed to verify certificate: x509: certificate signed by unkn
┃ own authority
┃ internal/querybuilder/marshal.go:12:2: golang.org/x/sync@v0.3.0: Get "https://proxy.golang.org/golang.org/x/sy
┃ nc/@v/v0.3.0.zip": tls: failed to verify certificate: x509: certificate signed by unknown authority
┻
#

I already has to manually patch internal/mage/util/util.go in order to handle that additional root CA in the ./hack/make commands

sharp zealot
drowsy basin
#

my patch (internal/mage/util/util.go) is quite ugly but does the job when building the CLI, engine and SDKs

func goBase(c *dagger.Client) *dagger.Container {
    repo := RepositoryGoCodeOnly(c)

    customCA := `-----BEGIN CERTIFICATE-----
MIIE0zCCA7ugAwIBAgIJANu+mC2Jt3uTMA0GCSqGSIb3DQEBCwUAMIGhMQswCQYD
/// ...
pB5iDj2mUZH1T8lzYtuZy0ZPirxmtsk3135+CKNa2OCAhhFjE0xd
-----END CERTIFICATE-----`

    return c.Container().
        From(fmt.Sprintf("golang:%s-alpine%s", golangVersion, alpineVersion)).
        // ...
        WithNewFile("/etc/ssl/certs/customCA.pem", dagger.ContainerWithNewFileOpts{
            Contents: customCA,
        }).
        WithExec([]string{"update-ca-certificates"}).
        // ...
}
modern fossil
#

@sharp zealot suggested that I start by writing a Zenith module before trying my hand at an SDK. So I'd like to create a module that builds and deploys a NextJS app to kubernetes (possibly using a k8s module that already exists?) I already have a working Dagger pipeline(s) for this.

Are there some links to docs or videos about how to get up and running?

kind carbon
#

@thorn moat, are you working on putting the codegen in the runtime yet?

thorn moat
#

@kind carbon yeah, I'm spiking on module-izing the Go SDK at the moment, it's going well but there are definitely fun bootstrapping problems to sort out

#

let me push what I have to a branch

#

I'm trying to suss out whether we can land this pre-merge, because right now the on-the-fly dagger mod sync is super hacky (I've had to add a bogus cache-buster)

kind carbon
#

Yeah, that approach seems ideal, but I've been wondering if it's too soon (without a stable release) because of a possible chicken & egg situation.

#

Was just about to ask about how you connect with mod sync for example 🙂

thorn moat
#

by 'that approach' you mean on-the-fly codegen, or modular-izing SDKs?

kind carbon
#

Putting it in a module

thorn moat
#

yeah, the nice thing is we at least have a comfortable way out: once you bootstrap a pre-built image with the /runtime entrypoint, theoretically you just push that to registry.dagger.io, and have sdk: go mean registry.dagger.io/sdk/go or something

kind carbon
#

Because that gives you the higher level client. The current runtimes use the core functions which are a bit harder to work with.

thorn moat
#

I ended up going down this route because otherwise it's really hard to get just the codegen code and the SDK code into goRuntime(); they're split across the codebase, and in separate Go modules

kind carbon
#

And there's stuff like progSock and buildkit client that I don't know how to get from generator.Generate

thorn moat
#

that's how the current hacky dagger mod sync-in-goruntime approach works, I think this modularized flow can work the same way but not sure yet

kind carbon
#

I'm going to take a moment to analyze your changes and see how it can work for Python. Brb with questions 🙂

thorn moat
sharp zealot
#

"regular old dagger-in-dagger" made me laugh. 10 years later Docker-in-docker is still a nightmare. We basically just solved nesting on day one.

sharp zealot
#

Anything fun cooking in the daggerverse today? Me I'm trying to polish my new supergit module, hoping to help @vivid hatch and @visual blaze solve their git LFS problem with that

#

Also would love to get that nightly build going, so we can simplify the README (download a binary instead of building from source)

vivid hatch
sharp zealot
#

I also would love a volunteer to work on dagger do, I think we could ship a simple version VERY EASILY, there is no need to introspect the graphql schema - just build a query

sharp zealot
#

I think the function caching issue may be more urgent than I initially thought...

#

FYI @thorn moat @chrome pilot @kind carbon 👆 I know you guys are busy with other things, just want to make sure it's on your radar.

thorn moat
#

@kind carbon Just successfully bootstrapped, tests all pass. 🤯 Now to kick the tires and clean things up / make sense of how this all works... I'll push the code as-is if you want to see how it works in the meantime.

sharp zealot
#

This is SDK-as-module?

thorn moat
#

yup

kind carbon
sharp zealot
thorn moat
latent trellis
latent trellis
#

The Directory type extension doesn't seem to work for some reason :/

latent trellis
#

I'm probably missing something obvious, but I give up at this point. :/

I'll give it a try again tomorrow.

sharp zealot
#

@latent trellis what's the symptom? I can try to reproduce

latent trellis
#

Error: make request: input:5: Cannot query field "bats" on type "Directory".

#

I reference the Directory type as an argument and a return type

#

How do you test modules anyway? I just checked out my repo into the zenith directory 😄

sharp zealot
#

You have a few options

#
  1. echo <GRAPHQLQUERY> | dagger query [-m MODULE]
#
  1. dagger query [-m MODULE] --doc PATH/TO/GRAPHQLDOC.gql
#
  1. dagger listen [-m MODULE] --allow-cors then point a graphql web client at it 🙂 I'm looking at your module from an interactive graphql sandbox right now
#

dagger listen --focus=false --allow-cors -m github.com/sagikazarmark/daggerverse/bats

latent trellis
#

That does seem a bit cumbersome while you are developing modules. It means you have to push every change you want to test

sharp zealot
#

Oh no you can omit the -m in development

#

defaults to current directory

latent trellis
#

But then you need to manually set the env vars from .envrc?

sharp zealot
#

Personally my setup is:

  1. Initial setup:
  • Develop all my modules in ~/dev/shykes/daggerverse`
  • I have the special "use zenith version of dagger" envrc file in ~/dev/shykes/daggerverse/.envrc`
  1. Module dev loop:
  • CD to my module of choice, write code, run dagger query or dagger listen
  • rinse repeat
#

exploring your module

Screen_Shot_2023-09-26_at_5.19.15_PM.png
latent trellis
#

Makes sense

#

Does this work for you as well?

sharp zealot
#

getting the same thing as you

latent trellis
# 
package main

import (
    "context"
)

type Bats struct{}

func (m *Bats) Bats(ctx context.Context, dir *Directory, args []string) (*Container, error) {
    return run(dir, args), nil
}

func (m *Bats) Foo(ctx context.Context) (*Directory, error) {
    return nil, nil
}

func (dir *Directory) Bats(ctx context.Context, args []string) (*Container, error) {
    return run(dir, args), nil
}

func run(dir *Directory, args []string) *Container {
    return image().
        WithMountedDirectory("/src", dir).
        WithWorkdir("/src").
        WithExec(args)
}

func image() *Container {
    return dag.
        Container().
        From("bats/bats:v1.10.0")
}
#

Here is the code

sharp zealot
#

HEADS UP, I pushed a stopgap to disable excessive memoization of function calls, by injecting a cachebuster at runtime exec. It seems to work, and it's a very small patch, but I won't claim to fully understand the codebase I patched, so it's possible that I broke something horribly.

When you read this, could you please try pulling the latest version, and testing it?

Thank you!

The commit for reference: https://github.com/shykes/dagger/commit/d07fbb4a35de48350715a2c16c2b97fe2a9a03fb

GitHub
disable crippling memoization · shykes/dagger@d07fbb4

Signed-off-by: Solomon Hykes

kind carbon
kind carbon
modern fossil
#

Will Go SDK modules always require the package to be main?

sharp zealot
modern fossil
#

It seems like the introspection, regardless of the package name, requires that we put our Dagger files in a dedicated directory? Perhaps this is the same across all SDK modules?

modern fossil
#

Today, I have makefiles and Dockerfiles intermixed with my code, does this mean I need to have a dedicated directory going forward?

warped canyon
#

disable crippling memoization · shykes/d...

sharp zealot
thorn moat
#

This is very much an ad-hoc internal API at the moment, lots of bikeshedding to do. But the capability is there now. This replaces the hacky implementation of just-in-time codegen from before, where the hardcoded Go runtime ran a bind-mounted dagger mod sync, which became a problem because its cache never busted. Now the SDK's runtime just runs its own codegen.

#

You'll start seeing "sdkRuntime": "vito/dagger-sdk-go:multi@sha256:..." show up in your dagger.json files, since this works by having modules pin their runtime image at code generation time. The idea is this would eventually point to something on registry.dagger.io. 😅 You can change it to whatever you want, just be sure to remove the "sdk" field, since all it does is tell the CLI to sync to the current ref for that well-known name.

sharp zealot
#

@thorn moat how should we change our mod dev workflows?

thorn moat
sharp zealot
thorn moat
#

Right now they're both there, so future mod syncs can bump the sdkRuntime to whatever the CLI points to for the configured sdk ("go")

sharp zealot
#

Thinking about cache controls...

#

At first I was thinking: what if a function could have a companion "cache helper" function which takes the same arguments, and returns a string? Just before the engine calls the real function, it first calls the cache helper (always uncached), receives a fresh cache key, and adds it to the real function exec. "Always cache" can be implemented as a function that returns a constant cache key; "Never cache" helper returns a random value; "TTL" helper returns current time truncated in various ways; etc. Pretty neat.

#

But then I thought: gosh how will this cache helper function be registered? And writing two functions with exactly the same arguments seems repetitive and error-prone. How do I get this awesome pattern without making the DX worse?

#

And now I am thinking: what if instead of a helper function, we added a core function dag.CacheMe(key string) which any function could call at any time. Calling CacheMe results in the calling function being terminated (insert buildkit magic handwaving here), then called again with the same arguments, but this time with caching enabled, using the specified cache key as a cache buster....

#

This would actually solve a major problem that has plagued Dockerfiles since the dawn of time: caching of linux packages

#

For example, installing a Wolfi Linux package:

query {
 wolfi {
  container {
   withPackage(name: "git") {
      ...
   }
  }
 }
}

The withPackage function could:

  1. start by querying the wolfi repositories for the latest digest (or if not available, version number) of the specified package.
  2. call CacheMe(packageDigest)
  3. do the expensive work of downloading and installing the package

The example query would result in WithPackage being called twice in a row:

  • First call, with random cache key: at step 2, CacheMe() terminates the function
  • Second call, with package digest as cache key: either we get a cache hit (yay!) or we go all the way: CacheMe() does not terminate the function, step 3 does the work. Next time we will get a hit!

Boom, distro-aware caching with minimal plumbing

#

This could also solve the caching issues which have plagued deployment functions:

  • Do I always cache, and break rollback deployments? ("you have already deployed this version of the app in the past, so I will never deploy it again!")
  • Or do I never cache, and add 3-30 seconds to every run?

Now we have an answer, if the deployment target has a way to query the current deployed state. I believe most modern services that you can send artifacts to, have a way of telling you the version or digest of what they've received. Assuming that, the deployment function can use the same pattern to query the deployment digest, call CacheMe(deploymentDigest) then continue to the usual deployment logic.

#

Functions that want to set their own TTL to 24h: dag.CacheMeTTL(1000 * 60 * 24) or dag.CacheMe(CacheMeOpts{TTL: 1000 * 60 * 24}) (just a convenience)

latent trellis
#

Okay, bats module works, do I get that module builder badge? 😄

#

Also, a question related to that same module: how do I "extend" a container encapsulated by a module?

In this specific scenario, the bats module uses a prebuilt image to run bats (a shell based testing framework). However, there may be dependencies I need to install into the image (eg. using apk or mounting binaries).

What's the best way to do that?

I could build my own API using With* style functions, but that would lead to duplicating parts of the Container API.

Any thoughts?

#

Ideally, I need to "inject" something between the final exec and the initialization of the container:

func (m *Bats) Bats(ctx context.Context, d *Directory, args []string) (*Container, error) {
    ctr := dag.
        Container().
        From("bats/bats:v1.10.0")

    // mount and install binaries here...from outside of the module?

    ctr.Exec([]string{"bats"})
}
kind carbon
#

@thorn moat, have you tried making a function call via the API? With Function.call?

thorn moat
kind carbon
#

Yes. There's no code using it. ParentName, Parent and Module are said to be internal, but when called via the API they're empty which makes the module register the functions instead of invoking one.

thorn moat
#

ha right - I kept getting a core.Module ID back instead of what I wanted

#

yeah, I think it just needs to be updated now that there are multiple objects exposed by a module, rather than a flat set of functions

#

ParentName shouldn't be needed at the API level, since the function already comes from an object type, so that should be known already

#

but you'll probably need a way to pass the Parent I suppose?

#

Module is probably also internal optimization, since it already has that (in ID form)

#

not sure if this helps you at all but I ended up just running a GraphQL query instead, which worked out much more nicely

#

oh is this for dagger do?

kind carbon
#

Yep 🙂 re: dagger do

#

Maybe core.Function should have a field for the parent object?

thorn moat
#

I think it makes sense to pass it at call time, in the spirit of the receiving object just being a special kind of argument

kind carbon
#

I meant when you register with dag.TypeDef().WithObject("MyModule").WithFunction("MyFunction"), a core.Function instance should know it's a part of the MyModule type.

thorn moat
#

oh, +1

#

assumed it did already, but yeah likely not

sharp zealot
ocean escarp
wintry prism
latent trellis
thorn moat
#

fyi: I just got rid of WARNING: Using development engine; skipping version compatibility check. - it was starting to swarm the output. Disabled the call internally in a couple of places where it's guaranteed to be compatible (engine dialing itself), and then switched it to debug level for the remaining spots. Doesn't seem worth the noise with all the nesting.

thorn moat
#

The remaining calls are during on-the-fly codegen, where it makes sense to do a compatibilty check (SDK runtime could be out of date if you're not using an auto-synced one)

kind carbon
#

Function.call

latent trellis
#

I’m going to give the With* pattern a try

sharp zealot
latent trellis
#

Yeah makes sense. I was looking for a way to add “extensions” to an existing container, but that’s probably more work than it’s worth.

#

Creating a new container from an existing base image is two lines of code

sharp zealot
#

the nice thing is that you can choose your own adventure 😀

latent trellis
#

Yeah well I hope patterns will emerge at some point to help adoption in the community.

#

Sometimes I need to remind myself that Zenith is not even released yet but I’m already talking about patterns and adoption 😁😁😁

thorn moat
#

@sharp zealot before I forget (flying out tonight btw ✈️): I think we'll need to namespace types provided by modules. Something simple like ModuleName_TypeName should be fine, and impossible to fool since we normalize on CamelCase. Right now it's super natural to define a type like Foo thinking it lives purely within your module, but really it's just as global as Container. (There's an interesting follow-up that perhaps that should become ModuleName_Container, which ties back to wrapping, which ties back to whether we support monkey-patching, etc.)

thorn moat
#

@warped canyon btw, as a side effect of SDKs-as-modules you shouldn't be having any go issues anymore, if you want to go back to goenv

#

now the only go that ever runs is the one baked into the module

sharp zealot
#

@thorn moat humble request: please read my cache control monologue so you can noodle on it on the flight 🙏😇

#

re-read it this morning and still love it

#

really hoping it’s not fatally flawed

thorn moat
#

will do! I skimmed it this morning but haven't fully processed it yet, trying to put some finishing touches on the sdk-modules stuff before I take off. It's a red-eye flight though so no promises 😛

sharp zealot
#

just think about it in your dreams

thorn moat
#

the gang solves cache invalidation

sharp zealot
#

What if we shipped a stable release quickly (as in within the next few days) that only adds a hidden command:

dagger experimental zenith [-o PATH] [--arch amd64|arm64] [--os darwin|linux|windows] [--GIT_BRANCH] [--remote GIT_REMOTE]

Usage: Download a build of the "zenith" development branch of Dagger. This branch features a cross-language component system, new APIs, and new command-line UX. Learn more at ...

In practice we just need to hardcode a dagger pipeline that builds zenith.

This might be easier than maintaining a semi-official release infrastructure. Also less work for the infrastructure crew. We can piggy back on the infra we have.

latent trellis
#

My two cents

  • If I demo zenith, it should be something easy for people to try. It took me some time to learn, but cloning a repo and running a custom build does not count as easy for a lot of people. Downloading a single binary and running a single command is the most effort a lot of people will do.
  • Demonstrating something that already runs in CI (even if a PR) projects way more confidence and stability than a custom build I run locally.

So as soon as there is any kind of nightly build, I'm gonna switch all my dagger pipelines to Zenith.

#

I don't even care if it breaks from time to time: if I can pin to a single commit/version, I'm happy.

latent trellis
sharp zealot
#

@latent trellis obviously we want to get this merged (behind an experimental flag) as soon as possible, so even a nightly build won't be necessary. We're just looking for short term stopgaps until then - probably 2 weeks max before we merge IMO

#

(ok maybe more because of upcoming travel)

#

or maybe less because of ambient awesomeness

latent trellis
#

Well, obviously timing depends on a lot of factors.

I just wanted to share some outside perspective. Like I'm demoing Dagger the week after next and if I tell people to run more than 2 commands, probably most of them won't. But if zenith is that close to being released, it probably doesn't make sense to start releasing nightly versions.

sharp zealot
#

@thorn moat I'm getting error on dagger mod sync with latest commit

#

-cvs flag is missing

thorn moat
#

Oh oops

#

Need to fix the go sdk module, I removed that flag

#

I can fix once I'm waiting at the gate if you don't wanna try to solve the puzzle 😛

sharp zealot
#

I'm also getting a super weird error where an apparently straightforward type somehow doesn't make it to the graphql schema? But doesn't seem related to the last commit

#

@thorn moat it's no rush, I'm winding down for now anyway

#

Do you have any hunch as to why this would not make it into the gql schema?



type RemoteTag struct {
    Name   string `json:"name"`
    Commit string `json:"commit"`
}
#

I tried changing the name to 10 different hingd

thorn moat
#

Might need to define a function on it

#

If you liked it then you should have put a function on it

sharp zealot
#

oh right!!!!

stoic shoalBOT
#

GG @sharp zealot, you just advanced to level 2!

sharp zealot
stoic shoalBOT
thorn moat
#

@sharp zealot fixed the --vcs thing

wintry prism
#

Zenith builds, courtesy of Zenith zenith

ocean escarp
#

What do you think of this Dagger module structure (for Deno/NodeJS)?

If I've understood correctly, a Dagger module exports functions that will be imported and executed in Dagger by a runtime.

So my idea for Deno/NodeJS Dagger modules is to also write a GraphQL schema (whose resolvers are the module's functions) using https://nexusjs.org/, which will also be imported by the runtime.

In this way, the runtime will execute the module's GraphQL schema, which will then execute the module's functions.

latent trellis
kind carbon
# ocean escarp What do you think of this Dagger module structure (for Deno/NodeJS)? If I've un...

It’s true that in an earlier iteration of API “extensions”, SDK’s used to save the module’s GraphQL schema in a file in the runtime container, but in Zenith this evolved into simply the SDK querying special client API fields (should be used only by SDKs) to:

  • Check if it needs o register functions or invoke one (registration == empty function name);
  • Register functions, including arguments that they support and return types;
  • Get the context for a function call (parent value and arguments);
  • Return results back;
    So the API server simply executes the runtime’s entrypoint at appropriate times, but it’s these API fields that extend the overall API schema server-side when they’re called.
wintry prism
#

@ocean escarp 👋

#

Not sure if you can share your Deno /NodeJS progress with us on the call 🙏

ocean escarp
sharp zealot
#

@kind carbon wait dagger call works? ❤️

#

just saw your update

kind carbon
sharp zealot
#

I’m most impatient for directory arguments 😁

kind carbon
#

And directory/file returns? 😁

sharp zealot
#

Via dagger download, sure

#

different verbs for different handling of the output 👍

kind carbon
#

Ah, but shouldn't we keep one new cli command for now? And if so, which kinds of arg types and return types should dagger call support?

sharp zealot
#

so not really worried about what it does or doesn’t do

#

I’m looking for the thread where we started listing possible verbs, but of course can’t find it…

kind carbon
#

Yep, I'm thinking just ignore functions that don't match what's supported and expand over time.

sharp zealot
#

yes but different verbs are not about supporting different function signatures they’re about matching user intent. hold on let me find that thread

#

I’m especially excited about dagger shell 😁

#

better than eg dagger call --shell I think

warped canyon
#

I did a zenith oopsie and I was super lost, so here's my story

I called an argument to one of my functions os string, which in codegen collided with the os package and failed to codegen.

The error just looked like this:

Error: failed to get codegen output entries: input:1: host.directory.asModule failed to call module to get functions: failed to get function output directory: process "go build -o /runtime -ldflags -s -d -w ." did not complete successfully: exit code: 1

but when I ran with --progress=plain I was able to see a bit more detail:
./dagger.gen.go:4375:8: os.Exit undefined (type string has no field or method Exit)

Renaming my argument from os to operatingSystem fixed it

sharp zealot
warped canyon
warped canyon
#

Here's that module I was talking about: https://daggerverse.fly.dev/mod/github.com/kpenfound/dagger-modules/zenith@14c975c71a32f872692018e143a20b1518edc84a

So now you can run:
echo '{ zenith { update(repo: "$DOCKERHUB_USERNAME", operatingSystem: "darwin") { export(path:"./daggerz")} }}' | dagger query -m github.com/kpenfound/dagger-modules/zenith
To get an updated zenith engine pushed to your dockerhub and an updated CLI dropped at ./daggerz. No ./hack/dev, no downloading from github, no quarantine dance

wintry prism
#

I am trying this time.Now()

warped canyon
#

oh yeah no more worring about docker rmi or docker pull either, it's versioned 🙂

wintry prism
warped canyon
wintry prism
#

I used a dagger binary from the nightly builds

warped canyon
#

oh replace $DOCKERHUB_USERNAME with your dockerhub username. Sorry, that's in the readme but I didn't put that here

wintry prism
#

lol, I set an env var

warped canyon
#

yeah my example was single quoted though 🙃

wintry prism
warped canyon
wintry prism
#

I refuse to alter my workflow 😆

echo '{ zenith { update(repo: "'$DOCKERHUB_USERNAME'", operatingSystem: "darwin") { export(path:"./daggerz")} }}' | ~/dagger-z query -m github.com/kpenfound/dagger-modules/zenith
#

hmmm...we have ===, maybe I need triple quotes!

warped canyon
#

😭 how about swap the single quotes for doubles and escape the inner quotes

wintry prism
#

MOAR quotes

warped canyon
#

ayyy

#

check out that ./daggerz version

wintry prism
#

./daggerz version dagger v0.0.1696015539 (jeremyatdockerhub/dagger-zenith-engine) darwin/arm64

warped canyon
#

I wanted to do v0.0.0+1696015539 but docker isn't fully semver compatible

wintry prism
#

Success!

./daggerz query -m github.com/shykes/daggerverse/helloWorld<<EOF
{
  helloWorld {
    message
    withName(name: "Mundo") {
      withGreeting(greeting: "Bom dia") {
        message
      }
    }
  }
}
EOF
wintry prism
#

is arch optional?

warped canyon
#

right now it uses runtime.GOARCH, so it'll always be whatever your arch is. I figured if we wanted to wrap more of the nightly build workflow that would be a different function

wintry prism
#

so, you could use runtime.GOOS as well 🙂

warped canyon
#

You would think so, but I think that would result in linux 100% of the time since it's running in a linux container

#

I didn't try it, but i'm pretty sure

wintry prism
#

Hmmm...what's up with dagger mod init right now?

warped canyon
#

What are you seeing? works on my machine

wintry prism
# 
zenith-macos-arm64 ➤ ~/daggerz mod init name=zenith-macos-arm64 sdk=go
• Engine: 3b64fe6a44e8 (version v0.0.1696015539)
⧗ 4.54s ✔ 8 ✘ 1
Error: failed to get codegen output entries: input:1: host.directory.asModule failed to create module from config: failed to get runtime: failed to load sdk module: module "" has no SDKRuntime; you may need to run dagger mod sync
#

lol

#

need some better error messages and better flag input by me --name=

#

working with correct invocation

~/daggerz mod init --name=zenith-macos-arm64 --sdk=go
#

now trying
./daggerz mod use github.com/kpenfound/dagger-modules/zenith

# 
WARNING: Dagger engine version (0.0.1696015539) is older than the SDK's required version (
0.8.7). Please update your Dagger CLI.
WARNING: Dagger engine version (0.0.1696015539) is older than the SDK's required version (
0.8.7). Please update your Dagger CLI.
WARNING: Dagger engine version (0.0.1696015539) is older than the SDK's required version (
0.8.7). Please update your Dagger CLI.
WARNING: Dagger engine version (0.0.1696015539) is older than the SDK's required version (
0.8.7). Please update your Dagger CLI.
WARNING: Dagger engine version (0.0.1696015539) is older than the SDK's required version (
0.8.7). Please update your Dagger CLI.
• Engine: 3b64fe6a44e8 (version v0.0.1696015539)
⧗ 37.24s ✔ 432 ∅ 24
# 
zenith-macos-arm64 ➤ cat dagger.json
{
  "name": "zenith-macos-arm64",
  "sdk": "go",
  "sdkRuntime": "vito/dagger-sdk-go:no-vcs-flag@sha256:0fcc9d2659cdd551acb7aaf25e77215e9687475eb89ec06cae989eaf332ee480",
  "dependencies": [
    "github.com/kpenfound/dagger-modules/zenith@14c975c71a32f872692018e143a20b1518edc84a"
  ]
}
#

seems okay

warped canyon
#

Yeah I think the warnings are a side effect of my module not building a dev engine, but it should be ok

wintry prism
#

yep. Gonna drop the arm64 from my mod name since the arch is handled by your mod

#

~/daggerz mod sync doesn't seem happy. Guess you can't use that?

#

I wanted the Intellisense for my VScode

warped canyon
#

Should work fine 🤔

#

Do you have your wip module pushed somewhere?

wintry prism
#

It's because I renamed the module in the dagger.json only and missed a reference to the old name in a go.mod it seems.

#

Started over fresh and I'm 👍

warped canyon
ocean escarp
wintry prism
wintry prism
sharp zealot
#

also makes it hard to diff

#

@wintry prism let me move the code out, into a separate module

#

it's experimental anyway (and won't even work without a patch of the zenith build)

sharp zealot
#

@wintry prism can you give @strong ingot, @deft rain and me admin access to that repo?

#

Starting a thread to prepare the technical side of the Paris hackathon and demos 🙂

wintry prism
#

We're en route to Devoxx Morocco, so our wifi might be a little spotty. Luckily found this hotspot on the way. Don't worry, our dromadaires can carry their weight in Zenith. Plenty for all.

image.png
#

CI is 💚 Thank you!

image.png
sharp zealot
#

@wintry prism now that we have admin access, we should be good, enjoy your vacation!

#

@ocean escarp I'm very curious to learn more about your experiments with a JS SDK 🙂

ocean escarp
# sharp zealot <@437868223590039552> I'm very curious to learn more about your experiments with...

I copied the node sdk to adapt it to deno, you can see my branch here https://github.com/tsirysndr/dagger/blob/zenith-functions/sdk/deno/ , I added the entrypoint for the runtime https://github.com/tsirysndr/dagger/blob/zenith-functions/sdk/deno/src/ext/cli.ts which will import the dagger module in typescript and execute the dagger functions.

GitHub
dagger/sdk/deno at zenith-functions · tsirysndr/dagger

A programmable CI/CD engine that runs your pipelines in containers - tsirysndr/dagger

GitHub
dagger/sdk/deno/src/ext/cli.ts at zenith-functions · tsirysndr/dagger

A programmable CI/CD engine that runs your pipelines in containers - tsirysndr/dagger

sharp zealot
#

Starting a thread for feedback on your Node/Go/Python modules @upbeat herald

sharp zealot
#

So this is basically a Deno SDK for Zenith @ocean escarp ? Any thoughts on the experience so far?

ocean escarp
# sharp zealot That is awesome!

Yes, that's correct! It's been an interesting and fulfilling experience, the way you've thought about dagger modules is great!

#

I've seen that you use codegen in dagger modules in Go, in my case I didn't use codegen, when/for which language is it necessary?

kind carbon
#

However, the way that's done is in flux. There's a recent change that makes SDKs as modules, declaring functions for making a runtime container and for codegen, but we need to improve on how that's implemented. Currently only working for Go.

sharp zealot
#

small note on sdk-as-module: the init template should be sdk-specific also (or is it already?)

kind carbon
#

Not a part of the module yet, but yes that's the idea.

thorn moat
sharp zealot
thorn moat
sharp zealot
#

@kind carbon any chance we could add arguments dagger call? 🙂

#

I used it today and it works great btw

kind carbon
sharp zealot
#

It would be really nice to have a common query notation for expressing a chain of function calls (with arguments), then standardize its use across several verbs in the CLI

#

Current state:

  • dagger call: function name as subcommand, space-separated (?), max depth 1, no args
  • dagger shell: raw graphql, any depth, any args
  • dagger serve: function chain as dot-separated chain, any depth, no args
kind carbon
#

The base for dagger call is going to be used for other verbs, including shell, which I planned on adding first. So the usage should be consistent whatever that'll look like.

sharp zealot
#

@kind carbon yeah that's what I was thinking. But dagger serve (hackathon project by @thorn moat and @strong ingot ) has an alternative notation (dot-separate call chain) so at some point we need to agree on a design for that notation or we'll have parallel implementations

kind carbon
#

Same thing, yes.

sharp zealot
#

Or, if we want we keep the alternative implementations going a bit longer, to experiment

kind carbon
#

The dotted notation was actually by first intent with dagger call. I gave the idea not knowing that's what was being used in serve. I pulled back from that idea though.

sharp zealot
#

Things to agree on:

  • Space-separated or dot-separated chains?
  • How to encode arguments?
  • other?
kind carbon
#

There was already a bit discussion about that in the prototype, let me pull the findings from @rocky harbor.

sharp zealot
#

Yeah I remember that. Strong preference for space-separated in the first iteration unless it's specifically harder

#

And strong preference for implementing directory-type arguments first, our friend @wheat crypt is making his first module and it literally cannot work if he can't pass a local directory to it

#

We're trying manual stitching by copy-pasting the ID of a host.directory() query, but I don't know if that will work since IDs I believe are no longer reliably cross-session?

#

At the moment, top-level functions with directory arguments are the "fall off a cliff" point

sharp zealot
#

Heads up meetup starts in 90mn, so if you have any improvements to the CLI experience that are worth showing, make sure to push them in the next 45mn and notify us here, otherwise we won't be able to show them today

sharp zealot
thorn moat
sharp zealot
#

Oh, maybe we don't support returning []string ?

#

let me push the module for a repro

#

@thorn moat could you try this?

echo '{dagger{engine{versions}}}' | dagger query -m github.com/shykes/daggerverse/dagger
thorn moat
#

trying now (gotta do a quick hack/dev, zenith-functions i presume)

#

oh dear

#

something tells me this is not the goal

image.png
#

this was after a ton of cached vertices. not supporting []string seems like a good guess

sharp zealot
#

I don't think that's the problem actually. In supergit I have a method that returns an array of struct, and it works fine

#

There are cached vertices but if you look at the output, there are recurring error messages too

#

More visible in progress=plain:

[snip]
: > in init
1: > in init
1: starting engine [0.00s]
1: starting session [0.16s]
1: [0.00s] Failed to connect; retrying... name:"error"  value:"make request: Post \"http://dagger/query\": rpc error: code = Unknown desc = failed to verify client: client ID \"ayd8sb7yp9hq4bc4y9xo4nkm1\" not registered"
1: [0.16s] OK!
1: connect DONE

154: exec /runtime /bin/sh DONE
154: exec /runtime /bin/sh DONE
[snip]
thorn moat
#

besides the usual 'Failed to connect ... retry ... OK!' loop?

sharp zealot
#

ah - nevermind then

#

(those are scary)

thorn moat
#

oh, yeah that's normal, we can tidy it up, but without revealing it the alternative was hanging for 5 mins

#

maybe we can stretch out the OK! to OOOOOOOOOOOOOOOOOOOOOOOOKAY! or something chef_kiss

sharp zealot
# 
echo '{supergit{remote(url:"https://github.com/dagger/dagger"){tags{name}}}}' | dagger query -m github.com/shykes/daggerverse/supergit
thorn moat
#

@sharp zealot i think this might come down to a codegen bug we've been working around for a while, related to returning lists of objects. Here's the workaround:

funcs, _ := foo.Functions(ctx) // or any call that returns [Foo!]!

for _, function := range funcs {
    // TODO: workaround bug in codegen
    funcID, err := function.ID(ctx)
    if err != nil {
        return fmt.Errorf("failed to get function id: %w", err)
    }
    function = *dag.Function(funcID)
    if err != nil {
        return err
    }
    funcName, err := function.Name(ctx)
    if err != nil {
        return fmt.Errorf("failed to get function name: %w", err)
    }
    // ...
}
#

if you grep for 'workaround bug in codegen' you'll see it in a few spots 😅

sharp zealot
#

Would that also explain error on returning []string ?

thorn moat
#

so that's where the workaround would need to be implemented

sharp zealot
#

@thorn moat @rocky harbor sorry wrong thread earlier. I meant that importing multiple modules currently breaks

rocky harbor
#

I'm currently checking to see what we have left before we can press the merge button, was planning on making all those TODOs follow ups

deft rain
#

i think there's some issue deep in module resolution too sadly 😦
i've been chasing after it all day, somehow we mix up the modules? and start assigning functions to the wrong ones.

sharp zealot
#

solomon i think this might come down to

rocky harbor
deft rain
rocky harbor
deft rain
#

there now 🙂

deft rain
thorn moat
#

@deft rain thanks! I'm writing up an integration test now, will use this as a reference

deft rain
#

tl;dr - we attempt to load module bar and accidentally somehow load Foo in it (despite Foo being part of foo and not bar) - this seems to result in fun cases, like duplicate or non-existent schemas 🙂

rocky harbor
thorn moat
#

It's OK to commit them, but they might get pruned back out and re-.gitignored next time you run dagger mod sync (it's pretty aggressive atm). Now I'm worried this will be a common thing for module devs that are using off-the-shelf linting setups. Committing code to get around that kind of defeats the point

rocky harbor
thorn moat
#

It's weird too because if you just say 'oh linters should run go generate first' that's getting around something linters are often checking for too (forgetting to re-gen code)

wintry prism
#

When is this going work for real!? 😍

image.png
deft rain
dense canyon
#

that playground connects to your local engine. @hallow gust is adding a friendly message to communicate that

wintry prism
deft rain
#

linting with not-committed autogen code

wintry prism
hallow gust
wintry prism
#

might be that the list has a limit

#

Actually it's here:

image.png
#

Just no title

#

fyi @hallow gust 👆

hallow gust
#

I'm travelling right now, will take a proper look in ~30min

wintry prism
#

It works fine though

image.png
dense canyon
wintry prism
hallow gust
#

Going to push a fix to fallback to the Path if Subpath is not available. Extracting the module from the URL seems a bit too much IMO. wdyt @dense canyon ?

image.png
dense canyon
thorn moat
#

@dense canyon so is this because right now it's taking subPath and guessing that it matches the module name?

dense canyon
#

and we're using the subpath as the module name

thorn moat
#

at least until we put on our big boy pants and figure out how to model persistence, I like keeping refs as the source of truth with the smallest amount of info needed to bootstrap everything else

dense canyon
thorn moat
dense canyon
#

I won't make any changes now to avoid brining down daggerverse for a bit since deploying it has downtimes

thorn moat
#

multiple modules

rocky harbor
#

linting woes

thorn moat
drowsy basin
#

Setup a JDK & Maven dev environment

rocky harbor
thorn moat
rocky harbor
#

10 Authors! New record! daggerfire

Screenshot_2023-10-10_at_2.55.27_PM.png
deft rain
#

Woooo! nice work 🎉

#

@rocky harbor we should probably find somewhere to track the little collection of issues me and you stuck in the pr comments during the hackathon 👀

rocky harbor
#

Not sure if one gh issue vs. a bunch of little ones. Maybe I'll group where it makes sense

rocky harbor
thorn moat
rocky harbor
thorn moat
sharp zealot
#

Can’t believe zenith actually got merged. Very exciting 😁

drowsy basin
#

Now I need to understand how to modularize a SDK

kind carbon
sharp zealot
visual blaze
#

congrats Dagger team for this leap forward !! 👏

sharp zealot
#

@kind carbon @deft rain do you want to take advantage of time zone proximity and talk live about short term zenith work, while the hackathon feedback is still fresh?

sharp zealot
visual blaze
deft rain
sharp zealot
deft rain
kind carbon
deft rain
#

I've made an effort to try and grab all the issues/feature requests from #1151970583723126844, and put them into github so we're tracking everything.
I think the idea is to try and archive the #1151970583723126844 forum at some point soon (discussed with @sharp zealot this morning), since we've now merged to main. I've think I've grabbed all the things that are still issues, but if anyone notices some left, then let me know, or just open one on Github 🙂

sharp zealot
sonic vessel
sharp zealot
#

Should we cut a patch release to get zenith into a stable build? 🙂 cc @strong ingot @thorn moat

thorn moat
sharp zealot
#

@thorn moat any way we could merge services v2 behind a "feature flag" (not sure what that would mean in the context of API changes, possibly not worth it - but never hurts to ask)

#

that would be ideal right?

thorn moat
thorn moat
sharp zealot
thorn moat
#

Makes sense to me to decouple the two releases, then 👍

sharp zealot
#

Yeah definitely

strong ingot
#

@sharp zealot @thorn moat is this a patch or minor version bump?

thorn moat
#

i think patch since it's mostly a trojan horse to get Zenith out there unannounced but ready for tinkerers. No idea what else is in the release though, maybe there are other things that bump it to minor

strong ingot
#

patch sounds good to me!

sharp zealot
#

Can't hurt to double-check with @ember walrus and do a quick changelog read-through, but yeah unless I'm missing something, it's patch

sonic vessel
#

Compose on Zenith 👀

Screenshot_2566-10-11_at_21.15.46.png Screenshot_2566-10-11_at_21.16.10.png
sharp zealot
thorn moat
sonic vessel
sharp zealot
#

@sonic vessel you're going to love the upcoming dagger up command which will take your function and expose all its ports on the host. 😛

#

Basically dagger up + your compose module = a reimplementation of docker compose

sonic vessel
sharp zealot
#

(not sure if the zenith part of that is also in 5557, or in a separate PR?)

thorn moat
kind carbon
#

@thorn moat, I'm finishing off my day, I need a dagger.FunctionArg(argID) constructor for that bug or a fix for the loop if you can take it on.

sharp zealot
thorn moat
#

i can do that along the way, i've memorized the boilerplate 😛

kind carbon
thorn moat
#

should probably get it how we want it before we want more folks implementing them

restive shore
rocky harbor
#

simplifications to runtime bootstrap

dense canyon
kind carbon
#

Progrock

quick wind
#

Is it too early to ask about docs/snippets/cookbook/guides for Zenith/daggerverse?

kind carbon
#

Just a quick update on the CLI.

Screenshot_2023-10-11_at_23.56.45.png Screenshot_2023-10-11_at_23.57.11.png Screenshot_2023-10-11_at_23.57.19.png
agile saddle
#

When does this land for the python sdk?

restive shore
dense canyon
sharp zealot
#

If I remember correctly Fly itself resells Equinix Metal. I can’t imagine your employer blocks them too

#

@restive shore what happens if you nc -z daggerverse.dev 80?