#daggernauts

1 messages · Page 1 of 1 (latest)

sharp zealot
#

@honest hearth 👋 🙂

honest hearth
#

Hello! Here are some high level thoughts around problems I’ve been facing recently around pipelines that perhaps dagger do plus the API itself can help solve. It might be interesting to look at each one of these and designate whether they should be handled by SDK libraries or in the API/CLI/engine itself, or left up to the end user to implement themselves

Pipeline orchestration

  • How to execute pipelines concurrently vs in sequence
  • How to arbitrarily nest pipelines
  • How to safely share data asynchronously across pipelines
  • How to pass one pipeline/step result to the next when it’s one pipeline
  • How to collect multiple pipelines/steps results that ran concurrently and pass them into a new pipeline or list of pipelines
  • How to inject configuration across pipelines/commands
    Project Entrypoints
  • How to discover commands within a project and across projects
  • How to scope what a project is
  • How to define commands and groups
  • How to nest commands and groups
  • How to make the things that commands do composable and have dependencies
    - Simple Example: I want to have one command: ci. I want it to run build, test, and publish pipelines. These should each be their own separate commands as well though. Publish should depend on test which should depend on build in sequence.
  • How to marry the concept of commands, groups, and pipelines
  • How to query for what commands and groups are available along with all of their metadata (arguments, descriptions, etc)
  • How to run my own hooks and validations on top of whatever dagger implements whenever I run dagger do
sharp zealot
#

@honest hearth do you have the need for multiple projects in the same monorepo?

#

for the “pipeline orchestration” part: our assumption is that we can keep that a separate topic, to prevent scope creep in Zenith (already a lot going on between CLI, entrypoints and possibly extensions). But I know you weren’t 100% sure how realistic that decoupling is, so we can double-check together.

The ideal outcome for me is that we address your pipeline orchestration issues (which I happen to share 🙂 with a combination of 1) incremental improvements to the core Dagger API, and possibly 2) opinionated extensions

#

My educated guess / wishful thinking for your project-related questions:

#

How to discover commands within a project and across projects
100% of project-related operations, including discovering and invoking commands, should be available via the Dagger API. Ideally via a simple client implementation.

#

How to scope what a project is
To keep things simple, a project maps to a directory with dagger.json. There are still open questions on the details of that mapping though 😁 including:

  • dagger.json optional?
  • relationship to parent and child directories with their own dagger.json ?
honest hearth
#

conor 8783 do you have the need for

muted plank
#

👋 I’m interested to see how this plays out ^^

spiral whale
#

Really nice summary from @honest hearth , For #github , I have similar thoughts as well, especially for pipelines. Just one addition

  • How to customize entrypoints like configuring unixsockets to support Dagger in Dagger scenarios?

I think, this would move UX experience for #github to another level.

sharp zealot
#

Really nice summary from conor 8783 For

carmine vortex
#

config management is a good topic for DX improvements. having something built-in would be pretty nice. without a solid configuration system to work with, i found that my dagger code quickly turned to spaghetti. ended up trying a bunch of different things, and settled on this implementation https://github.com/rancher/opni/blob/dagger/dagger/config (usage doc is in dagger/main.go) which works extremely well for my use cases. I hope this gives you some inspiration 😄

honest hearth
#

config management is a good topic for DX

sharp zealot
#

👋 @kind carbon @upbeat herald

rocky harbor
# honest hearth Hello! Here are some high level thoughts around problems I’ve been facing recent...

RE: Project Entrypoints

What Solomon said earlier about this all being 100% in scope for us to support, plus you can also see the (extremely primitive, subject to change, experimental, etc.) API here, which is part of the core API and thus available in each client SDK: https://github.com/dagger/dagger/blob/736a68be4640078cb2e67aca4da547d1b0e1e4ba/core/schema/project.graphqls#L26-L58

Lots missing still, but the biggest one is support for actually invoking a command in a project, which still requires submitting a raw graphql query. We plan on adding that too though, at which time you will be able to load projects, see all the commands provided by them (including doc strings, input arg names+types, output types, etc.), and invoke any of those commands all just with a dagger client.

rocky harbor
thorn moat
thorn moat
thorn moat
#

@rocky harbor @sharp zealot Just added support for importing OCI layout directories instead of tarballs. It's WAY faster! This should make it totally viable to use Nix (or any other tool) to build images.

Another nice thing is it doesn't need the client-side content.Store; it's all remote, and it only transfers blobs that haven't been imported into Buildkit already.

Will have a PR up soon, this is currently implemented on top of the "focus" changes for maximal dogfooding but it's totally independent

asciinema
vito
importing OCI layout dirs

Recorded by vito

thorn moat
sharp zealot
#

oh is that the call we discussed way back at api design time, but agreed would have to wait because of implementation difficulty?

#

or maybe I’m confusing with container.export

thorn moat
ember walrus
wintry prism
#

@young lintel this project is where the discussions around extensions will happen. We're keen to support your ideal use case 🙂

spiral whale
#

do we have any design docs or discussion around extensions? In #github we would need different components/extensions for certain tasks / behaviours. Maybe we can leverage extensions inside of the gale as well.

sharp zealot
#

more soon

sharp zealot
#

@ember walrus @thorn moat @rocky harbor I propose switching to this open channel for this second phase of our little hack week

#

it may not all make sense out of context, but it may prompt conversations

ember walrus
#

Sounds good! Let's do it

upbeat herald
#

@hexed flume I find out the problem with integration tests for Typescript entrypoint!

The Dagger TS library needs to be built in order to be used, but since we're not pushing build artifact, the only case where we cannot access to built library is in our own CI...
In others this would work since they use the release package, in local we can built our library to test it but in the CI we do not.
A fix would be to add an extra step in case of Typescript test so it build the library with new changes before we test it.

What do you think?
/cc @rocky harbor @strong ingot

ember walrus
rocky harbor
young lintel
sharp zealot
#

From Wikipedia:

In software deployment, an environment or tier is a computer system or set of systems in which a computer program or software component is deployed and executed. In simple cases, such as developing and immediately executing a program on the same machine, there may be a single environment, but in industrial use, the development environment (where changes are originally made) and production environment (what end users use) are separated, often with several stages in between. This structured release management process allows phased deployment (rollout), testing, and rollback in case of problems.

Environments may vary significantly in size: the development environment is typically an individual developer's workstation, while the production environment may be a network of many geographically distributed machines in data centers, or virtual machines in cloud computing. Code, data, and configuration may be deployed in parallel, and need not connect to the corresponding tier—for example, pre-production code might connect to a production database.

#

@ember walrus @thorn moat 👆

rocky harbor
sharp zealot
#

Dev environment:

The development environment (dev) is the environment in which changes to software are developed, most simply an individual developer's workstation. This differs from the ultimate target environment in various ways – the target may not be a desktop computer (it may be a smartphone, embedded system, headless machine in a data center, etc.), and even if otherwise similar, the developer's environment will include development tools like a compiler, integrated development environment, different or additional versions of libraries and support software, etc., which are not present in a user's environment.

#

Testing environment:

The purpose of the test environment is to allow human testers to exercise new and changed code via either automated checks or non-automated techniques. After the developer accepts the new code and configurations through unit testing in the development environment, the items are moved to one or more test environments.[3] Upon test failure, the test environment can remove the faulty code from the test platforms, contact the responsible developer, and provide detailed test and result logs. If all tests pass, the test environment or a continuous integration framework controlling the tests can automatically promote the code to the next deployment environment.

#

Staging:

A stage, staging or pre-production environment is an environment for testing that exactly resembles a production environment.[7] It seeks to mirror an actual production environment as closely as possible and may connect to other production services and data, such as databases. For example, servers will be run on remote machines, rather than locally (as on a developer's workstation during dev, or on a single test machine during the test), which tests the effects of networking on the system.

#

Production:

The production environment is also known as live, particularly for servers, as it is the environment that users directly interact with.

#

This is just a refresher on the baseline, mainstream understanding of the word. It may not be perfectly accurate - but it's roughly what a beginner will know about the word after some basic googling.

#

Notably absent is "preview environment" which is like a hipster, cloud-native take on "staging"

spiral whale
rocky harbor
#

Think I found an approach to move to the new "environment builder" model that has the same niceties of the previous hierarchy approach in terms of global configuration that is also lower boilerplate (deletes more lines than adds): https://github.com/sipsma/dagger/commit/695e3807998e8f0864fc35af46b60951d0f2450b

Gonna try splitting out at least one of the SDKs to get a feel for how we can have one environment use/consume/depend-on/whatever another environment.

Then I'll see what I can get actually implemented, cheating as necessary, and start hooking up with the stuff you've been doing @thorn moat to try out a real working universe 🙂 cc @ember walrus @sharp zealot

ember walrus
# sharp zealot This is just a refresher on the baseline, mainstream understanding of the word. ...

That' aligns to my existing world view of environment - and I don't think it matches 1:1 with what we're hacking out today for .

For instance, let's take a webapp.

  • The frontend team wants its own contained sphere for its setup, services, etc that they can cultivate.
  • The backend team wants the same.

Each team needs to be able to run, build, test these separately. And they also want to do that for the full webapp (backend and frontend).

We could look at frontend and backend being their own environments that get composed into the webapp environment.

I'm having difficulty reconciling the different uses of the word environment - in one case, it's basically a stage from development to production. In another, it's a definition of all of the pieces and logic that you need to ensure that your app works since all of those dependencies are built and tested the way that they need to be.

spiral whale
#

Hey,

I have been encountering some issues with managing individual components for the #github recently.I am considering the possibility of utilizing #daggernauts to develop a catalog or extension system that can effectively organize, manage, and maintain individual services and binaries.

if I need to extend this use-case:

The primary issue here, To ensure efficient execution of the 'gale', it is necessary to integrate various customized binaries and services. This strategy has revealed the need for additional specialized components in the future.

Ultimately, all these components can be categorized as File, Directory, or Container, each with its own designated entry point.

Couple of examples of extensions

  • Artifact/Cache Service
  • Workflow state for communication between multiple dagger.containers
  • Github Action Executor for executing steps/jobs

What I need here mostly:

  • Loading extension from a source(local , git or registry)
  • Configure the extension
  • Integrate it to my existing container (like copy file to container or bind extension as service to given container etc)

This kind of integration would make huge difference for #github

cc: @rocky harbor , @sharp zealot

sharp zealot
rocky harbor
sharp zealot
#

with git remote mapping, you could me that -e dagger implicit, so:

dagger do cli —output ./bin

rocky harbor
sharp zealot
#

We did a first pass of design discussion on an environment schema for Zenith. This would be additive to the current Dagger API schema.

extend type Query {
    "The current environment"
    here: Environment!

    "Load a new environment"
    environment(
        "Source code of the environment"
        source: Directory!,
        "Working directory of the environment"
        workdir: Directory!): Environment!
}

"API for your Dagger environment"
type Environment {
  "Command-line tools available in the environment"
  tools: [Tool!]
  "Lookup a command-line tool by name"
  tool(name: String!): Tool!

  "Artifacts available in the environment"
  artifacts: [Artifact!]
  "Lookup an artifact by name"
  artifact(name: String!): Artifact!

  "Tests available in the environment"
  tests: [Test!]
  "Lookup a test by name"
  test(name: String!): Test!

  "Services available in the environment"
  services: [Service!]
  "Lookup a service by name"
  service(name: String!): Service!
  
  "Interactive shells available in the environment"
  shells: [Shell!]
  "Lookup a shell by name"
  shell(name: String!): Shell!

  "The working directory for this environment"
  workdir: Directory!

  "Direct access to the environment as a data graph. Great for scripting and composition"
  graph: EnvironmentGraph!

  """
  Extend this environment with the capabilities of other environments.
  If no namespace is provided, environments are merged - this may cause naming conflicts.
  """
  withExtension(namespace: String!, environment: Environment!): Environment!
  "Extensions currently added to the environment"
  extensions: [String!]!
  "Lookup an extension by its namespace"
  extension(namespace: String!) Environment!

  "Warm up the environment cache by pre-executing as much of the DAG as possible. This may make subsequent operations considerably faster"
  warmup(timeout: Number)
}


type EnvironmentGraph {
    # This type will be extended by an environment-specific graphql schema
}

type Tool {
    name: String!
    description: String
    commands: [ToolCommand!]!
}

type ToolCommand {
    name: String!
    description: String
    subcommands: [ToolCommand!]
    # invocation?
}

type Artifact {
    name: String!
    description: String
    labels: [String!]
    contents: Directory!
}

type Test {
  name: String!
  description: String
  result: TestResult!
}

type TestResult {
  success: Boolean!
  output: String!
}

type Shell {
    name: String!
    description: String
    terminal: Stream
}

// A bi-directional byte stream (eg. for terminal emulation)
type Stream {
    // Path to the websocket stream, relative to the current HTTP server
    wsPath: String!
}
#

courtesy of:

IMG_4155.jpg
rocky harbor
sharp zealot
# 
extend type Query {
    "The current environment"
    here: Environment!

    "Load a new environment"
    environment(
        "Source code of the environment"
        source: Directory!,
        "Working directory of the environment"
        workdir: Directory!): Environment!
}

"API for your Dagger environment"
type Environment {
  "Command-line tools available in the environment"
  tools: [Tool!]
  "Lookup a command-line tool by name"
  tool(name: String!): Tool!

  "Artifacts available in the environment"
  artifacts(filters: [String!]): [Artifact!]

  """
  Checks available in the environment
  This could be unit tests, integration tests, linting, etc.
  """
  checks: [Checks!]
  "Lookup a check by name"
  check(name: String!): Check!

  "Services available in the environment"
  services: [Service!]
  "Lookup a service by name"
  service(name: String!): Service!
  
  "Interactive shells available in the environment"
  shells: [Shell!]
  "Lookup a shell by name"
  shell(name: String!): Shell!

  "The working directory for this environment"
  workdir: Directory!

  "Direct access to the environment as a data graph. Great for scripting and composition"
  graph: EnvironmentGraph!

  """
  Extend this environment with the capabilities of other environments.
  If no namespace is provided, environments are merged - this may cause naming conflicts.
  """
  withExtension(namespace: String, environment: Environment!): Environment!
  "Extensions currently added to the environment"
  extensions: [String!]!
  "Lookup an extension by its namespace"
  extension(namespace: String!) Environment!

  "Warm up the environment cache by pre-executing as much of the DAG as possible. This may make subsequent operations considerably faster"
  warmup: Environment!
}


type EnvironmentGraph {
    # This type will be extended by an environment-specific graphql schema
}

type Tool {
    name: String!
    description: String
    commands: [ToolCommand!]!
}

type ToolCommand {
    name: String!
    description: String
    subcommands: [ToolCommand!]
    # invocation?
}

type Artifact {
    name: String!
    description: String
    version: String
    labels: [String!]
    contents: ArtifactData!
    sbom: String!
}

union ArtifactData {
    Directory
    File
    Container
}

type Checks {
  name: String!
  description: String
  result: CheckResult!
  subchecks: [Check!]
}

type CheckResult {
  success: Boolean!
  output: String!
}

type Shell {
    name: String!
    description: String
    terminal: Stream
}

// A bi-directional byte stream (eg. for terminal emulation)
type Stream {
    // Path to the websocket stream, relative to the current HTTP server
    wsPath: String!
}
rocky harbor
#

could be wrong

sharp zealot
#

I want to point out one important aspect of the Zenith design: you use Dagger/zenith to develop environments, and environments have their own source code repository.

This is different from the current best practice, where environment code is embedded into the repository that will be used as its working directory. From the point of view of the environment's programmer, this mixes data and code. In the Zenith design, the environment's code will be cleanly separate from its data.

rocky harbor
sharp zealot
#

Rough draft of how we might explain Dagger post-Zenith:

# Dagger

Standardized environments for modern software teams.

## What is Dagger?

Dagger defines a new standard for packaging, distributing and running any software environment as a DAG. Once "daggerized", environments have the following properties:

* Runs anywhere
* Shareable
* Repeatable
* Cached by default
* Scriptable

## Why Dagger?

Using Dagger can solve or mitigate the following problems:

- New developers take too long to be productive
- Tests pass in development but not in CI
- CI environment lags behind development
- End-to-end testing requires artisanal scripts to glue incompatible environments together
- Platform teams struggle to integrate MLops environments into the continuous delivery pipeline
- Major stack changes cause disruption or delays, because environments are hard to change and test reliably
- CI/CD pipelines gets slower as the stack and team grow
rocky harbor
sharp zealot
#

Zenith README, take 2 🙂

# Dagger: standardize your dev environments

## What is Dagger?

Dagger defines a new standard for packaging, distributing and running development environments as a containerized DAG. Once "daggerized", environments have the following properties:

- **Portable**: any machine that can run standard containers can run a DAG.
- **Familiar**: keep using the tools you know, packaged in containers.
- **Scriptable**: automate custom tasks in any language, using a powerful API and growing list of SDKs
- **Extensible**: easily incorporate the capabilities of other environments into your own
- **Scalable**: all operations in the DAG are scheduled concurrently, which allows seamless parallelization across cores and if needed, across machines in a cluster (*coming soon*)
- **Fast**: all operations in the DAG are continuously cached, making all computation magically incremental. As a result, daggerizing an environment typically makes builds and other cpu-intensive tasks much faster out of the box.

## What is a containerized DAG?

A containerized DAG is a computing environment which can run programs made of many concurrent containers, each running a simple operation, with data flowing between the containers in a graph layout.

## Why Dagger?

Daggerizing your team's environments can solve or mitigate the following problems:

- New developers take too long to be productive
- Tests pass in development but not in CI
- CI environment lags behind development
- End-to-end testing requires artisanal scripts to glue incompatible environments together
- Platform teams struggle to integrate MLops environments into the continuous delivery pipeline
- Major stack changes cause disruption or delays, because environments are hard to change and test reliably
- CI/CD pipelines gets slower as the stack and team grow

## Why not just use a container?

There is an emerging trend of running development environments in a container (see Devbox, Devpod, Gitpod, Nix, Replit). This is a considerable improvement from the previous status quo. But it's not enough, because most environments are too complex and dynamic to fit in one container. Docker-compose is slightly better, because it can package a multi-container application; but in spite of its name, Docker-compose is neither composable nor scriptable.

## What kind of environments?

In theory, any software environment can be daggerized, as long as its individual components can run in a container. In practice, Dagger is most commonly used for non-production environments, such as:

* Development
* Build
* Test, especiall end-to-end integration testing
* CI/CD
* QA and staging, also known as "preview environments"
* MLOps
* Data engineering
* Research

Although Daggerized environments often integrate into production - for example by building production artifacts, or deploying to a remote production environment - the production environment itself rarely runs in Dagger.
ember walrus
#

I like this revision a lot. I have a few suggestions but will sleep on it and post tomorrow

sharp zealot
#

Let's move discussions of those two documents to the PR please

honest hearth
#

First draft of Zenith README and GraphQL...

restive shore
#

Will Zenith still work without dagger CLI? Would someone for example be able to go run main.go ? Today, we've built a CLI/SDK wrapping the Go dagger SDK that can work with or without the dagger CLI. I wonder what the experience will be like for something like that when Zenith comes into the picture. I am hoping the interface will be much simplified but also hoping we don't have to make developers rely on the dagger CLI if they don't want to.

sharp zealot
#

In practice I believe we will have a good way to continue supporting embedding dagger in custom tooling. We will need to work out some details but we don’t anticipate any dealbreakers

sharp zealot
#

Quick update @restive shore after discussing this live. Our educated guess is that SDKs will continue to work exactly as today for your use case. The only difference might be that SDKs would no longer-autoinstall the CLI. So your tool would have a new runtime dependency: the dagger CLI. So your end users would need dagger installed - even if they never interact with it directly.

restive shore
rocky harbor
# restive shore Hmm, do you mean the sdk won't download the engine automatically? Because the SD...

It currently does because it needs the CLI as a helper to run pipelines. It will skip the automatic download if you set an _EXPERIMENTAL_DAGGER_CLI_BIN env var to a pre-downloaded CLI: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#dsi-advanced---automatic-provisioning

We just want to move away from all the hacky _EXPERIMENTAL_* stuff, part of that will likely include not doing the automatic download anymore, which was always meant to just be a temporary convenience hack before we made it easy to install a CLI anywhere

restive shore
#

Oh? TIL.. I was under the impression that the SDK only downloaded the dagger engine image. Is the CLI embedded in the engine image? I assume what the SDK downloads is temporary. It's not the same as installing the CLI via something like homebrew right?

#

If that's the case, the SDK can technically do all the TUI stuff too?

rocky harbor
rocky harbor
restive shore
#

I see! It would be nice to be de-coupled from the CLI or have the dagger SDK provide a way to define how to pull all the required tooling for it to work, but I understand if that's not the direction you decide to go. I am happy that the SDK is still going to work pretty much the same way as today. We are building a library of sorts of capabilities for developers. A thin wrapper over the dagger SDK and the idea is to distribute that as a CLI that developers can just use OOTB to perform CI/CD for their applications. Mostly adhering to our company conventions and standards. More adventurous devs/teams can extend this SDK to compose their own patterns. At least that's the plan. I expect zenith to vastly improve tool/library developer experience as well as enable new and better interfaces for the consumers of our tool. Very excited for that but it's important to me that it doesn't require us to entirely throw away what we'd be building from now till when Zenith is GA.

rocky harbor
# restive shore I see! It would be nice to be de-coupled from the CLI or have the dagger SDK pro...

Totally, we have a few other user's with usecases very similar to exactly what you're describing and the direction we are heading will support it too!

The quick explanation is that while dagger do is special in that it's the "official" dagger cli and comes with our built-in TUIs, the implementation is actually just using core APIs available in any SDK to load environments and invoke entrypoints from them. So if someone wanted to build their own CLIs on top, their own web interface, their own code libraries, etc. that is still 100% supported and easier than ever actually since you'll have the Zenith APIs to help you out with low-level details rather than needing to reinvent everything from scratch necessarily.

#

@thorn moat Continued working yesterday fixing up the implementation in my zenith branch to get it ready for integrating with your environments, but I keep hitting problems related to the whole issue of sessions being separate. Most recent set was due to the fact that when you stitch the APIs, you don't stitch it in the other session, which I had a workaround for but then that workaround had its own issues and I was starting to workaround my workaround, etc.

Long story short, I'm gonna spend today seeing if I can get an MVP of the session frontend idea working, even if I just temporarily comment out a few of the trickier parts where some of the APIs have hardcoded assumptions about reading/writing local dirs. That seems more productive than fighting the current approach, especially since it will also simplify SDKs even further. I'll make a call EOD if it's the most time efficient path forward. Just FYI, cc @sharp zealot

thorn moat
sharp zealot
sharp zealot
kind carbon
rocky harbor
rocky harbor
# rocky harbor <@108011715077091328> Continued working yesterday fixing up the implementation i...

(copying from my linear update) Have session frontends passing basic tests now: https://github.com/sipsma/dagger/tree/session-frontend

e.g. you can pull that and run ./hack/dev go test -v -count=1 -run=TestDirectoryWithDirectory ./core/integration/

Will panic if you try to use any session attachables (secrets, sockets, etc.) or do an export as I haven't updated those implementations yet.

Going to get the MVP needed for environments working now, which is really just host import/export. Import shouldn't be hard, may actually work already. And I think there's a potentially easy route to support exports by just directly calling the worker exporter method.

If that goes as planned I'll start rebasing my zenith branch on that one and have smoother sailing going forward

sharp zealot
#

Nice demos @rocky harbor @thorn moat 🙂

rocky harbor
upbeat herald
#

I might take a break on the typescript entrypoint regarding all these changes that will happens with Zenith
But at least we could merge what has been done so far so people can already try entrypoints

sharp zealot
#

Thinking about next week's demo. It would be nice to highlight the future experience a bit more, in the current demos it's overshadowed by the current implementation. This makes it great for our current audience of power users looking for the next incremental improvement (project commands basically). But it won't show the potential of Zenith to make Dagger more approachable to a new audience (devs).

I know it's easier said than done, because the future experience is, well, mostly in the future. But if we could somehow "fake it" even partially, it would be really worth it.

#

IMO between now and next thursday, it would be a worthwhile investment to scaffold a fake (or partially fake) implementation of other entrypoints, and the CLI experience for them - by the way that was an unfinished topic we wanted to discuss @thorn moat

thorn moat
#

I can start a PR that just adds a markdown file or something and we can do linewise comments?

sharp zealot
#

Also we should try to introduce the standalone environment aspect of the design - even faked or partially faked - to start conversation on that part, since it's so important

sharp zealot
thorn moat
sharp zealot
thorn moat
sharp zealot
#

Dogfooding opportunity: I'm trying to make a small-to-medium contribution to the Dagger docs, and could really use a turnkey environment that guides me 🙂

rocky harbor
# rocky harbor (copying from my linear update) Have session frontends passing basic tests now: ...

Just to update on my progress here, ran into weird errors and flakiness while trying to rebase zenith, turns out I needed to implement a big chunk of work that I thought I could cheat on: https://github.com/dagger/dagger/pull/5415#issuecomment-1626392187. Got it working now though: https://github.com/dagger/dagger/pull/5415#issuecomment-1629571175

So now I'm gonna get zenith rebased on that new and improved branch as quickly as possible to unblock everyone else. Sorry about the delay! Besides just making it possible to implement zenith, the new architecture is gonna enable so many other neat things, so the wait should be worth it hopefully

sharp zealot
#

Thanks for the update! Quick update on my end:

  1. Having fun bikeshedding mock UI with @thorn moat here: https://github.com/dagger/dagger/pull/5420.

  2. Long story, but we may want to "backport" a minimal concept of environment into current Dagger, as a basic namespacing construct to unblock Dagger Cloud. See https://github.com/dagger/dagger/issues/3345

  3. I am concerned that by this thursday (first community demo) we will have lots of powerful plumbing to show, but not a lot of context for why it's awesome or how it changes the overall UX. @thorn moat I'm wondering if maybe we need 2 mockup tracks: one where we focus on depth (seek out hard design topics to bikeshed), and another focused on width (give the illusion of maximum functionality with minimum underlying design or implementation work)?
    --> 5420 is our depth-first track 🙂
    --> perhaps another track where we show off more detail on artifacts & checks, and do some "fake dogfooding" on top of that?

GitHub
Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 330 million projects.

GitHub
Basic namespacing in API · Issue #3345 · dagger/dagger

Overview In the new cloak core API, cache volumes are associated with a key provided by the client. Buildkit does not provide a facility for namespacing these keys, so by default every client that ...

#

I think point 3 is really about making the demo more "robin hood friendly".

For example as I mentioned in this thread: #1126976209591472158 . Today I have used Dagger to contribute docs to the dagger repo. At the moment my experience is worse than if we just had a Makefile or npm script. The main difference is that I get more noise in the output (something about containers?). But if I could somehow:

  1. Type one command to see all checks
  2. See a bunch of red: failed checks! With the reason why
  3. fix my code, repeat...

That would be a much better experience.

I would love to convey a little bit of that feeling to the audience on thursday

#

Still in the same use case: my docs contribution includes a code snippet in several languages. I am getting linter errors on those snippets. One of them is "code is not formatted with gofmt" but perhaps I'm not a go developer and am not sure how to godfmt correctly - it would be nice to have a tool to do that for me.

thorn moat
# sharp zealot Thanks for the update! Quick update on my end: 1. Having fun bikeshedding mock ...

It's hard for me to imagine a breadth-first track not immediately turning depth-first as soon as we start making comments. 😁 5420 was meant to be breadth-first and even robin-hood focused, but one thing leads to another...

From my POV we're at an "innovation thrives in mess" phase in the CLI design where we just need to hash out anything we can and try to let one answer lead to another. Anything is on the table. I'm hesitant to demo a totally made-up list of commands without having some idea of the underlying mental model, because I'd guess everyone watching would be the critical engineer type that would be running their brain at 120% capacity trying to draw connections that might not actually exist.

But by all means, I could use some help covering any gaps I missed along the way, since there are almost certainly parts of the DX you've given more thought than me. Is there an outlet you feel is missing for throwing more CLI ideas out there? Or did you have something concrete in mind for representing these parallel tracks?

thorn moat
#

Also all of this is based on the assumption that the actual task to create the faked-out CLI won't take very long once we know what to fake out.

sharp zealot
#

yeah I'm with you, I don't have a specific outlet in mind. Just wanted to point out that on the current trajectory, our demo will be plumbing-heavy.

thorn moat
thorn moat
#

@sharp zealot Going to try adding real commands into the dagger CLI, but hardcoded against a certain environment (so e.g. dagger test will just always run Progrock's test entrypoint). Should be easy enough to iterate with code at this point and just add as many aliases as we want, and the upside is we'll have a functioning demo at the end where we can actually change code and re-run tests.

sharp zealot
sharp zealot
thorn moat
#

I thought that already existed not_sure_if

#

oh I may be thinking of passing repeated -f flags

sharp zealot
thorn moat
#

@sharp zealot if you want to kick the tires, this branch has the functional CLI: https://github.com/vito/dagger/commits/vito-zenith

Usage:

$ go install ./cmd/dagger
$ cd progrock
$ dagger checks # or dagger tests
$ dagger check  # runs first defined check, TODO should run all in parallel
$ dagger artifacts
$ dagger export demo # or dagger build

Some of these commands probably work in other Go repos too, feel free to try. 😛

#

oh, you'll need a ./hack/dev in there somewhere actually

sharp zealot
#

Nice! Thanks for cranking this out so quickly

#

I still feel like we won't need a top-level verb for "build" or equivalent. Argument: in modern software projects there is rarely only one thing to build, or only one way to build it; so devs are ready for a noun-oriented approach. Instead of "build the thing!", I say the world is ready for "show me the artifacts!"

thorn moat
#

We definitely don't need it, but a little sugar in the CLI can go a long way imo. If you go too far towards the theoretical/abstract truth of it all, you get further away from meeting most users where they are, and start designing for things they don't need. Most of my projects have one important thing I want to build and test, and I don't want to have to type too much for commands I run all day. But part of this exercise is to see how that actually feels, so I'll be playing with all the variants anyway.

sharp zealot
#

Yeah I agree with that - my guess is the ergonomic polish will keep moving quite a bit, while we figure out the foundations.

#

Which is why I'm so happy you're shipping something real so fast, gives us more opportunity to try things

thorn moat
#

totally - feel free to toss more command ideas my way btw, it's a good time to just try things on. I can start adding noun-verb hierarchies like artifact get too

sharp zealot
#

One related itch I can't help scratching: in theory, a noun-oriented interface makes it way easier to create a super simple web UI. For example a list of red/green checks... I have a feeling that we could make that a prominent feature, possibly a game changer for robin hood hats, with relatively little effort... but I'm not sure exactly how

#

I think part of the blockage is that, as soon as we talk about "API", we think "codegen, SDKs in every language, top hat".

But maybe there's a simpler version of the API, that is specifically for developing web UIs for your Dagger environment, which means no codegen, and only JS support?

#

I mean, if we're talking about "wow effect" in a demo...

thorn moat
#

Most of that could be accomplished by using the GraphQL API directly right?

sharp zealot
#

Yes I believe so. But would need to try to be sure. Maybe we'll find there are easy ways to make the API more accessible in this use case? But could be a no-op

thorn moat
#

Yeah hopefully it's a no-op API wise just to minimize surface area. There are other devils in the details too. Showing red/green check status for example implies that they have run at some point, and that we're able to "cache" failed results. I don't think we would want the page that shows all the tests to require running them all, so it'd be nice if there was a way to show a green check but only if there's a cache hit. I've wanted something like that from Buildkit before

#

I'm thinking something like dagger checks => shows each check status, which is pending if their cache busted, or successful if they've passed and will just be a cache hit if you run again. Then you run them all with dagger check and they should all have an up-to-date state, until you change code again.

sharp zealot
#

The web UI could reflect this constraint (that some queries are expensive), and deal with that in the UI. For example, list all the checks, but require that the user click a button before querying the result (unless they're cached, which would require the ability to check for cache status in the API)

thorn moat
#

yup!

sharp zealot
#

But maybe a first version could just go ahead and make the queries - firing off a flurry of pipelines in the background. Might not be so bad

#

(ie. we may not have to wait for a new API to query cache state to explore cool web UIs, either built by us or by the community)

thorn moat
#

pushed a few noun-verb aliases: check list, check run, artifact list, artifact get

#

"check" seemed a little iffy because it's both a noun and a verb, but it works in practice. dagger check still runs the first check, dagger check test still runs the test check, and sub-commands work fine; I guess they take precedence

#

small UX note: it'd be really nice if exporting artifacts listed the files that it wrote. especially because that sometimes means "overlaying" onto your working directory, updating a bunch of files across the tree (e.g. go generate ./...)

rocky harbor
#

Gonna go look at what you're working on @thorn moat and see how plausible it could be to join that with the implementation PR before the demo, which would be great obviously, but we'll see if the timing works out

#

(also planning on catching up on the other discussions now that there's a minute of breathing room 🙂 )

#

@thorn moat @sharp zealot any concrete ideas yet on what specifically we'd like to show in the demo? I can focus on integrating those parts.

rocky harbor
sharp zealot
#

I think if we had a working, credible example of querying checks and artifacts - that would really help show the potential of Zenith to make Dagger more accessible via the CLI

#

ideally the first part of the demo involves no code - just using the CLI. Then we switch to top hat, and show the code that made the magic possible. Ideally it looks like not a lot of code, for so many cool features

rocky harbor
#

Cool yeah I totally agree, I'll start there and see how much we can get working. Won't start here, but if there's time I'm very tempted to see if we could throw together dagger shell using @chrome pilot's tty-over-websocket implementation from back in the day quickly. For me personally that would be maximally 🤯

sharp zealot
#

yeah that would be killer

tough sentinel
thorn moat
#

My head is spinning with all the things in flight already though:

  • WithFocus() API
  • importing OCI layout dirs
  • services v2
    • upstream buildkit PR
  • today's zenith CLI spike
  • prospective universe code
#

a few of these have mutual rebase conflicts, too, and that's not counting the rocket surgery going on on your branch @rocky harbor 😛

rocky harbor
#

One thing is that we should talk through my pr for re-architecting the graphql server and yours for services and see if there's new ways of reconciling them. Part of the re-arch is that we now have extremely fine grained control over sessions, which I'm sure will impact the approach in your upstream buildkit PR (and possibly give new routes for accomplishing it)

thorn moat
#

if anyone has a moment, WithFocus() is ready to go, and almost definitely has an impact on the demos, and is the source of some merge conflicts, so it'd be nice to land it: https://github.com/dagger/dagger/pull/5364

thorn moat
rocky harbor
rocky harbor
thorn moat
#

we could also rename Container.import to Container.fromOCITarball but I'm tempted to just deprecate it tbh

#

ah I guess we'd want it for symmetry with Container.export?

sharp zealot
#

If import is just not going to cut it because it’s too slow, and we don’t think we will ever miss it (because importing from a directory is just as easy, and/or importing from a tarball can be implemented in userland) then maybe simpler to just break import to take a directory instead of a tarball

thorn moat
#

hmm, yeah I think the only weird thing is what to do with export. it's weird to have it write a .tar, which is more generally useful with the outside world at the moment, but then you have to untar it to pass it back in. (maybe that's not so bad...)

sharp zealot
#

yeah that would be weird

#

ok so maybe something like importDir?

thorn moat
#

that's the current PR. the part I like about fromFoo is it keeps us from being boxed in to one format, and aligns it with from. but I guess that argument goes for build too - would that be fromDockerfile? thinkspin

#

(OK with importDir, just exploring other options while an API breaking change window is opening)

thorn moat
#

that would be most ideal, really. maybe I should look into that instead... 🤔

sharp zealot
#

In the future if we do ship an ‘artifact’ type that can be either directory or file, we may not need two different calls at all…

rocky harbor
# thorn moat also, to be totally fair, there's always a chance that we can fix `import` perfo...

Just took a quick look at your oci layout PR, yeah the big bottleneck (IIUC) of "export back to the client, unpack, import back to the engine" would no longer be an issue. Sincesince the gql server is in the engine process, even if we still do the same "round trip" it's not actually going far (it will traverse a session that's actually just over a net.Pipe, so very hard to become a bottleneck).

The other thing we can do now is when we make solve calls, the result/reference has the same API as before but also can be turned into the underlying buildkit cache ref, which means you can just straight-up mount it and make direct syscalls to interact with it. I already used that in the PR to simplify the process of getting stdout/stderr after a failed exec: rather than using the shim as a proxy we just read the stdout/stderr files directly.

Not sure if that'd be useful in this context, but seems like it possibly could be

thorn moat
#

Cool, yeah this seems like it'll almost certainly help. I was also thinking we might be able to just import directly to Buildkit's content store, and use that as the OCI layout directly, so the oci-layout:// source is always a no-op. Not sure if there are dragons there

#

it already exposes its content.Store through the client interface, but it's read-only (sensibly). It was originally exposed for build history reasons iirc. (edit: actually it looks like it's a different interface anyway)

rocky harbor
thorn moat
#

I think it does, assuming it's same store. When you import a layout dir for example the first oci-layout:// call is slow, and the rest are immediate

# 
--- a/engine/server/server.go
+++ b/engine/server/server.go
@@ -101,8 +101,8 @@ func (s *Server) Run(ctx context.Context) (*frontend.Result, error) {
                                BuildkitClient: s.bkClient,
                                Platform:       s.worker.Platforms(true)[0],
                                ProgSockPath:   progSockPath,
+                               OCIStore:       s.worker.ContentStore(),
                                /* TODO:
-                               OCIStore content.Store
                                Auth     *auth.RegistryAuthProvider
                                Secrets  *session.SecretStore
                                */

that was easy (TODO: see if it actually works)

sharp zealot
#

btw, it would be really really useful to have a way to export a container as anything (directory or tarball) without requiring an export to the host filesystem. I'm guessing that's a separately annoying problem to solve, but thought I'd put it out there - people are asking for it

rocky harbor
chrome pilot
thorn moat
#

@rocky harbor Does this also fix the longstanding issue of Export/Publish/etc establishing a new session and possibly rerunning things (due to source refs resolving to new things)?

rocky harbor
rocky harbor
# thorn moat <@949034677610643507> Does this also fix the longstanding issue of Export/Publis...

Yeah it does because we just call the worker exporters directly now: https://github.com/sipsma/dagger/blob/07f666be1964900123ffe6588890536468b9479b/engine/session/manager.go#L187-L187

I would still greatly prefer upstream supported exports right in the gateway API (technically we use the "LLBBridge" api now, but it's pretty much the same thing, just slightly different layer in the abstraction cake). Then we could just pick everything up rather than have to do it ourselves, but yeah we at least have this much better option in the meantime now.

thorn moat
#

strangely the examples from the PR worked fine, can't see what I'm doing differently

rocky harbor
thorn moat
#

oh wait, they're not working now either. maybe I broke something.

rocky harbor
thorn moat
#

oh good, I remember that being borked at one point so that was the first thing I tried when I saw that error 😛

rocky harbor
#

maybe the go sdk generated code was out date

thorn moat
#

ah yeah that was it. I regenerated it on my local branch

rocky harbor
# thorn moat oh good, I remember that being borked at one point so that was the first thing I...

Yeah that should be de-borked now. Also btw, I skipped implementing namespacing for now to save time, so every command (and soon check, artifact) has its named stitched right under query, so there's annoying conflicts if you e.g. have commands named "lint" in separate environments. Let me know if that gets very in the way, I can bump priority on fixing if helpful for demoing, I don't think it's too bad to setup

thorn moat
rocky harbor
#

Feel free to push that to the zenith PR if you want

thorn moat
sharp zealot
rocky harbor
rocky harbor
#

@thorn moat I pushed a commit with a placeholder Checks API and support in the Go SDK: https://github.com/dagger/dagger/pull/5443/commits/51c40b82b0faf80d775a58da9c960f02aba19fd6

I also changed WITHCommand to be WithCommand_ in addition to adding WithCheck_, so it's at least slightly uglier pending the codegen fixes to make that work as intended 🙂

Haven't tried using it yet, but gonna try pulling in bits and pieces of your branch w/ the CLI updates + new universe envs and see what happens. Or actually if you already have a rebase of that branch on the zenith one, feel free to just push your commits to the PR and I'll go from there

thorn moat
sharp zealot
#

maybe it’s too much for thursday, but any chance we could support sub-checks, so that eg. progrock’s go test suite could be exposed both 1) as a single top-level check, and 2) as a list of individual sub-checks, one per go test? I’m happy to be the guinea pig and try implementing the top hat side (by generating gotestsum’s json file, parsing the result and plugging into the zenith checks API)

#

there’s a pseudo-code version of that in the draft schema/readme PR

#

it may seem superfluous but IMO it could be the difference between “meh just a complicated shell script wrapper” to “ok I want access to that API”

rocky harbor
# sharp zealot maybe it’s too much for thursday, but any chance we could support sub-checks, so...

I can't currently think of any reason that would be very much work on top of the current state of things, at least in terms of the API+sdk. Once I successfully run a single top-level check, I can look at adding subchecks to the api + go sdk. In the meantime, if you want to write the part that parses out the json file and calls to some dummy placeholder api, that sounds great, we'll need that code eventually so no risk of wasted time.

Just to double check, the API should let you list all checks, invoke individual subchecks and get individual subcheck results?

thorn moat
#

fyi: trying out the latest changes now, I noticed dagger do was printing cryptic "vertex is not in group?" errors, so I just fixed it in Progrock (it had to do with focus mode)

thorn moat
rocky harbor
thorn moat
# rocky harbor Nice! I have to run in a sec and don't have the types we use there offhand, but ...

They return a Container, so maybe we could model that as an artifact, as long as it can be parameterized sufficiently. Otherwise I figured it'd just be an API extension yeah. You can see the usage in the Base function in the previous link.

And sounds good! I may take a swing at it in the morning too, will let you know (update: I haven't - trying to figure out if services can benefit from the new architecture, instead)

rocky harbor
# thorn moat They return a Container, so maybe we could model that as an artifact, as long as...

Right, yeah it looks like we'd need to support

  1. Container return, which is trivial to support in terms of low-level plumbing since the code can handle any id-able return value pretty generically
  2. []string input, fairly easy in terms of go sdk updates. For the CLI i guess we could just accept repeated flags like dagger artifact --pkg foo --pkg bar
  3. ...ApkOpts, which requires a bit more work since we'll need the schema to include an input. Less clear whether we'd even try to expose that in the CLI, maybe that is just an api extension only thing.

For the api extension part, obviously codegen is the ultimate solution, but I realized the type-unsafe invocation builder api I added to command and check could in theory be made to support generic resolvers too. It would just be ugly, e.g. .SetStringSliceFlag("pkgs", []string{foo, bar}), .SetStructFlag("opts", map[string]any{...})

#

Starting on the CLI bit now!

thorn moat
# rocky harbor Right, yeah it looks like we'd need to support 1. `Container` return, which is t...

Thinking about it more, to me these APIs feel more like extensions. Closer to scripting/helpers than making use of an 'artifact.' Maybe because the result of these APIs is rarely the final value you use; all the call sites actually chain more project-specific things after it, like installing gotestsum. As for ApkoOpts, that was an attempt at modeling the ideal codegen'd API, so maybe the actual implementation would just have all that as params, assuming there's a way to mark optional params.

rocky harbor
#

Also, @thorn moat @sharp zealot how about we check in at 1pm PT to finalize what we want to show for sure in the demo tomorrow based on what's working so far? I have a hard stop from 6-10 PT tonight, so want to get that straightened out earlier than later.

#

Flexible on 1pm too

sharp zealot
#

2pm?

rocky harbor
rocky harbor
#

Pushed barebones dagger check support: https://github.com/dagger/dagger/pull/5443/commits/8d38eb0e8dc9201591aba2f9c3828650e3e9e57a

Was pretty straightforward, mostly the same as commands except some trickier handling around the fact that a check entrypoint returning an error needs to get converted into a CheckResult rather than propagated as an error per-se all the way back to end clients.

Examples:

  • Listing checks: ./hack/dev ./bin/dagger check list -e ./universe/dagger
  • Passing check: ./hack/dev ./bin/dagger check -e ./universe/dagger python-lint
  • Failing check (warning: there is a ton of output): ./hack/dev ./bin/dagger check -e ./universe/dagger engine-lint

TODOs:

  1. The output of check list looks a little weird. Also not delighted by the output of check, especially when there's a ton of output, but it's maybe okay enough for now.
  2. Integrate with the demoenv, I just used the existing lints in universe/dagger to start. I cherry picked your commits but a lot of the files needed updates to compile, so saved time by just rm'ing for now.

@thorn moat Do you want to try that out and then see what you think makes sense to polish some more for the demo?

My next step is to try out subchecks quickly. After that, artifacts, which I'm hoping is even easier than checks? @thorn moat feel free to try out artifacts too if you have time, I think the commit for adding check support is a good skeleton for how to add new entrypoints

thorn moat
#

@rocky harbor Just hit an interesting scenario: 1. run something that uses Container.import, 2. run buildctl prune, 3. run that thing again, and you'll hit an error where oci-layout:// refers to an unknown digest. This happens because Container.import caches in-memory, and now that spans across Dagger runs. I think the fix for now is "don't do that" but in general the global caching is something to watch out for now.

rocky harbor
thorn moat
#

I've been trying out some more polish for the focus-mode TUI: https://asciinema.org/a/6YyLozDrqW38U1ZLgLYy6JNxq

  • Now shows the last line of output from whichever vertex is running.
  • For vertexes that have sub-tasks (e.g. image fetching), a single progress bar is shown summarizing all inner progress bars (by just summing current/total), and the last active task is shown
  • Got rid of the info section and help text to make room. Info is still shown at the end.
  • Current vertex name is in bold instead of in yellow, for better readability.
#

will push soon and try it out on top of the checks stuff 🙌

#

@rocky harbor fyi, pushing a tiny go.mod bump to your branch for ☝️

thorn moat
rocky harbor
thorn moat
#

@rocky harbor re:

// TODO: why doesn't cmd.Printf work?
// cmd.Printf("Check %s: %t\n%s\n", envCheckName, success, output)

the session's *progrock.Recorder wasn't being added to the ctx, so that (along with other places) was logging to the void. Working on a fix

#

Some of the progrock recorder/writer setup is a bit clunky, related to the way we collect pipeline labels, since those have to be on the root group, which is automatically created when you construct the Recorder. Would like to clean it up someday so the Recorder can be set up in the outermost layers somehow.

thorn moat
#

hmm, checks interact kind of strangely with WithFocus(); you get duplicate output if you use it. I guess we would suggest against using WithFocus() for checks, except for troubleshooting?

#

Worth noting that in this case the top-hat (me) added WithFocus() to the Gotestsum helper, so sometimes you can't tell how it's going to be used

#

also, one perk of using WithFocus() is that you can see the logs streaming, rather than having everything buffered up into the string return value

rocky harbor
#

Joining teams in 1 min

thorn moat
#

oh hey, we already have $DAGGER_PROJECT 👍

rocky harbor
#

@thorn moat For the demo script, I guess for now it would make sense just assume the current set of features, and then provided nothing unexpectedly explodes I should push the subcheck support in the next hour or two and you can update to include that?

thorn moat
#

makes sense!

rocky harbor
thorn moat
#

@rocky harbor Not sure if I'll be able to env-ify all the code under ci/pkgs/ actually, even the Go stuff. You need to be able to pass in a base image, which doesn't seem to really fit into our model at the moment. thinkspin Unless you had something in mind there. (Example: https://github.com/vito/progrock/blob/b4bf857f7c958b51e453e285a04a61567c49db3d/ci/ci.go#L30)

I mean, I can add a progrock environment, but that feels like it sends a weird message since it's so specific.

At least for the first part of the demo, this is fine, since I can just set DAGGER_PROJECT=./ci and it's all invisible anyway

rocky harbor
thorn moat
#

yeah, I figured the Go parts of ci/pkgs/ could be the start of the canonical Go environment in universe, but in practice I also expect to maintain an environment for Progrock anyway that builds upon it. I think it's TBD how these "canonical <X language>" environments pan out

#

but at least for really simple Go packages it'd be nice if you could just dagger use go and dagger check away

#

whether that's a real useful thing and not just a parlor trick, we'll see 😛

rocky harbor
# thorn moat yeah, I figured the Go parts of ci/pkgs/ could be the start of the canonical Go ...

Okay right, so I think it would be okay then to have your progrock specific env for now. Like you said, shouldn't really impact the first part of the demo. And then when we show the code, you could just call directly to the "go environment", which is actually just a separate package under universe at the moment that you are calling directly, as opposed to through the APIs. That will get the idea across enough to hold everyone over for future demos I think?

#

Besides entrypoints not currently covering accepting inputs like container, we also still haven't split out "environment" and "context" or added WithExtension yet, which you'd need for the "no-code" version

thorn moat
#

ah got it - yeah, I can try that. Hopefully I don't hit a dagger <=> progrock dependency cycle not_sure_if

#

if that happens i can just demo using Booklit instead 😛

#

actually Booklit might be a less confusing demo anyway, since it also has a binary to build

#

I'll make sure that works too, maybe we can show off two repos

#

@rocky harbor pushed a couple of small commits fyi, one to change Container.import to use a separate content.Store (so it doesn't flake during the demo), and just some more UI polish

#

pushed universe packages too, universe/{goenv,apkoenv,nixenv} - ain't the prettiest names, but go is a reserved word, and these will be replaced by real envs at some point anyway (which will be package main think_about_it)

sharp zealot
#

@thorn moat will be setting up my demo env soon, where should I start?

thorn moat
#

1 sec!

#

still need to actually add the lint check, could have sworn I implemented that, maybe I dreamt it...

sharp zealot
#

@thorn moat so I should not use your dagger branch, correct?

thorn moat
sharp zealot
#

hummm..

The path /home/sipsma/.docker is not shared from the host and is not known to Docker.
rocky harbor
#

I can push a fix so it finds your homedir instead of hardcoding mine

#

I'm confused how @thorn moat didn't hit though

thorn moat
# 
❯ ls -al /home/sipsma/
total 12
drwxr-xr-x 3 root root 4096 Jul 11 10:22 .
drwxr-xr-x 4 root root 4096 Jul 11 10:22 ..
drwxr-xr-x 2 root root 4096 Jul 11 16:55 .docker
rocky harbor
#

oops

sharp zealot
#

😄

rocky harbor
#

sorry 😅

sharp zealot
# 
Stderr:
go: downloading github.com/juju/ansiterm v1.0.0
go: downloading github.com/rs/zerolog v1.29.1
go: downloading github.com/spf13/cobra v1.6.1
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/shurcooL/graphql v0.0.0-20220606043923-3cf50f8a0a29
go: downloading github.com/lunixbochs/vtclean v1.0.0
go: downloading github.com/mattn/go-colorable v0.1.13
go: downloading golang.org/x/tools v0.10.0
go: downloading github.com/99designs/gqlgen v0.17.31
package github.com/dagger/dagger/cmd/dagger
        imports github.com/dagger/dagger/core
        imports github.com/dagger/dagger/core/reffs
        imports github.com/dagger/dagger/engine/buildkit
        imports github.com/dagger/dagger/engine/session
        imports github.com/moby/buildkit/exporter
        imports github.com/moby/buildkit/cache
        imports github.com/moby/buildkit/snapshot
        imports github.com/containerd/containerd/snapshots/overlay/overlayutils: build constraints exclude all Go files in /go/pkg/mod/github.com/containerd/containerd@v1.7.2/snapshots/overlay/overlayutils
Please visit https://dagger.io/help#go for troubleshooting guidance.
rocky harbor
#

I didn't mean to infect your machine with my name

thorn moat
#

all good, I'll disinfect it

rocky harbor
#

obviously that'll have to be the long term fix either way

sharp zealot
#

I'm running ./hack/dev though, wouldn't that build everything in a container?

rocky harbor
rocky harbor
#

and that has to be built for darwin if you are running there

#

i'm looking to see how much it'll take to disentangle

sharp zealot
#

I'm assuming a simple go build won't work?

rocky harbor
#

I think it might be easy to workaround, trying something quick

#

hm okay I got rid of the dep you were hitting, but that just revealed another path. I'll see how long the whackamole lasts

sharp zealot
#

ok, sorry for adding that complication

rocky harbor
# sharp zealot ok, sorry for adding that complication

no worries, it was gonna happen someday anyways! Took about 7 rounds of whackamole and a lot of copying, but I got the build working again. Pushed a fix

However, I think I must have copied something incorrectly in the process because when I'm trying to run the commands from before it's freezing. I'll keep looking but @thorn moat if you're around would appreciate a second pair of eyes, I'm sure it's just some dumb oversight

thorn moat
#

I'll pull and try everything out 👍

#

seems fine here

#

what command freezes for you?

rocky harbor
#

oh weird, I was doing ./hack/dev ./bin/dagger check -e ./universe/dagger python-lint

#

@sharp zealot worth a try again on your end

sharp zealot
#

on it

thorn moat
#

@rocky harbor hm, I can repro the hang in Booklit zenith branch and in Dagger, investigating

#

oh, it eventually errored:

Error: connect: make request: Post "http://dagger/query": command [docker exec -i dagger-engine.dev buildctl --addr=unix:///run/dagger/server-vkixxy6or4eapjzxb4z97k0x
7.sock dial-stdio] has exited with exit status 1, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=error: dial uni
x /run/dagger/server-vkixxy6or4eapjzxb4z97k0x7.sock: connect: no such file or directory
sharp zealot
#

Can I wrap ./hack/dev in a dagger run or is that asking for trouble? 😇

#

Ok I have a working binary, thanks

thorn moat
#

@rocky harbor ah, the shell what worked was running the old dagger binary - can confirm everything fails with the previous error

sharp zealot
#

I don't have direnv installed, how crucial is that step?

thorn moat
#

pretty crucial - but you could maybe just source .envrc

#

that's the step that lets you say dagger check foo instead of ~/src/dagger/hack/dev ~/src/dagger/bin/dagger check --env ./ci foo

sharp zealot
#

ah I see 🙂

#

I'll make it my excuse to finally learn it

thorn moat
#

it's a pretty tiny tool, one of my favorites really. does one thing, does it well

sharp zealot
#

Is DAGGER_SRC_ROOT only needed for setting $PATH? Or also used somewhere outside that .envrc?

thorn moat
#

it's for $PATH and the DAGGER_CLI_BIN env

sharp zealot
#

I literally just installed direnv. All I need to run is direnv allow?

#

I don't understand how it can affect my shell's environment without a builtin

thorn moat
sharp zealot
#

ah! thanks 🙂

#

beautiful

#

Is export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine.dev important?

thorn moat
#

yup

sharp zealot
#

Ok. so far dagger check list hangs for a while then fail with a docker error that seems related to engine provisioning

#

(will copy-paste in a bit)

#

waiting for it to timeout again

# 
 % dagger check list
• Cloud URL: https://dagger.cloud/runs/b4db1f8f-fd3d-4901-a31a-db10a2cbae86
• Engine: b730877185eb
Error: connect: make request: Post "http://dagger/query": command [docker exec -i dagger-engine.dev buildctl --addr=unix:///run/dagger/server-rmgox0zau9n65q7xtvxa1upoj.sock dial-stdio] has exited with exit status 1, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=error: dial unix /run/dagger/server-rmgox0zau9n65q7xtvxa1upoj.sock: connect: no such file or directory
rocky harbor
#

lol yeah that's where we are too, I am truly confused as to what could have changed since all I did was copy paste some stuff to different packages. Hoping it's not related to some init() or something somewhere

#

i'm going back to the previous commit and running one by one with each change

thorn moat
#

looks fine on further analysis...

rocky harbor
#

Yeah I can't see anything wrong either...

#

I have to get ready to leave at 6:10. It's to go see a show I bought tickets for for my girlfriend for her birthday a few months ago, so non-negotiable :-). Will keeping looking until then

thorn moat
#

I'll keep looking too 👍

rocky harbor
#

oh... I think I may have found it, one min

#

@thorn moat can you double check what I just pushed fixes it for you too

thorn moat
#

yup! 🎉

rocky harbor
#

@sharp zealot freeze should be fixed, hopefully nothing else darwin-related remains

#

@thorn moat gotta go, I was very close to subchecks, was just implementing the part that ran checks w/ subchecks in parallel, i'll push my progress to a separate branch in case you happen to have time and want to finish it. But no worries if not since it's super late for you obviously.

I'll be back at 10, think I can finish it then. I've also been waking up super early lately, so I'll be online 7:30ish my time tomorrow morning for any last minute coordination

thorn moat
#

sounds good! I'll send a ping if I jump into it, but odds are low 😛

rocky harbor
thorn moat
#

@sharp zealot added lint to Progrock and Booklit, and docs for each command:

❯ dagger check list
[0.97s] list checks
check name  description
unit        Unit runs all Go tests.
lint        Lint runs golangci-lint against all Go code.
• Engine: 51021dfa219a
sharp zealot
#

thanks! sorry I’m on family break, will resume in 1h or so

thorn moat
sharp zealot
#

back at it

#

@thorn moat when I run dagger check list is immediately "bricks" my terminal - cursor disappears, all inputs ignored etc. This is while it times out as described above. Was wondering if that's expected

#

also still stuck at: Error: connect: make request: Post "http://dagger/query": command [docker exec -i dagger-engine.dev buildctl --addr=unix:///run/dagger/server-j6kxgoz3t8ecp1t0t3hqby4jz.sock dial-stdio] has exited with exit status 1, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=error: dial unix /run/dagger/server-j6kxgoz3t8ecp1t0t3hqby4jz.sock: connect: no such file or directory

thorn moat
#

The timeout should be fixed now, if you pull and re-run hack/dev (and pull in Progrock/Buildkit too)

sharp zealot
#

I thought I just did that, but maybe did somethign wrong. trying again

#

This commit?

commit 867b364b9307dc45244310f2b34f27b2b2e9190a (HEAD, sipsma/zenith)
Author: Alex Suraci <alex@dagger.io>
Date:   Wed Jul 12 23:24:38 2023 -0400

    engine client: faster retries
    
    Signed-off-by: Alex Suraci <alex@dagger.io>
thorn moat
#

yup

sharp zealot
#

and pull in Progrock/Buildkit too)

--> not sure what that step is

thorn moat
#

oops I meant Booklit there - just the repos where you'll be running dagger commands

sharp zealot
#

Ah ok - possible that not having the latest progrock explains still having the timeout?

thorn moat
#

what does which dagger say?

sharp zealot
#

it's the right location (inside the dagger repo)

#

I'm rebuilding just in case

#

is it safe to just go build ./cmd/dagger to save time, or important to go through the whole ./hack/dev?

#

Ah, rebuilding failed:

Error: input:1: pipeline.pipeline.container.from.withExec.withEnvVariable.withExec.withWorkdir.withDirectory.withMountedCache.withExec.withMountedDirectory.withMountedCache.withEnvVariable.withEnvVariable.withExec.file process "go build -o ./bin/dagger -ldflags -s -w ./cmd/dagger" did not complete successfully: exit code: 1

Stdout:

Stderr:
package github.com/dagger/dagger/cmd/dagger
        imports github.com/dagger/dagger/engine/client
        imports github.com/dagger/dagger/engine/server
        imports github.com/dagger/dagger/core
        imports github.com/dagger/dagger/core/reffs
        imports github.com/dagger/dagger/engine/buildkit
        imports github.com/dagger/dagger/engine/session
        imports github.com/moby/buildkit/exporter
        imports github.com/moby/buildkit/cache
        imports github.com/moby/buildkit/snapshot
        imports github.com/containerd/containerd/snapshots/overlay/overlayutils: build constraints exclude all Go files in /go/pkg/mod/github.com/containerd/containerd@v1.7.2/snapshots/overlay/overlayutils
Please visit https://dagger.io/help#go for troubleshooting guidance.
exit status 1
~/dev/dagger/dagger %
#

maybe it had failed the first time and I hadn't noticed - would explain the timeouts not going away

thorn moat
#

o_O that's back to the error from before the darwin fix?

sharp zealot
#

looks like it

#

trying again after git clean

#

there was a universe.tar lying around, with a little luck that was the issue

thorn moat
#

I think there's just more moles to whack, working on a fix

sharp zealot
#

god I am such a robin hood hat

thorn moat
#

@sharp zealot ok try pulling

#

I think you'll need a full ./hack/dev, but you can check with go build first

sharp zealot
#

pulling

#

Would be awesome if even the git jugglery could be daggerized

#

This whole scenario we're going through right now, feels like a good use case for a standalone env 🙂

#

so instead of telling me "try pulling again, and re-run the steps on the new commit", hoping I don't get it wrong - you could just say "refresh the artifact" or something like that

#

OK it built successfuly this time!

#

yay no more hanging either!

thorn moat
sharp zealot
#

Moving to next problem 😛

# 
Error: failed to get environment commands: input:1: environment.load failed to load environment: failed to get runtime container for schema: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out: ``
#

I'm guessing my hosts's docker config.json ends up mounted into a container or something like that?

thorn moat
#

maybe do a docker login?

#

I think it reads from that file

sharp zealot
#

yeah tried that, didn't fix it

thorn moat
#

hmm. do you have docker-credential-desktop?

sharp zealot
#

yes 🙂

thorn moat
#

(wondering if $PATH is getting unset/reset somewhere)

sharp zealot
#

my $PATH is looking weird, but can't tell for sure that it's because of this

#

/Users/shykes/dev/dagger/dagger/bin:/Users/shykes/.docker/bin:/Users/shykes/.docker/bin:/Users/shykes/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/shykes/.docker/bin:/Users/shykes/bin:/opt/homebrew/bin:/opt/homebrew/sbin

#

never seen that before 😄

thorn moat
#

theoretically it should be mounting ~/.docker to /root/.docker in the engine container - you could exec into it and see if it made it in?

#

the first entry is added by the .envrc, but yeah some of that repetition is a little funny

sharp zealot
#

that's the problem exactly

#

my ~/.docker has a config that says "use this binary helper to authenticate" but it's a mac-only helper

thorn moat
#

lol, makes sense

sharp zealot
#

sadly you can't trust .docker in this scenario

#

we had this problem a long time ago and found a solution that we liked - which of course I can't remember at all

thorn moat
#

hmm is there some way to skip the cred helper and have docker login just write a plain old token?

#

to config.json

sharp zealot
#

are you actually trying to mount .docker/config.json?

thorn moat
#

yeah

#

that's my guess, since that's how all this behaves on Linux (without Docker for Desktop I suppose being the difference)

sharp zealot
#

why do we need a containerized docker client in the first place?

#

I'm a bit lost

thorn moat
#

this is an artifact of the session-frontend refactor, there's probably a TODO in there to move this part back to client-side somehow

sharp zealot
#

ok

#

man I wish I could remember our fix, it was exactly this problem

#

I think I'm down to crafting a config.json manually using creds copy-pasted from mac os keychain

#

ok I've done that, testing now..

rocky harbor
sharp zealot
#

weird, I definitely succeeded in fixing my docker/config.json... but I'm still getting the same error

# 
 % dagger check list
[0.41s] ERROR dagger check list
[0.00s] connecting to Dagger
[0.00s] loading environment
[0.41s] ERROR list checks
[0.24s] ERROR resolve image config for docker.io/library/golang:1.20-alpine
• Cloud URL: https://dagger.cloud/runs/3f252714-c328-4d1c-8ce3-66af6dca036a
• Engine: 4b7070b6a638
Error: failed to get environment commands: input:1: environment.load failed to load environment: failed to get runtime container for schema: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out: ``

Please visit https://dagger.io/help#go for troubleshooting guidance.
~/dev/progrock % cat ~/.docker/config.json | grep docker-cred
~/dev/progrock %
#

there's definitely a base64-encoded password in there 😛 yay

thorn moat
#

did the change propagate into the container? 🤔

sharp zealot
#

looks like it didn't. But I don't know anything about that container's lifecycle

#

should I nuke something long-running?

thorn moat
#

./hack/dev again should work

rocky harbor
#

I have to take the trash out, I’ll see whether reincorporating auth attachable is quick or not after

sharp zealot
#

@rocky harbor I don;t think you need to worry about it for the demo at least, I think we're close

rocky harbor
sharp zealot
#

Hurray!!

 % dagger check list
[2.92s] list checks
check name  description                                                                                               
unit        Unit runs all unit tests.                                                                                 
lint        Lint runs golangci-lint against all Go code.                                                              
• Cloud URL: https://dagger.cloud/runs/7d77143b-6d2f-4593-8d26-ed9f1dfbcbd7
• Engine: 02ffb188d9f8
rocky harbor
thorn moat
rocky harbor
thorn moat
#

haha, I want to fix that, they used to show sha256 digests but it broke

sharp zealot
#

is linter check supposed to have tons of errors?

thorn moat
#

yes 😅

#

both progrock's and booklit's fails, I could fake one out if you want

#

I don't think we have a run-all-checks command at the moment though so it might not make much difference (vs just showing passing tests)

sharp zealot
#

it's fine as it is, just making sure those were errors I should be seeing 🙂

thorn moat
#

fixed booklit's linter issues if you want to pull either way. 😛 there weren't too many

sharp zealot
#

@thorn moat no .envrc in booklit, expected?

thorn moat
rocky harbor
#

@sharp zealot @thorn moat Got subchecks working, but just pushed to a separate branch for now to keep demo safety maximized, probably a bit too late to incorporate, but maybe we can sync up tomorrow morning, will merge it into the pr after either way https://github.com/sipsma/dagger/tree/zenith-subchecks

./hack/dev ./bin/dagger check -e universe/dagger lint automatically runs all the sdk lints as subchecks in parallel in individually cached execs and shows the results.

rocky harbor
#

Just want to confirm this as a general outline for the demo:

  1. @sharp zealot pitches and shows robinhood cli-based demo
  2. Q&A
  3. @thorn moat shows the progrock+booklit env code (with caveats that the DX there is extra WIP still). I'll chime in as needed too of course but think it makes sense for you to drive this part Alex since you wrote that code.
  4. Q&A

Sound good? Obviously if questions pull in a different direction we can improvise too. Also, feel bad I'm just running support this time, I'll drive parts in future demos so we can share the load.

thorn moat
rocky harbor
#

I'll go look through the progrock/booklit code quick to refresh

thorn moat
#

I'll hop in Lounge if you wanna chat over anything

#

Thanks a bunch for covering 🙏

rocky harbor
sharp zealot
#

@rocky harbor as a stretch, if you have a terminal with your latest sub-checks version ready to go, can't hurt to have that ready to go

#

I was really hoping to have it ready, but had family emergencies so couldn't do it

rocky harbor
rocky harbor
thorn moat
#

nicely done 🎉

sharp zealot
#

Lots of great questions 🙂

spiral whale
#

That was a great demo and most exciding dagger feature

sharp zealot
spiral whale
#

Great work guys. I can't wait to use zenith as part of a gale . This saves me. from lot's of nightmares

thorn moat
hoary geyser
#

IDK how familiar folks here are with zuul-ci, but that community would align with what you're trying to do with Zenith.
Zuul can be implemented with different components, but the core benefits are:

YouTube
OpenInfra Foundation
Project Gating with Zuul

Zuul is a Project Gating System, or a CI/CD system like no other.

It brings multiple projects together to test the future as if it were the present.

It spans code review systems and clouds. It uses Ansible for orchestration and has no single points of failure. It tests some of the largest Open Source projects in the world and can run on your...

▶ Play video
#

@dense canyon made a little joke about dagger-ci but I wonder if something like that is possible? I would be open to hosting a dagger CI service.

sharp zealot
#

it’s 100% possible 😁

#

(or will be, with Zenith shipped)

#

CI is really a software platform + batch compute infrastructure. They are tightly coupled but really shouldn’t be

thorn moat
sharp zealot
#

With dagger/zenith it becomes possible to decouple them.

rocky harbor
#

FYI @thorn moat Pushed the subcheck commit to the Zenith PR and I'm now gonna go rebase it on main so the focus-related commits aren't there anymore to simplify a bit, just in case you have anything in flight atm.

Doing that as part of trying to plot out how to get everything incrementally merged since the Zenith PR is now at +10,064 −5,241 lines changed 😵‍💫. I'll start a thread about that after I finish the rebase

thorn moat
rocky harbor
#

Zenith Development Planning

thorn moat
#

I've noticed when I run buildctl prune now it seems to keep a lot of data still, only pruning a relatively small portion. I wonder if the long-lived session is keeping refs alive somehow?

rocky harbor
# thorn moat I've noticed when I run `buildctl prune` now it seems to keep a lot of data stil...

Yep, though it's actually not the long-lived session exactly but instead the fact that our buildkit.Client needs to release all the refs that get solved through it. The closest thing to that object in buildkit's codebase is this BridgeClient, which has this discard method, which we'll need to add the equivalent of: https://github.com/sipsma/buildkit/blob/cdf28d6fff9583a0b173c62ac9a28d1626599d3b/frontend/gateway/forwarder/forward.go#L209-L209

#

That reminds me we need to add test coverage for this though. Could even just be that when any integ test suite finishes it calls prune on the engine and then fails if anything isn't releasable. That's what buildkit's own integ tests do

thorn moat
#

Yeah that sounds like a good idea

rocky harbor
# thorn moat Yeah that sounds like a good idea

Pushed a commit with a stub of the code and TODOs so it's harder to lose track of: https://github.com/dagger/dagger/pull/5415

Btw I backported the relevant stuff in the Zenith PR to that one, except for the oci store things since it's being re-worked. Feel free to push that (or anything else relevant) to that PR. Rebasing the Zenith one on it is gonna be painful until merged but we'll be able to merge faster by separating out

#

@thorn moat re: CacheKey #1129130729469657218 message

Nice! Yeah totally agree that the digest after resolving sources is more often what we want rather than the llb vertex digest.

One thing to be careful of is that the digest you get when calling op.CacheMap is a digest of that op only, it doesn't include the (recursive) digests of any inputs. Still useful, but not always what you'd expect

thorn moat
rocky harbor
# thorn moat right - that was a bit confusing. That's handled by this right? <https://gist.gi...

Oh I missed that part, but while that would make sense, it also is sadly more complicated. Deps is a slice that has to have the same length as the number inputs and configures how the digest of the input should be calculated, but it doesn't actually have the digests right after you call CacheMap. E.g. see merge-op here: https://github.com/sipsma/buildkit/blob/cdf28d6fff9583a0b173c62ac9a28d1626599d3b/solver/llbsolver/ops/merge.go#L51-L51

Selector specifically is used when you want "something" specifically from the input rather than the input as a whole. So when you mount a subpath of an input into an exec rather than "/", Selector will be set to a digest of the subpath (approximately, going off memory here).

#

But there's also a way to specify specific callbacks for how the digest gets calculated

thorn moat
#

oo, interesting, that explains some things, thanks!

thorn moat
#

@rocky harbor in merge-op's case (your link above), is every field zero-value to indicate that the outer Digest is sufficient as a cache key? (edit: oh, I just read the later part again, it does sound more complicated. hmm)

sharp zealot
#

Dear Santa. I've been a good boy this year. I would love for dagger query or dagger listen to support Zenith so that I can show all my friends the cool graphql queries they can do.

Thanks Santa!

Sincerely,
Solomon

rocky harbor
# thorn moat <@949034677610643507> in merge-op's case (your link above), is every field zero-...

Yes, unfortunately the code in buildkit around all this is extremely complicated, you'd expect that at some point buildkit calculates a digest that combines the digest of the op itself with the digests of the inputs into one hash (similar to git commits, but for a DAG), but after reading through all that code for magicache stuff, to my knowledge it never does that. What it actually does is... hard to describe laughcry

rocky harbor
rocky harbor
# sharp zealot Dear Santa. I've been a good boy this year. I would love for `dagger query` or `...

It was really easy, pushed support for query+listen here to zenith here: https://github.com/dagger/dagger/pull/5443/commits/b73bfa1a892b84454885e049b94e559043da0f21

Example: echo '{pythonLint{success}}' | ./hack/dev ./bin/dagger query -e ./universe/dagger

The only hard part I got stuck on for a while was that I forgot the environment load API is lazy and was confused why nothing was loading 😂 I guess we need sync there too sometime.

sharp zealot
kind carbon
#

@rocky harbor how are thinking on supporting API namespacing?

sleek nest
#

If anyone is following this channel, but wants to learn more about the Project Zenith, check out this demo from the last community call - #1129800905798193263 message

sleek nest
honest hearth
#

I got the chance to go back and watch the rest of the community call (had to drop out early for another meeting). Nice work! Some stream of consciousness thoughts and questions here:

Checks: Are we thinking something like a CheckResult data structure that stores individual test results for various checks? Seems like it could be neat. Imagine a dagger extension that you can install in your pipeline that takes pytest or junit output and stores it in a DaggerCheckResult object, which gets first class visualization and control flow treatment. And otherwise if you have some custom custom test suite that doesn't write to a standard like junit, you are given the primitives to easily write to this object for Dagger to store and conflate and visualize

Environments: Based on what Erik mentioned about how to distribute the environments, my naïve first thought is that it makes more sense for end users to be generating their own client with codegen, since it means that it offloads the responsibility of hosting and distributing every single permutation of dependencies of every single environment. picture running dagger env build when you need to update your env dependencies to produce the client that you need for your code, as Erik mentioned. It feels more scalable, at the cost of pushing slightly more work to end users. Otherwise, adding more SDK's and hosting packages for them becomes more difficult as "universe" hosts more environments. Feels like misaligned incentives

sharp zealot
sharp zealot
# sharp zealot checks: yes that is the idea 😁

@honest hearth checks are nested. So “running the python linter” would be a check. But you could refine the implementation to extract a list of each individual linter test as a sub-check. Same idea for any other test suite that supports some sort of structured report output (so, all of them 😁)

kind carbon
#

Python DX

thorn moat
#

@rocky harbor Throwing an idea out there, trying to de-risk the stuck Buildkit PR (going on a month now 😬). What if I replace the whole "session attachable-based dynamic network config" part with just adding llb.DNSConfig instead, and align its behavior with llb.ExtraHosts? So it won't count towards the cache key in Buildkit1, but we would still have to remove it when we're calculating a stable digest, using the same mechanism we just used for stripping llb.SessionID from local sources.

The difference on Dagger's side would be that now our ContainerID/FileID/DirectoryIDs would have their session-specific search domain baked in to every WithExec call, and every HTTP or Git call that uses a service. I think that can work as long as any embedded service bindings also retain the same domain, so that they use it when they're lazily started regardless of which Dagger session starts them.

thorn moat
#

Alternative services v2 ideas

sharp zealot
#

Thought experiment: what would it mean to package something like this exciting new AI playground (one of many) as a Dagger environment? https://twitter.com/mascobot/status/1681334370584637441

Marco Mascorro (@Mascobot)

✨NEW LAUNCH! LLaMA2 chat API & open-source playground💫:

We're releasing tools that make it easy to test @meta's latest LLM & add it to your own app with @replicatehq.

Playground: https://t.co/YRxDyl5fVW
Live chat API here: https://t.co/TFUOsy44oT

Repos & instructions below:

Likes

471

Retweets

108

Twitter ·
sharp zealot
strange arch
#

Hello, I’m testing entrypoints with Go SDK v0.7.3, and I’m wondering how I can export a directory (containing my build artifact)? I’ve tried to do mydir.Export(ctx, exportPath) in my target, but it doesn’t seem to work. If I return a *dagger.Directory from my target, its content is put at the root of my repo, not in the subdir I’d like to. Also, I’m wondering how I can remove this directory in clean target? Using os.RemoveAll doesn’t work (I guess it’s expected since it runs in a container).

thorn moat
#

Good question about cleaning it up, I don't think we've talked about that at all. 🤔 First thing that comes to mind is having some way to list files to clean up, but this definitely needs more thinking if we want to support it. cc @rocky harbor

strange arch
thorn moat
strange arch
rocky harbor
# strange arch Hello, I’m testing entrypoints with Go SDK v0.7.3, and I’m wondering how I can e...

Yeah I don't think this is quite possible yet, but the underlying plumbing for "applying deletes" to a host dir exists already, we'd just need to find a way to expose that.

The thing I'd want to be careful about is that it's not easy to accidentally write code that e.g. deletes all your source code or something. Like if you were using this to build+export a binary based on local source code changes, it would be extremely sad if you made a typo somewhere and accidentally overwrote all your unstaged changes and lost them. So just need to think about how to make the DX safe in that sense.

rocky harbor
# strange arch Hello, I’m testing entrypoints with Go SDK v0.7.3, and I’m wondering how I can e...

Also, thank you for trying this out! As you can tell it's a very early state, but appreciate you letting us know how it goes, happy to hear any more feedback any time.

For mydir.Export(ctx, exportPath), as you noted all that code runs in a container, so when you call export you are actually exporting to the filesystem within that container. This can be useful for certain use cases, but it definitely isn't obvious enough right now.

strange arch
#

How do optional parameters work? From a glance at the code, it seemed to me that using pointers would do the trick, but pointers aren’t affected even when specifying the parameter on the command line. Also, I saw a weird thing where parameters which name contain upper letters were considered mandatory, whereas they were optional when using only lower letters. It seems so strange to me, that I’m wondering if I did my tests correctly 🤔

rocky harbor
# strange arch How do optional parameters work? From a glance at the code, it seemed to me that...

Yeah at the moment all parameters are essentially optional, if you don't provide them when calling dagger do they should just be set to e.g. an empty string. We have talked about making that more explicit such that you could mark parameters as optional or required (and probably also define default values), but doesn't exist yet.

We are currently in the midst of a big refactor of all of this, which won't yet include support for all of that but should make adding support a lot easier.

Also, I saw a weird thing where parameters which name contain upper letters were considered mandatory, whereas they were optional when using only lower letters. It seems so strange to me, that I’m wondering if I did my tests correctly
That in particular does indeed seem very strange, highly likely it's just a bug on our part 😄 . If you can share the code you're using and how you're invoking it I can confirm. Even though we're refactoring it's highly possible that bug would still be there, so would be good to identify and fix anyways.

sharp zealot
#

Quick thoughts on terminology:

  • cli perhaps a better entrypoint name than tool. Even though it's annoying to capitalize; it's still less ambiguous. (low conviction)

  • functions might be a better entrypoint name than graph. Like "here are composable functions which you can use for scripting your own pipelines". What bothers me with "graph" is that it makes sense once you explain it (kind of), but it doesn't have a name for the "things" that you're getting.

dusky magnet
#

Hi. Been a while since i looked at what's coming to dagger. I might get involved into doing "real" CI/CD stuff for some teams, and i would love to try bringing dagger into it (it would be a little hard knowing our client x) ). I would love to redo some dagger related code and try zenith UX at the same time, is there a way to start ? Also would love to try the dagger cloud alpha 😄

wintry prism
sleek nest
thorn moat
#

Too bad we have to implement Zenith before we can use it. 😭 Would really like a high-level way to navigate test output right now.
Currently:

image.png
strange arch
strange arch
strange arch
#

Hello, how are Docker credentials handled with entrypoints? I guess that, contrary to the “classic use” of SDKs, host credentials aren’t accessible, right? Then how to pass secrets to entrypoints? Again, I guess that host environment variables aren’t accessible.

sharp zealot
strange arch
strange arch
#

Hello, will it be a valid use case to use Dagger entrypoints to start long-running services (like, for example, hugo server, which watches filesystem changes and generates website on the fly, exposing it on an HTTP server on port 1313)? If so, what would be the UI to stop these services (pressing q, I guess, though it doesn’t work in some situations for me)? Also, I’m pretty sure I saw a discussion on service port exposition on host, but I can’t find it anymore with the search, could you point me to it please?

thorn moat
#

I'm planning to work on host => container networking soon (as in within the next days/week)

strange arch
#

Also realizing probably why pressing q doesn’t work in my case (using a shell script as an entrypoint which doesn’t pass signals to child, I’ll try using an exec instead).

thorn moat
strange arch
#

Curious to hear if you find the reason

rocky harbor
#

Just pushed barebones support for shell entrypoints in environments to the Zenith branch: https://github.com/dagger/dagger/pull/5443

It just barely works at the moment and only on Linux (not sure what's different w/ MacOS, maybe some TTY thing but haven't investigated yet), there's a long awkward silence while things build before you drop into a shell, you need to ctrl-C after you ctrl-D to fully exit it, etc. etc. etc. but if you want to try pull that branch down and run ./hack/dev ./bin/dagger shell --env ./universe/dagger dev-shell

You can change the container the shell executes in by just updating the DevShell func in ./universe/dagger/engine.go and re-running the above. Pretty fun to play with already, even in its extremely unpolished form.

Screenshot_2023-08-02_at_4.27.05_PM.png
rocky harbor
kind carbon
#

Ah, I have the thing again: : "Can't add archive to itself" 😁

#

It's because the tar flags are different in mac (go generate)

#

I have to put --exclude universe.tar because -C works differently in linux vs darwin. On does it in the beginning the other in the end, or something like that.

#

Just finished building 🥁

rocky harbor
kind carbon
#

Err..

❯ dagger shell --env ./universe/dagger dev-shell
Error: unknown command "shell" for "dagger"
Run 'dagger --help' for usage.`
rocky harbor
#

with the latest commit pulled?

kind carbon
#

Oh, I didn't notice I didn't reset 😁 Let me try again 🙂

Screenshot_2023-08-03_at_01.43.33.png
rocky harbor
kind carbon
#

Still waiting here

Screenshot_2023-08-03_at_01.53.42.png
rocky harbor
# kind carbon Still waiting here

Ah okay... that's what I got too before I rebooted... I will just need to do some investigation, thank you for trying though! At least Linux has worked 100% of the time so far for me.

The code implementing this is a beautiful frankenstein of the new architecture's support for opening net.Conns between client<->server, @chrome pilot's tty-over-websocket code and bits and pieces of @thorn moat's services v2 code, which are all good by themselves but glued together pretty haphazardly in the PR atm, so there's a good chance there's just a race condition somewhere still, which I suppose could somehow be nudged enough by a reboot to work 🤷‍♂️

#

python envs

kind carbon
#

Sweet lord, this is awesome! Many people will love this

#

Btw, I just restarted Docker desktop!

#

cat-this-file chef_kiss

rocky harbor
# kind carbon Btw, I just restarted Docker desktop!

Wow yeah, I guess that's what I inherently did when I rebooted my machine. I wonder what this could be... it's such a strange symptom to be fixed by restarting docker desktop... there's a lot of networking shenanigans going on though, so in combination with docker desktop's networking shenanigans maybe something is going wrong in such a way that restarting fixes

kind carbon
#

I also dagger-reset before the restart so I don't know which was which.

#

So we could do just this:

+@env.shell("dev-shell")
async def build(client: dagger.Client) -> dagger.Container:
    return (
        client
        .container()
        .from("alpine")
        ...
    )

To debug this container. 🎉

rocky harbor
kind carbon
#

Well.. that's it for me! Outstanding work @rocky harbor 🙂 That session PR was massive! 64+ commits! Can't believe it's merged 🙂

#

Cya tomorrow 🙂

rocky harbor
thorn moat
#

Looking forward to playing with shell when I'm back! 👀

strange arch
kind carbon
#

@rocky harbor, just tried to build dev:

universe/embed.go:9:5: embed universe.tar: file too large (2706312704 bytes > 2000000000 bytes)
rocky harbor
#

btw that whole thing is not going to be a permanent fixture, it was just a quick way to bootstrap being able to call stuff from universe without needing to rely on an already published git repo

#

so shouldn't be a permanent headache thankfully

kind carbon
rocky harbor
kind carbon
#

I manage my python venvs centrally, outside projects. Node is a pain because you need a node_modules in the file hiearchy.

azure lily
#

OK this is the fun channel 😄

#

shell is so back

kind carbon
#

@rocky harbor can you rebase? I need some changes from main.

rocky harbor
rocky harbor
azure lily
#

here's a troublemaking question, what is the npx of dagger?

sharp zealot
#

what's npx?

#

"npm but better"?

azure lily
#

npm but irresponsibly short time from command to executing anything on the npm registry 😆

sharp zealot
#

I see, so like an alternative UX, on top of the same underlying platform, but optimized for a very specific use case?

azure lily
#

Sort of like shortening:

curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION=0.6.2 sh

./bin/dagger version

(+ a handful of environment identifiers?)

to something like (Extreme version):

curl -L https://dl.dagger.io/dagger/node/run.sh | sh -- npx create-react-app
#

(not sure if that's a format haha)

#

that would be more like "curating context of packages for an environment to run in" sort of one-liner

#

but it'd be cool to have a one liner that can run anything in a known working environment with some magic caching support or somethin

rocky harbor
#

iiuc the closest for us is dagger do --env git://github.com/org/repo <some command defined in an environment in that repo>, with the biggest difference between npx being that the command executes containerized (and with dagger caching, etc.), so it's actually less irresponsible 🙂

#

Though obviously being containerized means you need to explicitly pass host resources more often, which might be slightly more tedious in exchange for a lot more security-by-default

azure lily
#

it'd also enable my use case of having one short project.sh shell script that can be a self-contained "define and run headless doom and a web server that serves a stream of that to local 8080 " sorta dealie

#

it's been a struggle to have it e.g. write out its own dockerfile or use some specific one when it could just run dagger cmd along the way for what it needs

rocky harbor
azure lily
#

finally we can have our dream of a bash heavy one file project format that's not a zip file that runs anything anywhere consistently 😛

#

ohh I see as the way to handle proxying traffic from sub-contexts to host executor?

rocky harbor
#

The only thing I'd add is that ideally you wouldn't even need bash (unless you really want to of course), just SDK code and dagger cli one-liners

azure lily
#

I'm not above shoving some message bus dealie in there to proxy any key traffic haha

#

what I like about sh file installers is just that they work in pretty much any shell context, and about dagger that it always works once you reach that point haha

azure lily
sharp zealot
rocky harbor
azure lily
#

"I trust dagger" (or some local dagger proxy, or that install file saved out) is better than "I trust npm, and whoever wrote these packages, with root access to my computer while I'm logged in lmao"

sharp zealot
azure lily
#

right, not bad! kind of deno-esque in the secure by default-iness

#

not sure how sort of downloading package of contexts remotely works, is that sort of building on a git repo based system?

sharp zealot
#

Yes, all git

#

Actually, even the git part is optional. You can run a Dagger pipeline to download any directory from anywhere; then dynamically ask Dagger to load it as an environment

azure lily
#

that is awesome

#

can it be a base64 encoded string that's part of the sh curl one liner

#

joking but only kinda

sharp zealot
#

And so on recursively; since Dagger environments can be nested. For example you could load a top-level environment, that gives you a pipeline that takes say, a git repo URL as argument; then from there it will iterate over all branches of that repo, and load a sub-environment with that branch checkout as input, and run the same test pipeline in each. For a sort of programmatic testing grid

azure lily
#

damn yes

sharp zealot
#

Another example is that we use it to test Dagger with Dagger 🙂

azure lily
#

types.d.ts but for execution environments + docs + other useful programming context (looks like there's already support for completion environment?)

#

could be cool to have like a well known environment combinations system too for given git/node/etc repos

sharp zealot
#

Yet another example is that @modern fossil embeds Dagger as an engine in his tool - but then he also wants to use Dagger to run CI for his tool. So he has a Dagger environment loading and instrumenting another Dagger environment

azure lily
#

that is sweet

sharp zealot
azure lily
#

does dagger have a mechanism for fanning out across computers / adding computers to a fleet to run stuff on?

#

(I guess gitlab-runner style for the latter part? fly-machines-like for the first part)

sharp zealot
#

We actually discussed hardcoding the types of that catalog directly in the Dagger API. So for example, if you want to run an AWS-specific pipeline, the Dagger API may include (graphql notation) query { aws(auth: ....) { s3 { bucket(id: "foo") { upload(source: <directory>) { ... } } }

agile saddle
#

Is that something that could be a dagger environment or tool?

azure lily
#

yeah I ask about compressing the shell initialization to one line because I've been experimenting with LLM code-generation and finding it's really good at outputting and iterating on entire projects one-shot—as long as you stuff in enough context about what it needs to do + allow it to define its entire own execution environment. so defining the Dockerfile and executing the dockerfile in the same project.sh file works very consistently across languages/frameworks/project types

#

and I imagine a format for defining dagger dependencies via CLI is more ergonomic / intuitive / can have instructions compressed more than writing out a Dockerfile covering all used tech and docker run executing it (the build + run phase separation is just awkward for one-shot executions)

sharp zealot
agile saddle
#

Yep

#

It would probably need a few “escape hatches” for exposing /usr/include and other system directories, but yes

sharp zealot
#

But yes, definitely a great use case IMO

azure lily
#

it is building tiiime! 😄

sharp zealot
#

Sourcegraph extension

thorn moat
#

Possibly sacrilegious suggestion to add to the check vs checks debate: what if it was dagger checks/foo instead of dagger checks foo? Kind of conveys a hierarchical query selection, and leaves the door open for passing additional commands after, like dagger checks/foo history (totally making up a command there but you get the idea). There's some precedent for path-style selectors from tools like Bazel, though this is a different form of it.

sharp zealot
rocky harbor
#

I had to run docker system prune -a on my dev vm to clear out 130GB of disk space, but I just restarted my devenv and the dockerfile build lost all its cache, and of course now I am getting 502s from github for the step that installs neovim 😩 😩 😩

Only mentioning here because I am very much looking forward to the day where I can throw that dockerfile away and just use dagger shell... Between the better caching (including cloud cache service so I can clear my local disk without losing everything) and the ability to easily use nix/wolfi/etc. from universe rather than yolo RUNs that depend on flaky internet services, I think these sorts problems will be a thing of the past zenith

thorn moat
#

@rocky harbor I have the Go universe Build function working, which took a bunch of fixes to support optional params, but I'm thinking of changing how they work. (🧵 since I'm about to paste a large code snippet.)

sharp zealot
#

Let’s start a thread of possible Zenith extensions 😁

rocky harbor
#

Fixed checks in the Zenith branch, ended up including some improvements relevant to our discussions @sharp zealot @thorn moat ( @kind carbon already knows since I was chatting with him about it).

Previously, user code implementing a check returned (string, error) (or equivalent in python), with the string being the output and the error indicating whether the check failed, which also resulted in the environment exec itself to fail.

  • This caused all sorts of problems though, i.e. calling checks from other envs was extremely convoluted and it wasn't possible for us to have the caching behavior we want in terms of being able to cache a check failure.

Now, SDKs are expected to return an EnvironmentCheckResult object instead, with fields set to indicate check pass/fail and any output. If an error is returned (or an exception thrown), that's treated like an "internal server error", would leave check pass/fail in an unknown state.

Among other things, this brings us much closer to the caching behavior we want. EnvironmentCheckResult is returned by environment code, so that means even if it's marked as failed, the execution of the environment code will still be cached until something changes that invalidates the cache. Basically, a check failure can actually be cached now, and will be uncached only once it actually makes sense to retry it.

Examples:

  1. Implementing a check directly in user code: https://github.com/sipsma/dagger/blob/232874f1ead482d4f7f64f7829346c10011a2162/universe/_demo/main.go#L43-L43
  2. Implementing a check by combining checks from other envs as subchecks: https://github.com/sipsma/dagger/blob/232874f1ead482d4f7f64f7829346c10011a2162/universe/_demo/main.go#L35-L35

As the comments on the code mention, the change here currently results in some extra boilerplate, but that's just for now as I was trying to lay out the new model. We can still support the less boilerplatey style of code by adding sugar on top that e.g. allows you to return (string, error), merge all checks from another env into yours, etc. It's just that underneath the hood all that sugar will be deconstructed into shortcuts for constructing EnvironmentCheckResult.

sharp zealot
#

Pretty sure you can control the command name manually?

rocky harbor
#

Again, could easily be missing something

#

I guess I can set an alias, but that's not really what we want

sharp zealot
#

My money is on "you're running the wrong dagger binary" 😛

rocky harbor
spiral whale
#

it's a well known developer curse, damn these code goblins. This is their job.

sharp zealot
spiral whale
#

I have issues with them as well 😄

rocky harbor
#

I firmly believe that the moment you publicly announce something isn't working as expected, the universe quantum fluctuates into a state where it's working totally fine and it's all your fault

sharp zealot
#

Several physicists spent their career on this theory, but somehow they can only replicate it when nobody is looking

sharp zealot
#

Thinking about my “simple web UI” obsession…

#

With C2H coming up thanks to @thorn moat , and introspection already possible, I guess I have everything I need to implement a toy web UI for zenith, as a zenith extension?

#

does introspection work with the Zenith API?

rocky harbor
#

You can use introspection too though

sharp zealot
#

but how would I get the ID?

rocky harbor
# sharp zealot but how would I get the ID?

Once you load the environment you can query the ID, but I guess we should also add support for something like {environments { name id } } to make that easier since you probably aren't going to be loading the environment from the gql client and instead just rely on the environments that are automatically loaded when you call dagger listen -e ...

sharp zealot
#

yeah I’d just bootstrap from the current CLI context

#

so the current environment (here ?)

#

I was hoping for something like query { here { checks { name } } }

rocky harbor
sharp zealot
#

@rocky harbor sorry I know you already wrote this, but I can't find it anymore... What README should I use as my entrypoint in your branch?

rocky harbor
sharp zealot
#

anything special I need to do to build dagger itself in that branch?

rocky harbor
sharp zealot
#

Ironically the perfect example of why standard entrypoints are useful 🙂

#

Can't wait to dagger artifacts my way out of this one

rocky harbor
sharp zealot
#

yay

#

I wonder if we'll want a "document" entrypoint, content for human consumption?

rocky harbor
rocky harbor
sharp zealot
sleek nest
#

In case anyone missed it, here is the Project Zenith update that @rocky harbor shared at yesterday's community call - https://youtu.be/-QMGe0QX6JI

YouTube
Dagger
Project Zenith Update - 8/11

Erik provides the lastest update on Project Zenith development

We will have full demo ready at the upcoming Dagger Community Call on August 24th. You can register here: https://dagger-io.zoom.us/webinar/register/7316691559833/WN_USQjVBGXT0SWhNMvqYVvCA

Join the conversation on our Discord:
https://discord.com/channels/707636530424053791/11205...

▶ Play video
thorn moat
#

Some DX feedback: I'm finding it hard to remember to re-codegen, which results in cryptic schema errors. Have we thought about making the codegen somehow happen transparently?

rocky harbor
thorn moat
thorn moat
#

Here's a checks TUI update. This is currently just using the same graph TUI as dagger run but with focus support integrated. Finally got the data flowing in all the right places after all sorts of hacking about. Next step could be to switch it to a more succinct tree UI somehow.

image.png
rocky harbor
#

Yeah in-universe is the same problem as adding a core api right now. Though I have been questioning whether we actually should put universe envs in the standard codegen. If we instead just make it easy to add a dependency on a universe env (i.e. dagger env add universe://go universe://nix ... for adding a dep on something in universe), that might simplify a lot for us and create an incentive to use environments. Still unsure, would rely on making adding deps super trivial, but that's a good thing to do anyways, so we'll see what makes sense once we get to the point of actually merging this functionality into main.

But either way, for non-universe, totally agree we should help out more here. At the very minimum we should do what our SDK codegen does and check if there's a mismatch between the current generated code in the user's source and what would be there if generated again. We could either make that a warning or an error.

#

(cross-posting from a thread) Just pushed artifacts support; dagger artifact list lists them and dagger artifact export --output /some/path <artifact name> exports it to a local path. File and dir exports behave in obvious way, container is exported as a tar.

Can add more subcommands as desired (dagger artifact publish would be good for container, for file/dir it could either publish an OCI artifact or just be an error for now, not sure yet)

#

For user code, you can return either a File, Directory or Container. I don't love the approach I went with in terms of the API for all this yet, but works atm and was interesting to try out.

Basically, user code doesn't ever have to create an artifact specifically; instead all the artifact apis that weren't already on file/directory/container have now been added to each of those objects (version/withVersion, sbom, etc.). So the EnvironmentArtifact object is actually just a wrapper around file/directory/container that has the common fields between them (and dispatches to the correct method).

I think what we really want is to update our SDKs w/ support for graphql interfaces and then make EnvironmentArtifact an interface. Obviously too much work to update every sdk w/ that right now, but will be worth consideration once we're trying to merge this for real.

sharp zealot
#

Agree + it can just be Artifact 🙂

rocky harbor
# sharp zealot Agree + it can just be `Artifact` 🙂

Oh for sure, I forget exactly why I prefixed everything with Environment* (there was some conflict or something at some point probably due to lack of namespacing previously), but I'll go back and de-prefix everything now that it's not needed

sharp zealot
#

Yeah I remember that and also don’t remember why 😁

rocky harbor
#

Just updated demo code to have the python client actually have source code and a build step, I just gotta say even in this very early state it's SO ridiculously nice to have these absurdly simple dev loops. I change some source code in one or both of the dependency envs, call dagger shell dev-shell and everything is rebuilt and I can play with it directly. Same for dagger checks. And while the code for everything is already simple, it's only gonna get simpler as universe is fleshed out more and you can do higher-and-higher level stuff out of the box

#

We're very much onto something great here daggerlife

#

Oh also, as part of adding the above I updated dagger shell so that when it's called with no args (i.e. no shell entrypoint) it will instead drop you into the container that the environment itself executes in. I'm not sure that's exactly how we'll want to expose that functionality in the end (maybe, or maybe it should be a special builtin shell name or something), but it was really nice while trying to figure out how to package stuff in python because it gives me a shell where both the base python toolchain is pre-installed and the whole environment source code is already loaded in as my workdir.

thorn moat
thorn moat
#

More checks TUI progress

thorn moat
#

I pushed some fixes to the above branch to make dagger shell work more reliably, mainly this commit: https://github.com/vito/dagger/commit/2ff893312c6e89b912326d5d0cfc0c3493621c42

Previously it would either hang (before attempted fix #1) or raise a cryptic error 90% of attempts (after attempted fix #1), after the above PR it starts reliably 👍

I'm also working on running it with the fancy TUI, which is something I've been meaning to support in Progrock in general (otherwise you can't run a Bass REPL against Dagger). It's going well, but it's forcing me to fix more things in the vterm package, so I'll see how much I can get done; we can skip if it doesn't make the cut.

Will probably just merge my branch when I'm back from 🦮🚶‍♂️since the other TUI stuff feels pretty solid now

rocky harbor
#

I pushed some fixes to the above branch

thorn moat
#

@rocky harbor pushed the above changes, I'm investigating an issue with labels affecting v0.8.2 at the moment, but let me know if there are any issues! Also check out the new TUI, there are more changes since the update above 👀 - just didn't want to spam with asciinema links 😛

#

notably the Big Ugly Bar has been replaced with a Small Pretty Status Summary

#

Along with the rest of the changes is a medium-size refactor of the CheckResults API to be ID-able, which you might want to sync up on. Was just trying to align everything on the same patterns to make it easier to pass data around without guessing structures.

#

I was also thinking we might want to change "ID-able" things that we serialize (core.Directory, core.File, now core.EnvironmentCheckResult) to write to a different file from regular output, or something like that, so we can tell the intent more clearly, but haven't explored trade-offs there

sharp zealot
#

@rocky harbor @thorn moat any chance you'd be available to show a sneak preview of the new demo to @bleak nest (ideally with @ember walrus at the same time), so we can start workshopping the docs, and overall storytelling? I'm thinking we could do in #911305510882513037 in case any Daggernaut wants to listen in 🙂

rocky harbor
#

Erik Sipsma pushed the above changes I

rocky harbor
sharp zealot
#

Yeah timezone math is why we're trying to do it earlier

#

Ready too, @bleak nest @ember walrus ?

bleak nest
#

Yup, joining in 2 min

sharp zealot
#

yeah that demo was great @rocky harbor 🙏

#

(I didn't want to consume everyone's day, but today is my "zenith day" so I'm happy to chat some more as needed, just ping me)

#

@rocky harbor I had a note about dagger checks being more exciting if there are at least 2 different kinds of checks. This addresses upfront the objection of "how is this better than just running go test natively?". But actually if we start with the combined environment, it takes care of that.

#

For follow-up discussion: how do we organize built-in extensions:

  • Maintenance burden. How many can we afford to maintain, and how do we know the number?

  • Namespacing. How do we avoid making the API a mess? Can we add intermediary fields for categorization? Like query { linux { alpine { ... } wolfi { .. } ubuntu { ... } } } ?

#

For follow-up discussion: so should dagger do be dagger cli, for consistency with other subcommands that match the entrypoint name?

bleak nest
#

dagger cli is going to create some weird sentences eg. "Use the Dagger CLI to run dagger cli"...

#

(not an objection, just a note)

#

Did we already consider dagger functions or dagger operations or similar?

sharp zealot
#

well functions is taken (there is a different entrypoint called that, for scripting and composition) And yes agree with your note

#

I mean there’s dagger run we could coopt… 🤷

#

could we somehow land on our feet without breaking backwards compat?

rocky harbor
#

For follow up discussion how do we

rocky harbor
# sharp zealot I mean there’s `dagger run` we could coopt… 🤷

I've wondered about this exact thing previously since it is kind of confusing to explain the difference between dagger run and dagger do to someone uninitiated. I think if we made it such that dagger run --env ... becomes dagger do (or, equivalently, dagger run executed in an env dir, where it defaults to dagger run --env .), maybe that could be made to work??

#

But idk, then we have a single command that ultimately has pretty divergent behavior depending on subtle things

#

Which is maybe even worse than before

thorn moat
#

Interactive dagger shell progress, still not perfect but getting there. I'm deep in the weeds of ANSI escape sequences and ancient sparsely-documented terminal behavior, since this involves implementing a terminal emulator. ablobsweats (It did before too, but people weren't running htop before, so it only had to slightly work!)
https://asciinema.org/a/PRVFm5tKJnCVKSee4zvcCPWi4

asciinema.org
untitled

Recorded by vito

sharp zealot
thorn moat
#

none that I could find, there are about 10 abandoned/half-finished ones

sharp zealot
sharp zealot
#

Trying to run the latest zenith build and not sure what to do with this line of the README:

When running any dagger commands, you will need to either prefix commands with ./hack/dev ./bin/dagger or use a direnv-like setup that accomplishes the same.

Running that exact command is insanely slow, so I'm assuming there are env vars I can set to achieve the same result. But which env vars is not spelled out

sharp zealot
#

(got it to work)

rocky harbor
rocky harbor
thorn moat
sharp zealot
#

@here anybody want to see a sneak preview of zenith?

tough sentinel
sharp zealot
#

We had a demo scheduled but had a last minute cancellation, so might as well use the slot to show it to someone, cc @rocky harbor @thorn moat @ember walrus @sleek nest 🙂

normal blaze
#

Just when I have to go out too,

sharp zealot
sharp zealot
sleek nest
normal blaze
#

ty 🙂

lapis willow
#

Hey guys, long time lurker, you've maybe seen me around (Disc name was luis guzman for a while.... lol) . Had to drop but wanted to say thanks for the adhoc demo and letting folks like me listen in. Really stoked for the zenith release!

sharp zealot
#

FYI @normal blaze Project Zenith has reached a point where it makes sense to do such a call every week. This will be separate from the bi-weekly community meetup, more lowkey and less polished. We'll send notifications here.

normal blaze
#

Awesome, I’ll look forward to catching up

sharp zealot
#

Quick demo notes @rocky harbor @thorn moat (before I forget) 🧵

stone phoenix
#

well I was going to start on a refactor today but now I think I'll wait until zenith is ready 😂

sharp zealot
stone phoenix
sharp zealot
rocky harbor
sharp zealot
rocky harbor
# sharp zealot I’m still unsure how to just get a zenith build that “just works” without ./hack...

In order to run zenith you need a CLI built off that branch and an engine running built off that branch. That's what ./hack/dev does, builds the CLI and puts it in ./bin/dagger and builds the engine and runs it in docker container dagger-engine.dev. Then in order for that dev engine to be used you need to set the correct _EXPERIMENTAL_DAGGER_RUNNER_HOST value and make sure you are using the ./bin/dagger cli (which is easiest by just updating your PATH to prefer that dagger).

That's what these lines are saying basically: https://github.com/sipsma/dagger/blob/b0d40157b455e18aaef34e2fc5a6f85960cb79df/universe/README.md?plain=1#L4-L7

#

Would it help if I update the README to be more explicit like that? I am just coming in having run that thousands of times since it's the normal development workflow (for now), so I'm probably still assuming too much pre-knowledge

#

Pushed a quick update with that: https://github.com/sipsma/dagger/blob/ca5fc66308ed51b47d30eae655879eb0c787d6dc/universe/README.md?plain=1#L2-L11

In order to run dagger with Zenith functionality, you will need to build a Dagger CLI off this branch and build a Dagger Engine off this branch.

Our existing command for doing this is `./hack/dev`. That script will:
1. Build the Dagger CLI and export it to `./bin/dagger`
1. Build the Dagger Engine and run it in your local docker installation in a container named `dagger-engine.dev`

In order to ensure you are using the dev CLI+Engine; it's suggested to set these environment variables:
* `_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine.dev`
* `$PATH=$(pwd)/bin:$PATH`
  * This assumes `$(pwd)` points to the root of the dagger repo, replace it with the actual root if that's not the case

I just removed the other stuff for now. The whole thing with wrapping commands in ./hack/dev is only useful for dev loops where you're making changes to the engine and need to rebuild it constantly. That's not the case for anyone playing with Zenith, and if you're developing on the engine you are most likely already familiar with that, so probably not needed in there.

sharp zealot
#

I just read the whole thing and still don’t get it. Is the problem basically that there’s no registry that the dev dagger CLI can pull the corresponding dev engine image from?

#

Now I understand what you meant with zenith dogfooding zenith: there could be a zenith-dev environment that runs a mock registry.dagger.io service and the CLI could download from that

#

Also maybe the dagger CLI could use a “dev mode” escape hatch to do this without a wrapper script?

rocky harbor
sharp zealot
#

possibly via a custom “engine driver”?

rocky harbor
#

Also to be clear, none of this is custom to zenith, this is just how it works today

rocky harbor
# sharp zealot Also maybe the `dagger` CLI could use a “dev mode” escape hatch to do this witho...

I don't know really how that would work; the fundamental need is for a dev cli and a dev engine to exist, both built off your local code, that's what ./hack/dev does. Builds both and runs the dev engine. If the dev CLI itself was responsible for building the dev engine you'd create a bunch of catch-22 situations and also I think end up constantly rebuilding the engine everytime you run the CLI

sharp zealot
#

I’m picturing hack/dev almost exactly, just as a hook rather than a wrapper. anyway it’s a side quest, not important

rocky harbor
#

Yeah we've discussed various improvements but decided that it was better to just wait for what is now called Zenith 🙂 Once we can bootstrap ourselves, I think the workflow will change so that all we assume is you have a previously released dagger cli installed, we'll most likely delete ./hack/dev.

Then you'd just call dagger -e universe/dagger artifact export cli --output ./bin/dagger to get the dev CLI and something like dagger -e universe/dagger engine up (or whatever we settle on for running a service). Then you just use ./bin/dagger and you will be good to go

sonic vessel
honest hearth
#

I would also be interested if you guys put together another session, it would be interesting to start thinking about integrations and the like

atomic crown
thorn moat
#

that one was a one-off since a demo got rescheduled, but we have been talking about doing a weekly office hours type thing!

spiral whale
#

➕

rocky harbor
sharp zealot
#

FYI I think it’s an anti pattern to consume checks, commands, artifacts etc with codegen, because it makes 1) dynamic entrypoints and 2) vanilla graphql clients second-class citizens

rocky harbor
#

Quick idea on slightly modified go sdk approach (🧵)

sharp zealot
#

Insight from demos so far: one of the benefits of Zenith is allowing each dev team to package their dev environment into a standardized interface that makes it easy for other devs to use it, even if they're not familiar with the tooling. So it's not just about centralized dev-to-platform integration; it's also about decentralized dev-to-dev collaboration

sharp zealot
#

FYI I'm starting this: https://github.com/dagger/dagger/issues/5655

in theory we could do all API bikeshedding in the PR directly, but in practice we're doing a lot of doing it in Discord. It seems like a fast-moving, well-curated issue would be a nice middle ground, wdyt?

GitHub
Zenith: API v1 · Issue #5655 · dagger/dagger

Overview "Project Zenith" is the codename for a future release of Dagger which adds major new functionality. Among other things, it expands the Dagger API. This issue tracks the design wo...

rocky harbor
# sharp zealot FYI I'm starting this: https://github.com/dagger/dagger/issues/5655 in theory w...

I was just finishing up some docs and was gonna message about my plan for next steps, so good timing 🙂 My thinking is that the Zenith PR was great for putting the demo together and exploration, but we're not gonna merge that. It's a big pile of prototype-quality code and too much to really review or use for design discussions.

Instead I was gonna next start working on breaking out one component from it and open a PR for that. I was actually thinking Checks first since they seem to be fairly interesting. So that PR would include the API for checks and a clean implementation (using the lessons we learned from the Zenith PR)

#

So my thinking was that we would do the api bikeshedding in that PR, and just for checks to start.

#

Obviously we lose a little bit by not bikeshedding all the APIs all at once, but I also think that might get overwhelming. No strong opinion either way on that

#

If you agree with that though, we can start quickly by just sending out a PR with the API schema changes only for checks. It will take a few days to clean up all the relevant code (there's a lot, even discluding the other entrypoint types, and most of it needs to be refactored), so there will be plenty of meantime to bikeshed APIs before the implementation even catches up

#

Pushed a few updates to the Zenith branch:

  1. Improved go codegen, all environments use clients more consistently and less verbosely
  2. New dagger env extend, dagger env sync commands for adding dependencies (currently just used for codegen) and refreshing codegen from changes to dependencies, respectively
  3. dagger env init now also handles running custom codegen for Go environments (custom codegen is a TODO for Python)
  4. An updated README with instructions on creating an environment (using those commands): https://github.com/sipsma/dagger/blob/zenith/universe/README.md#creating-an-environment
#

Like I said above, I am not planning on there being a ton more updates to that branch going forward, and instead am going to start focusing on breaking the monolith PR down and getting it into a mergeable state

rocky harbor
# sharp zealot FYI I'm starting this: https://github.com/dagger/dagger/issues/5655 in theory w...

Here's that PR for just checks: https://github.com/dagger/dagger/pull/5659

Consists only of schema for Environment and Check so far. I like having a PR for the api bikeshedding just because it's easier to leave comments on particular parts of the api, but I don't mind using an issue either. If others prefer using that issue, then I'll just use the PR for implementation side of things and have it track the API discussions.

thorn moat
rocky harbor
thorn moat
#

vim also works. 😛 Dang, I should have have shown that.

rocky harbor
thorn moat
rocky harbor
thorn moat
#

too late, already worrying about it. (this might be a freebie, since I already had to implement something similar)

thorn moat
#

@rocky harbor OK if I keep pushing to sipsma/zenith for more dogfooding and experiments? I just pushed before thinking since I still had it as my tracking branch.

rocky harbor
thorn moat
#

sweet 👍

#

I think I implemented OSC52 support, but to test it I want to install Neovim via nixpkgs, which led to implementing the Nix builtin env, which led to [...] - almost done now 😛

#

It would be pretty cool if I could write that as an actual Go test, since it's so close (I even called it TestPkgs initially)

#

I had to tweak things to support returning just an error from a check, since I was too lazy to construct a result 😅

rocky harbor
thorn moat
#

NB: not sure if this is a bug, but (*dagger.Client).Pipeline returns a *dagger.DAG instead of a *dagger.Client now, which is a breaking change

┃ pkg/runtimes/dagger.go:89:37: cannot use client (variable of type *dagger.DAG) as *dagger.Client value in argument to runtime.container                             
┃ pkg/runtimes/dagger.go:117:37: cannot use client (variable of type *dagger.DAG) as *dagger.Client value in argument to runtime.container                            
┃ pkg/runtimes/dagger.go:138:37: cannot use client (variable of type *dagger.DAG) as *dagger.Client value in argument to runtime.container                            
┃ pkg/runtimes/dagger.go:171:37: cannot use client (variable of type *dagger.DAG) as *dagger.Client value in argument to runtime.container                            
┃ pkg/runtimes/dagger.go:214:37: cannot use client (variable of type *dagger.DAG) as *dagger.Client value in argument to runtime.container                            
┃ pkg/runtimes/dagger.go:492:12: cannot use client.Pipeline(fmt.Sprintf("import %s [platform=%s]", name, archive.Platform)) (value of type *dagger.DAG) as *dagger.Cli
┃ t value in assignment                                                                                                                                               
┃ pkg/runtimes/dagger.go:507:12: cannot use client.Pipeline(fmt.Sprintf("docker build %s", build.Context.ToValue())) (value of type *dagger.DAG) as *dagger.Client val
rocky harbor
# thorn moat I had to tweak things to support returning just an `error` from a check, since I...

Perfect, we should always enable maximal laziness 🙂 For the separated out checks PR, I'm currently working on cleaning up all the various kludge by unifying everything around Container. So whether the check result is being returned by invoking a env resolver or it's being determined by running a container set via Check.withContainer, it's all just a container. And then the server handles the polymorphism as much as possible.

The connection point being that I think we'll be able to handle that case and all the variations in a minimally convoluted way now.

rocky harbor
#

Let me know if it's causing hardship currently though, I can patch it in the zenith branch quickly

thorn moat
#

Not blocked, I just tweaked the code, just wanted to flag 👍

rocky harbor
thorn moat
#

Sure thing

sharp zealot
#

@random beacon during the demo you mentioned you were initially confused by "environment" (but then no longer confused). Another words we considered is "DAG". Would that have been less confusing?

sleek nest
#

HI all!

We are starting a weekly low-key community call cadence for anyone who is interested in Project Zenith updates.

Come join us as we show the latest demos and get in the weeds of the project to discuss design, features, feedback etc.

**When: Every Friday at 9 am PDT

Where: Dev-audio on Dagger Discord server.** If you'd like an event reminder, I suggest going to the Discord events tab at the top of the server and clicking "interested" for each of the events. You can find the event invite for the next one on August 25th here:

https://discord.gg/GhAmHCJc2G?event=1141889494476406885

sleek nest
sharp zealot
#

The main difference from the regular community meetup, is that 1) obviously it's all about Zenith, and 2) it will be a mix of demos and work sessions. The call will be used to give updates and get work done in the open

thorn moat
rocky harbor
thorn moat
rocky harbor
rocky harbor
thorn moat
#

lol, yep - e.g. printf "\e]52;c;%s;\x07" $(date | base64)

rocky harbor
thorn moat
thorn moat
honest hearth
#

HI all

random beacon
sharp zealot
sharp zealot
random beacon
#

Ohh, I think I see what you're getting at. You're saying that an instance of that reusable functionality is a dag.

sharp zealot
random beacon
#

Because they are different things, right? In oop terms the first one is the class and the second one is the object

#

To call the first one a DAG seems wrong to me. The second, possibly. Am I thinking about this the right way?

sharp zealot
#

I’m wondering about what to call the former at the moment. Currently that’s “environment” but at some point we considered “DAG”. The former we can discuss too 🙂 Currently I call that a pipeline, made by chaining functions.

Actually I’m not sure. I think possibly both need naming.

#

you’re talking about “the code” vs “a running instance of the code” right?

rocky harbor
#

My mental model pre-Zenith has been that there's this infinitely sized DAG of every possible Container, Directory, File, etc. that could possibly exist. The types (or classes, as mentioned above) are types of vertexes in that DAG. And our Core API lets you query that DAG, resolving each vertex type to particular vertex instances in the infinite DAG.

In that frame, the new types we're adding in Zenith like Checks, Artifacts, etc. are just new types sibling to existing core types like Container, Directory, etc. The true fundamental heart of Zenith is really just a way of allowing users to define ways of resolving vertex types to vertex instances. The new vertex types are just coming along for that ride because they are useful for our overall goal of improving usability.

So, that almost ends up feeling like an argument for dropping "Environment" and just calling the new feature "Extensions" (or equivalent term). You are extending the Core API with new ways of resolving vertex types to vertex instances.

That personally makes sense to me, but that's because I have my head buried in this 24/7. Fairly sure my eyes would start to glaze over if I heard the words "inifinite DAG" if I was coming from a totally different context 🙂 That's why I also personally don't mind the term Environment necessarily; it feels like a nice relatively-concrete idea to grasp onto without devolving into abstract stuff. But I nonetheless also get the confusion due to pre-existing usages of the term "environment" in other contexts.

sharp zealot
#

Terminology 🚲🏠

thorn moat
rocky harbor
#

@sharp zealot @thorn moat The checks PR is waiting and ready for api bikeshedding 🙂 https://github.com/dagger/dagger/pull/5659 I slightly modified it last Friday, but I don't currently foresee any other changes to the API. Been squashing more corner cases and fixing the progress stuff today, but haven't had to change the API anymore 🤞

random beacon
sharp zealot
#

@rocky harbor I have time blocked tomorrow to work on the Zenith demo FYI. I took notes after the last batch, but didn't have time to sit down and act on it.

rocky harbor
wintry prism
#

@warped canyon 👆 FYI in case you could get involved and get all of your extensions questions answered for the authoring guide 😉

sharp zealot
#

OK I have 30mn now, and more free time after lunch (Pacific time) if we need it

#

Joining dev-audio in 2mn

#

Quick notes before going in there:

  • Thinking about terminology and the discussion here #1142151291108335698 . Wondering if we should tweak the demo flow to try any of the ideas in there

  • OK one specific terminology topic so I don't forget. "micro-pipelines" vs. "DAG". We talk about pipelines (and sometimes "micro-pipelines" to differentiate with the big, static yaml-defined pipelines). and we talk about DAG. But we never clearly explain the relationship between the two.

  • We throw lots of concepts all at once at the audience. Maybe we could start from the bottom up, with only functions. How you can compose pipelines from them. Then, after a digestion phase, add the next layer: convenience to expose your pipelines as checks, artifacts, CLIs, shells etc

  • We need a "DAG thinking 101". Not sure how or when, but we can use the demo to experiment. Clearly, to "get Dagger" you need to shift how you look at the problem - you need to adopt "DAG thinking". Right now we don't explain that at all - you have to survive long enough to "get it" on your own.

  • More cosmetic things. top hat vs. lower hat. Which sequence do we like. And can we switch to 4-hands: one person delivers each. Seems silly but can make a huge difference in clarity.

rocky harbor
sleek nest
#

For anyone who is interested, we are doing a low-key project zenith community call in dev-audio right now. Feel free to join if you want to see the latest or to ask questions

sharp zealot
#

Quick list of interesting "side quests" we've discussed in the last few days:

  • A simple web UI to show your artifacts, checks etc
  • A simple CI tool that "replaces CircleCI in 100 lines of code" on top of Zenith (by plugging github events into dagger checks)
  • Bringing back the VS code extension (see: web ui above)
  • Writing an extension, duh 🙂
  • A simple local registry backed by your dagger artifacts (so you can docker pull straight from your DAG)
#

Separately: starting a thread about the idea to use the word "configuration" in the UI, to see if it helps with the confusion caused by "environment"

sharp zealot
#

Now that there are several PRs, what branch should I build from?

rocky harbor
# sharp zealot Now that there are several PRs, what branch should I build from?

If you are iterating on the demo, just continue using the original Zenith branch: https://github.com/dagger/dagger/pull/5443

To bump the previous messages about this that got buried, the idea is that Zenith branch will still be what we use for demos until everything is merged and also the perfect place for experiments and stuff; we'll pull stuff from it into separate PRs until there's nothing left to pull.

The Zenith Checks PR is just the first thing we are extracting out from there, it has a cleaner implementation but only has Checks, so not good for demo iteration. Very good for API bikeshedding comments though! 🙂 https://github.com/dagger/dagger/pull/5659

sharp zealot
#

I see so the Checks PR is downstream of the WIP branch?

#

Checks PR -> next candidate for an experimental feature merge?

rocky harbor
sharp zealot
#

It definitely did 😛 Thanks for reposting

#

One thing we did, that might help. At some point we merged into main a subdirectory just called "cloak" (and I think we did something for Europa). Initially it didn't have any code. Just a source of truth for "what is this project and where do I go from here"

#

Might be time to merge a zenith/ directory in the main repo. With a super basic README that is a source of truth for this kind of information

rocky harbor
sharp zealot
#

I'm happy to get it started, you have more pressing things to do

#

It will be my excuse to get a basic README going, too

rocky harbor
sharp zealot
#

on it

sharp zealot
sharp zealot
#

Question @thorn moat . We have this capability of environments being anonymous objects in the API, which you can instantiate and reference at will. We imagined some cool use cases for this, like eg. you're implementing a test matrix which loads the same environment from N commits etc.

--> How much do we actually depend on that capability today? Ie. is it worth me exploring versions where the "environment" is simply the current session, and if you want a new environment, you just open a new session? It would make things simpler; but would also maybe make some of those use cases impossible (how do you reference things across sessions etc)

(moved this out of the thread because of topic creep)

#

Keeping in mind that those environments aren't perfectly encapsulated today... ie. schema stitching happens in the global namespace

thorn moat
#

I'd say it's worth exploring alternatives either way

#

There might be other ways to do that, and they might be even cleaner. iirc those APIs were very type-unsafe and didn't feel great to use; there's a chance that a more constrained API ends up with a different but more ergonomic pattern. (I'm being extremely vague here, sorry 😅)

sharp zealot
#

Yeah, maybe dagger-in-dagger gets us further than we think in those scenarios

#

(it's the "option-2.graphqls")

thorn moat
#

you had me at "modules"

sharp zealot
#

ha ha 🙂

#

such a rubiks cube

thorn moat
#

I've been wondering that lately, we're so close already with "functions" which was like the first name we unanimously agreed on

sharp zealot
#

you make a bunch of tradeoffs, later the preconditions for your tradeoff change completely, but you never go back to revisit the tradeoff, rinse and repeat 10,000x

thorn moat
#

lol, yep

sharp zealot
#

oh right, “module” and “package” I remember objecting to because it sounded like a regular code import, instead of an API extension

#

worry being that developers expect everything to work exactly like a regular Go or NPM package , when in fact it’s subtly different (codegen etc)

#

but maybe that worry was unfounded?

thorn moat
#

Yeah, that's true, you'd have to be clear whether you mean Dagger module or Go module. (for example)

sharp zealot
#

It meets the bar for exploring further though

rocky harbor
rocky harbor
# sharp zealot Question <@108011715077091328> . We have this capability of environments being a...

--> How much do we actually depend on that capability today? Ie. is it worth me exploring versions where the "environment" is simply the current session, and if you want a new environment, you just open a new session? It would make things simpler; but would also maybe make some of those use cases impossible (how do you reference things across sessions etc)
We rely on it currently to implement the CLI. E.g. dagger checks needs to handle arbitrary envs, so it just lists the checks and calls them that way.

In general, while that sort of "dynamic" api is definitely only for power-users, it's pretty invaluable there. It's also not a burden to support in terms of the implementation at all, it kind of just falls in place as part of having all the underlying plumbing to support envs/entrypoints anyways; not work on top of that.

Keeping in mind that those environments aren't perfectly encapsulated today... ie. schema stitching happens in the global namespace
One thing worth noting is that right now you can't use an environment without loading it and stitching its API in; but it would be extremely trivial to modify it such that the schema stitching is optional (just a matter of a single optional parameter to environment { load }). So then if you want to dynamically load a ton of envs and call them dynamically, you wouldn't need to worry about conflicts.

In general the schema stitching is useful for:

  1. Raw graphql clients, since they can make high level queries like { myCoolEnv { myCoolCheck { result { success } } } }
  2. Codegen, since our existing codegen relies on the actual schema existing.
    • However, this is not a fundamental requirement at all. We could change it so that codegen clients make the "dynamic" api calls rather than relying on the schema being stitched.
    • The barrier to that is mainly just that it will take a lot of time+effort to update everything.
#

Having written that out, it does make me wonder if we should put more serious consideration into whether we actually need the schema stitching anymore.

It used to be a pretty hard requirement because it was baked in everywhere; e.g. we used to require SDKs to actually generate a graphql schema file and output that. But we've now replaced that aspect with the whole "environment builder api" (i.e. Environment().WithCheck(...).WithArtifact(...), etc.).

Getting rid of that made implementing the SDKs quite a bit easier, but I guess it also made it much easier to skip the schema stitching too as a side effect.

The remaining pain of changing our codegen to use the "dynamic" api instead of relying on the schema being pre-stitched is real, but it does feel like it would simplify quite a bit in the long term. So it could be worth just eating that annoyance now rather than deal with it perpetually...

#

Raw graphql clients would be our sacrificial lamb in this, but I wouldn't be surprised if most of the use cases for a raw graphql client end up needing the dynamic api anyways, in which case they are completely unaffected

sharp zealot
rocky harbor
# sharp zealot I confess I don’t really understand this part. What would it mean to remove sche...

  1. The only graphql API to interact with an environment would be the "dynamic" ones that rely on querying for names of entrypoints and invoking them based on those names (which are all just strings, thus "dynamic"). e.g. { env { load(...) { check(name: "myCoolCheck") { result { ... } } } }

  2. Our codegen would present the same type safe interface as today though. The inner implementation would just change so that rather than relying on the schema being stitched in an creating graphql queries based on that, it would be implemented by calling the dynamic API, e.g.

func (env *MyEnv) MyCoolCheck() *Check {
  return env.Check("myCoolCheck")
}
  1. Raw graphql clients could no longer use stitched schemas like { myEnv { myCoolCheck { result { ... } } } }. They would have to use the dynamic API. Which is a sacrifice, I'm just not sure how much of one in practice relative to the possible benefits of entirely avoiding the namespacing problem.
sharp zealot
#

oh I see! first reaction is I think the schema stitching is too valuable to give up

#

because it makes graphql the source of truth which feels right at various levels

rocky harbor
# sharp zealot because it makes graphql the source of truth which feels right at various levels

Yeah I agree, this would be a divergence from graphql. And no longer having it as a source of truth is what causes it to be painful to go and update all the SDK codegen to handle this, since it currently is all based on the graphql introspection.

But nonetheless I feel torn since removing the headache of namespacing is so tempting. It's probably worth keeping this in our back pocket if nothing else.

sharp zealot
#

Is namespacing problem that bad ?

rocky harbor
# sharp zealot Is namespacing problem that bad ?

I don't know yet because we've pretty much just ignored it so far 🙂 The main potential source of problems are environments w/ overlapping names that get loaded at the same time. And honestly, depending on how we approaching versioning, if we end up allowing two versions of the same env to be loaded at different points in the dependency DAG, that's essentially the same sort of problem.

IIRC our most recent thoughts on this were that if we don't want this to be an error cases, we'd need to stitch in objects with some sort of unique id (combination of name, version, and hash of the env definition). So we'd be stitching in like MyEnv_v_1_2_3_abcdef123 rather than MyEnv. Which should work, but obviously does make things ugly for raw graphql clients anyways. (edit: not sure if that's true since it'd just be object names, which usually aren't in queries; only field names are there)

#

So it might not be that bad if that idea works out. And it's not like we've really thought through all that in detail, I'm sure we could also come up with more ideas on how to approach it.

#

So yeah, that's a good point, it's probably not worth changing everything right now to avoid, just worth keeping in mind if namespacing ends up harder than it currently seems

thorn moat
#

Warning: this is deep rabbit hole, feel free to ignore. 😁 The sort-of-content-addressed types/functions reminds me of https://www.unison-lang.org/learn/the-big-idea/. Lots of curiosities and some parallels to what we're doing. Haven't found a reason (or time) to really dive in, so still don't know what to make of it myself.

modern fossil
#

I'm going to be trying to "sell" Dagger into a client, could I get a small update on the state of Zenith, when might we expect the ability to reuse JS pipeline from Go?

tough sentinel
wintry prism
#

Looking at the README to try the demo out and notice the same directory and environment name being used for the go and python examples if I follow from top to bottom. Guessing that we'd want two different dirs and environment names? Or could I use two dirs and init an environment with the same name in each? Or one dir and init two differently-named enviros?
https://github.com/dagger/dagger/blob/19fe2de0c22a485d9456b4ed471d97346566af3c/universe/README.md

mkdir -p ./example && cd ./example
dagger env init --name example --sdk go --root ..
...
mkdir -p ./example && cd ./example
dagger env init --name example --sdk python --root ..

Is the second example env (python) going to clobber the first one (go)?
Or should it throw an error?

sharp zealot
#

IMO the 2nd command should fail, a-la git, wdyt?

rocky harbor
rocky harbor
rocky harbor
sharp zealot
#

I have a PR I want to make against your dev branch (README stuff) @rocky harbor but not sure what's the best way. PR against your repo?

#

Maybe time to move the dev branch to dagger repo so we can push to it in parallel without the overhead of PR to a fork?

thorn moat
rocky harbor
sharp zealot
#

Demo notes...

  • I want to try some of the new terminology we talked about... Modules, configuration
  • We need diagrams
  • Need to talk about the component system first, then move up to higher-level entrypoints as we discussed
  • Still need to show the GraphQL API more obviously (simple queries and/or simple web UI)
  • We haven't resolved CLI vs command vs tools..
  • I worry that we're moving too much work to the SDKs. Maybe time to make a list of things we could move to the core (free for all SDKs)
lapis willow
#

Just finished walking through the demo, interested in starting to build some environments, but seems a little challenging outside dagger repo.

Is it supported to init a zenith dagger env with a root from git? I see some references to git support here, but getting an error when running the below.

Example:

dagger env init --name foo --sdk go --root 'https://github.com/sipsma/dagger?ref=zenith'
# codegen runs, things seem ok, I can write a pipeline in `main.go`
dagger checks
➜  ci git:(master) ✗ dagger checks
• Engine: 0a64c1c084ee (version devel ())
• Engine: 0a64c1c084ee (version devel ())
⧗ 2.83s ✔ 8 ∅ 7 ✘ 1
Error: failed to load environment: environment config path "/Users/steven.tobias/repos/misc_projects/foo/ci/dagger.json" is not under environment root "/Users/steven.tobias/repos/misc_projects/foo/ci/https:/github.com/sipsma/dagger?ref=zenith"
GitHub
dagger/cmd/dagger/env.go at zenith · sipsma/dagger

A portable devkit for CI/CD pipelines. Contribute to sipsma/dagger development by creating an account on GitHub.

rocky harbor
#

(demo notes) May want something like dagger env override similar to go mod override where if you want to dev on a dependency env you can temporarily use a local checkout of it, even if that dep is git://

sharp zealot
#

I'll join dev-audio around 11:30 pacific in 90mn if anyone is around

#

sorry I can't join before

rocky harbor
rocky harbor
# sharp zealot Demo notes... - I want to try some of the new terminology we talked about... M...

I worry that we're moving too much work to the SDKs. Maybe time to make a list of things we could move to the core (free for all SDKs)
For this specifically, that's being done in the Checks PR, maximizing how much is done by the engine and, for what's left in terms of what the engine needs from SDKs, it's encoded into a set of "internal" graphql apis. Alex asked about this too yesterday, I outlined some more details here: https://github.com/dagger/dagger/pull/5659#discussion_r1309460335

#

I m going to be trying to sell Dagger

honest hearth
#

Thoughts from our chat (thanks Dagger team for the presentation!). I wanted to spend some time anchoring Zenith in some real world use cases and unwrap this a bit more (these may not have immediate answers, they're just top of mind):

Envision a system with two different components implemented in two different langues. One is a Golang backend system, and the second is a Typescript frontend system. Let’s identify some users for our stories.

  • Platform engineer (Alice)
  • Backend engineer (Bob)
  • Frontend engineer (Charlie)

Alice wants to create a shared functions library for Bob and Charlie. An example that Alice would like to expose to everyone is "uploadS3", a function that takes a Dagger file or directory and uploads it to S3 bucket. This is a generic function that both Bob and Charlie will use in Go or Typescript since they each need to upload things to s3 as part of their pipelines.
Alice wants to create a CLI and pipelines zenith environment that exposes the following composable commands and artifacts:

  • Build (calls frontend build and backend build)
  • Test (calls frontend test and backend test)
  • Publish (Calls frontend publish and backend publish)
  • Deploy (Calls frontend deploy and backend deploy)

Alice also wants to allow Bob and charlie to be able discover each other’s pipelines from the CLI and call things from them

What does the above setup look like for end users in Zenith?
How might Universe come into play here, especially for Alice’s “uploadS3” function?
What might be a typical workflow for someone who just wants to be handed an environment and be able to build and test it with no local dependencies?
What might be a typical workflow for Bob/Charlie, who want to modify their code but not the others
What might be a typical workflow for Alice, who is responsible for exposing integrations to end users?
Who is responsible for writing checks/artifacts in this setup? Is it bob/Charlie or Alice? Or maybe it’s up to the end users?

sharp zealot
#

be right there

#

@rocky harbor ping 🙂

sharp zealot
#

Conor's test case

wintry prism
#

Running through the README of the Zenith PR #5443

I'd love feedback from

image.png
#

as it sits silently for a while. Maybe like
generating Go code...
or similar

sharp zealot
#

Finally getting around to reviewing the Zenith Checks PR, sorry that it took so long. I hadn't really realized that it was the desired place for API bikeshedding.

rocky harbor
warped canyon
#

Question that came up from chatting with @wintry prism earlier:
Now that the top level pipeline code is running in a container rather than host runtime, does that basically restrict us from using os.Getenv in pipelines, or is there some way around that?

rocky harbor
#

We've talked about formalizing a whole api for extensions to request access to host env vars (and for hosts to selectively grant access), but it doesn't exist yet and will be a medium-size project of its own most likely.

warped canyon
#

Makes sense!

wintry prism
#

The calling shell's state is a more variable beast from the filesystem state of a host too, I suppose. Either way you have to do a snapshot of that state at some point.

rocky harbor
# wintry prism Ah, so that's why with Zenith we have `dag.Host().Directory(".")` as with SDKs,...

Yep, that is a whole topic we'll need to cover in our docs (and maybe even consider renaming some APIs to improve clarity). Basically, dag.Host() always refers to the "host" of wherever it's being called; it doesn't refer to the "caller host" (as in the host of the CLI).

Which is great for consistency and makes it possible for envs to do stuff like write a local file and then load it into dagger (we already rely on that in the port of dagger's integ tests to be a Check in the Zenith Check PR).

It's also great for security-by-default and reproducibility since envs are by default only able to refer to their own resources, not just grab arbitrary stuff from the caller's host (or from other envs' hosts).

kind carbon
rocky harbor
rocky harbor
warped canyon
kind carbon
warped canyon
sharp zealot
#

IMO we need to revisit the host access API so an env can declaratively say “I need these things from the host” and the user can decide whether it trusts that env to get access or not

rocky harbor
sharp zealot
#

btw I plan on reviewing/bikeshedding checks pr today

#

sorry slow to get going

#

wondering how much bikeshedding I can fit in comments but will try

rocky harbor
# sharp zealot sorry slow to get going

No worries! And if the PR comment format becomes a limitation, no problem at all discussing elsewhere. We can just try to use the PR's schema as the "current schema intended to be merged into main" and update it as needed from discussions elsewhere

warped canyon
sharp zealot
sharp zealot
#

Kyle Zenith first pass

#

A few questions on Zenith internals...

#
  1. @rocky harbor on the checks PR, I see some types eg Check seem to be the same for consumer and provider: there is result to query the result, but also withName etc for registering the actual check function. Is that right?
sharp zealot
#

Another question about zenith codegen…

rocky harbor
sharp zealot
#

Registering functions?

rocky harbor
#

Pushed fixes for dagger env and friends not showing output (cc @wintry prism) + support for git deps to the Zenith (and Zenith Checks) branch (cc @warped canyon @lapis willow).

Updated docs with this section: https://github.com/dagger/dagger/blob/4e0ad448ad3041fb5ad18550399a07e8242d96cd/zenith/EXAMPLES.md#referencing-environments-in-git Copying the main part here so it's not buried since it's kind of cool to see:

For example, you can run this to execute the current demo:

dagger checks -e 'git://github.com/sipsma/dagger?ref=zenith&subdir=universe/_demo'

Or you can execute this to add a dependency on the demo server environment:

dagger env extend 'git://github.com/sipsma/dagger?ref=zenith&subdir=universe/_demo/server'
#

The git:// format is awful, we need to improve that, but was simplest to start with

sharp zealot
#

Request: can we retire the word “entrypoint” as a load-bearing term in the UI and API… I know it’s annoying but I think the way we use it today will end up confusing people. And with the ongoing terminology refactoring & bikeshedding, the less words to worry about, the faster we can move…

rocky harbor
#

If so, SGTM

sharp zealot
#

alternatively, instead of messing with checks pr, we try a parallel experiment, a branch that only talks about functions and not checks (yet). And that branch doesn’t have the word entrypoint. Then we compare.

sleek nest
#

Hey Daggernauts! If you want to join the open project discussion, feel free to join dev-audio right now for the Weekly Zenith Community Call. It is a low-key open forum to discuss the project and answer any community questions. Everyone is welcome!

sharp zealot
#

FYI I'm trying to clear my calendar to resolve the checks PR bikeshedding from morning call, I don't want to throw a wrench then walk away.

rocky harbor
sharp zealot
#

I'm free in another 15mn or so

sleek nest
#

@latent trellis Thanks for your time today! We're excited to finally bring you closer to the extension experience that you wanted 🙂 @warped canyon is working on a lot of the examples and documentations for it, so I mentioned that you'd be a great Daggernaut to get feedback from 😁

#

He may reach out as we get further along

sharp zealot
#

@rocky harbor maybe we should leave the Checks PR untouched, and start another "Functions" PR?

#

to preserve the history so far?

rocky harbor
sharp zealot
#

I think it might be simpler to reset the discussion yeah

rocky harbor
sharp zealot
#

on it!

#

Oh should I just kickoff the PR with that?

rocky harbor
sharp zealot
#

@rocky harbor so I rebase from main, with only a README and mock graphql schema, and you'll cherry-pick from other branches on top?

rocky harbor
#

I've been reworking the "internal SDK" api to account for generalized functions, so we'll have that to bikeshed too. I'll push it as a separate schema file though to keep things less confusing.

sharp zealot
#

you know that could be a mutation too

#

not the constructing of the "$THING" - but the registering it

rocky harbor
#

That's not finalized, but yes, I agree 🙂

sharp zealot
#

installModule is called from the "outside" and installFunction from the "inside" right?

rocky harbor
sharp zealot
#

what do you currently call the code executed by the engine, that "runs" the module, registers its functions etc?

#

The module's init function?

#

The module's entrypoint? (confusing I know 🙂

rocky harbor
sharp zealot
#

This may be a stupid idea: could there be a Module type that represents both 1) what a client gets when loading a module source directory, and 2) what a module init function gets when registering all its types and functions?

The former is the "frontend" of the module. And the latter is its "backend" if that makes sense

rocky harbor
#

It is even underneath the hood using the same plumbing we use to invoke functions, it's just a special invocation that requests the module definition as a return value

rocky harbor
sharp zealot
#

Nice, ok let me try something in that PR then 😛

#

Let's put my last 15mn of freedom to good use

sharp zealot
rocky harbor
#

I've thought of ways of loosening that restriction, but I'm not sure it's worth the complication.

sharp zealot
#

eg. query { module { fromDirectory("./ci") { withFunction(name: "hello", ...) { id } } } } --> If I install this module, will it successfully install my ci module + a monkeypatched "hello" function?

EDIT: assume I'm doing this during initialization of my own module, to account for your last message

#

this is really to test my own understanding of the plumbing

rocky harbor
# sharp zealot eg. `query { module { fromDirectory("./ci") { withFunction(name: "hello", ...) {...

EDIT: assume I'm doing this during initialization of my own module, to account for your last message
Ah I see, yes actually we literally already do the equivalent of that in the demo where the CI environment registers artifacts from other environments:

dag.CurrentEnvironment().WithArtifact(dag.DemoServer().ServerImage())

Which results in the ServerImage artifact being part of the CI environment's definition, but when it comes time to actually evaluate ServerImage, it's DemoServer that's invoked.

#

So yeah that all works

sharp zealot
#

Where's the best withFunction and type Function to plagiarize for the PR?

#

@rocky harbor I need to switch to childcare mode before the PR is fully ready... But will very gladly send it tonight after putting the kids to bed. I did want to share this snippet 🙂

# 
type Module {
    """
    Serve a module's API in the current session.
        Note: this can only be called once per session.
        In the future, it could return a stream or service to remove the side effect.
    """
    serve(workdir: Directory, environment: [String!]): Void!

    withFunction(id: FunctionID!): Module!
}

extend type Directory {
    "Load the directory as a Dagger module"
    asModule: Module!
}
#

Just trying a fun alternative to the mutation

rocky harbor
sharp zealot
#

also addresses @thorn moat 's concern about "from"

rocky harbor
sharp zealot
#

Note the serve, that's the part that would replace the install... mutation

#

the idea being that maybe down the road, it doesn't have to be a special case with side effects at all...

#

once the API has a stream multiplexing feature - which we know we'll need anyway

#
  • "please serve this module's API"
  • You got it, please head over to stream ID 'foo' for a working http session with that graphql API loaded
rocky harbor
# sharp zealot - "please serve this module's API" - You got it, please head over to stream ID '...

You got it, please head over to stream ID 'foo' for a working http session with that graphql API loaded
Interestingly enough, dagger shell in the demo works by adding a field to Container called shellEndpoint, which returns a subpath of the http server that can be connected to over websocket and will stream the shell.

We wouldn't even need streaming or websockets for this though. We already today could change serve to just return an http endpoint that will serve the schema

#

All the plumbing to support that exists across various branches, would just need a little surgery. The part that would cause some actual elbow grease to be needed is the fact that SDKs would all need major updating

#

Which inclines me to think it's not worth it at this exact time. But should leave it open as an option in the API maybe

sharp zealot
#

Yes totally this would be a future improvement

#

in the meantime module.serve would take over the current session

#

but the API style would set the stage for that future improvement

#

this really feels like all the pieces are falling together

latent trellis
#

My initial feedback based on the demo is I really like that the interface is centered around the common use cases of a build system. Dagger is no longer just a portable CI environment, but an actual build system at this point which is great. It's an easier sell to developers IMO. (Especially when you tell them they can use any language, not just Starlark :D)

I'm still not used to environment as a term. Module or package would better reflect the reusable aspect. Or you could just come up with some fantasy names, like Orbs (although I pretty much hated Orbs, so don't use that one 😄 ).

sharp zealot
sharp zealot
#

getting that PR going. sorry for delay

sharp zealot
rocky harbor
#

Another Go SDK approach

agile saddle
#

I saw the XRC2 post yesterday, and I was wondering if Zenith is far enough along to be worth trying. In addition to the source graph functions I mentioned earlier in a seperate thread, how would dagger check handle the following?

I have 5 distros that I want to validate against, each each distro runs a cmake ctest suite with something like 350 unittests. How can I help Zenith report this from the Python SDK?

#

Ideally, I would be able to see primarily that say Fedora 38 is green, but maybe centos 7 has a build failure, and centos 8 has a single test failure relating to say some function (e.g. unit test 119 failed). What would this look like with dagger check? What code do I need to write in the SDK?

latent trellis
#

Can check type actions also have artifacts as outputs? eg. code coverage reports, lint results, etc

kind carbon
latent trellis
sharp zealot
#

@kind carbon @latent trellis we got a lot of questions like this (basically “how strict is the straight jacket and how do I wiggle out of it?”), and for the next experimental release we’re going to remove the straight jacket and see how it feels. Specifically we’re shelving (for now) specialized entrypoints like artifact, check, command, shell - in favor of general-purpose functions that map to graphql resolver and therefore are not limited at all.

Flipside is that there will be less convenience and “magic” for users, and each project will still need to reinvent some conventions on how to name their functions, what arguments etc.

The idea is that once we get more usage with functions, patterns will emerge and we can incorporate them gradually into the core.

kind carbon
#

Have we decided on how to invoke the general-purpose functions from the CLI?

rocky harbor
#

Have we decided on how to invoke the

latent trellis
thorn moat
#

nesting + Unix sockets/C2H networking

sharp zealot
rocky harbor
#

For those who couldn't attend the Zenith Call today, we talked about the new Modules+Functions approach today (see above linked PR), some of the new possibilities it unlocks.

Biggest takeaway for me were the big +1s from @spiral whale and @warped canyon to support for extending existing types like Container from Modules, which I was a bit surprised by since I thought it would be a niche feature at most, but bumped much higher in priority in my mind now 🙂 Sounds like it will make a lot of code cleaner by being a more advanced version of .With essentially. Fortunately, the plumbing so far is mostly setup to handle that, so won't be a huge lift

sharp zealot
sharp zealot
sharp zealot
sharp zealot
#

FYI, I'm writing an issue on dagger script to get us started, since I think it's a linchpin of the xrc2 design. And one of the medium-to-large pieces we'll need to build on top of @rocky harbor 's branch while he's away

sharp zealot
#

(added rough architecture diagram)

wintry prism
#

@thorn moat with the latest changes to #5757, when I run
./hack make engine:build I get a dagger binary in the bin dir that is killed immediately after I invoke it:

dagger-fork ➤ which dagger                                           git:zenith-functions
/Users/jeremyadams/src/dagger-fork/bin/dagger
dagger-fork ➤ ls -al `which dagger`                                  git:zenith-functions
-rwxr-xr-x  1 jeremyadams  staff  32418274 Sep 12 07:31 /Users/jeremyadams/src/dagger-fork/bin/dagger
dagger-fork ➤ dagger                                                 git:zenith-functions
[1]    50441 killed     dagger

I also see a strange env var that is set

_=/usr/bin/env
thorn moat