Zenith GraphQL schema, draft 2 | Dagger | Page 1

stable ocean Jun 29, 2023, 12:46 AM

#

A few updates:

Tests become Checks + API improvements
Fleshed out Artifact API

stable ocean Jun 29, 2023, 6:29 PM

#

FYI @oak matrix @misty sapphire @keen glen @rigid mango

keen glen Jun 29, 2023, 7:01 PM

#

Left some feedback over here this morning: #1120536656940892190 message

rigid mango Jun 29, 2023, 8:54 PM

#

What's the intention of the source/workdir split in Environment? Is the idea that the source is the "input" and the workdir is updated in newly-generated versions of the Environment?

#

nvm saw @stable ocean's other message about this

stable ocean Jun 29, 2023, 9:18 PM

#

for reference 🙂 #1124057482877218905

radiant jasper Jun 30, 2023, 2:07 AM

#

What are checks, artifacts, tools, tool commands, and shells and how do these actually differ?

stable ocean Jun 30, 2023, 2:09 AM

#

A check is something that can fail or succeed. For example you could render an environment's checks as a test matrix with green/red cells. Or you could export it as a spreadsheet, etc

radiant jasper Jun 30, 2023, 2:09 AM

#

Another random thought here is could this be used to export one or more thinks like a devcontainers?

radiant jasper Jun 30, 2023, 2:10 AM

#

stable ocean - A check is something that can fail or succeed. For example you could render an...

What about benchmarks (for regressions) or coverage changes? Are those checks?

stable ocean Jun 30, 2023, 2:10 AM

#

An artifact is a piece of data which can be downloaded, archived, signed, uploaded to a ~~OCI~~ registry of some kind

#

Tools are command-line tools. Something that you can interact with from the terminal. Takes arguments and flags, has output and input streams, etc. Maps roughly to a shell script, npm script etc

radiant jasper Jun 30, 2023, 2:13 AM

#

stable ocean - An artifact is a piece of data which can be downloaded, archived, signed, uplo...

What about docs or RPM/DEB, or a Maven JAR? You don’t upload them to OCI repository, but are common “artifacts” in many CI systems.

stable ocean Jun 30, 2023, 2:13 AM

#

shells are programs that a user can interact with on a terminal. Could be bash, or nix-shell or a sql repl

stable ocean Jun 30, 2023, 2:13 AM

#

radiant jasper What about docs or RPM/DEB, or a Maven JAR? You don’t upload them to OCI reposit...

Yes those would be considered artifacts. And you definitely can upload them to an OCI repository 🙂 (but you don't have to of course)

#

let me cross out the "OCI" part

radiant jasper Jun 30, 2023, 2:13 AM

#

stable ocean Yes those would be considered artifacts. And you definitely can upload them to a...

I thought those were just for containers 🙂

stable ocean Jun 30, 2023, 2:15 AM

#

Bottom line: those entrypoints to an environment are meant to be familar, and map to well-understood concepts. We are looking to standardize interactions with an environment, with just the right amount of abstraction

radiant jasper Jun 30, 2023, 2:15 AM

#

stable ocean - shells are programs that a user can interact with on a terminal. Could be bash...

Let’s say a user has a script - as I understand it would be a tool. Now suppose I want to debug it under GDB/PDB so it be comes interactive? What do I do now?

stable ocean Jun 30, 2023, 2:15 AM

#

radiant jasper I thought those were just for containers 🙂

It's increasingly popular to use OCI as a generic artifact storage and distribution layer. That's what we'll do under the hood for Dagger artifacts, but it will be transparent to the end user

stable ocean Jun 30, 2023, 2:18 AM

#

radiant jasper Let’s say a user has a script - as I understand it would be a tool. Now suppose...

Clarifying question: is this user A) creating an environment for others to develop in , or B) developing inside an environment created by someone else?

radiant jasper Jun 30, 2023, 2:18 AM

#

Also where would a “notification”/“timer” fit in?

radiant jasper Jun 30, 2023, 2:19 AM

#

stable ocean Clarifying question: is this user A) creating an environment for others to devel...

I’m interested in both 🙂

But my brain went to the latter first.

stable ocean Jun 30, 2023, 2:19 AM

#

We have been discussing a future "event" or "trigger" entrypoint 🙂

stable ocean Jun 30, 2023, 2:20 AM

#

radiant jasper I’m interested in both 🙂 But my brain went to the latter first.

Ok, for further reference we have a nomenclature for those two "hats" that a Dagger user could wear:

Creating the environment: top hat
Using the environment: robin hood hat

#

So in the context of robin hood hat: if you're using an environment to develop your app; and your app repo contains a script; then that script is just a file in the environment's working directory. What you can do with that script depends on what capabilities your environment has

#

Can I ask what that script is for? It would help me make a more realistic answer

#

Most environments are likely to have at least one shell available. We could make it a convention that a shell should have the environment's working directory mounted in. So with dagger shell you would have an interactive shell, running in a container, with your script available on the filesystem. What you do next depends on the project

radiant jasper Jun 30, 2023, 2:33 AM

#

Suppose it is some experiment that a scientist wrote that will collect some benchmark results, and then produce a plot. However the program doesn’t quite crash, but the output is otherwise nonsensical (eg the code with 5 iterations of a task took more time with the one with 10 iterations of the same exact task) - what do I do to “get under the covers” with a debugger or some other interactive inspection shell on this task? Or attach a profiler or a bpf program to something that would otherwise be a “tool” or an “artifact” in this parlance.

#

The other part that I’m not sure of is how does this schéma handle failures, does this provide enough to extract the container stage when the dag “failed” to allow extracting and debugging it? I don’t speak enough graphql (yet) to know 🤣 :

stable ocean Jun 30, 2023, 2:39 AM

#

Ideally you're using a "python-science" environment that gives you all those capabilities via custom tools, shells etc. For example a specific debugger + specific configuration could be exposed as a custom shell. dagger shell -s debug or something like that.

#

If the environment falls short, then you can extend it yourself by adding your own tools, shells, checks etc. Which are really just code, using a Dagger SDK

radiant jasper Jun 30, 2023, 2:40 AM

#

As long as that extension story is seem less, I think a lot of my concerns are assuaged. I could see it getting complicated if the scientist wrote their code in go, and I know say rust or python or visa versa.

stable ocean Jun 30, 2023, 2:41 AM

#

radiant jasper As long as that extension story is seem less, I think a lot of my concerns are a...

Yes I agree, we'll have the sweat the details on that part

#

See #1124057482877218905 for example

#

To be honest, we haven't completely figured out the UX for that

radiant jasper Jun 30, 2023, 2:45 AM

#

My initial intuition is/was (I’m not sure now) that tools, artifacts, and shells is an artificial distinction. Under the hood everything is a container layer. It produces a shell, a command, and one or more artifacts. However this discussion has convinced me that you can provide a better UX by labeling things.

stable ocean Jun 30, 2023, 2:46 AM

#

Yes exactly, it's all the same thing in the backend. But today Dagger lacks a good frontend, to be directly usable by developers. These are all frontend/UX concepts.

radiant jasper Jun 30, 2023, 2:46 AM

#

In this simpler model, it’s just a matter of committing layers at appropriate times and exposing them or portions of them at appropriate times

#

In this case you just provide a mechanism to spawn a shell at a particular point in the dag perhaps stealing some intuition from content adressable systems

#

Things like execution groups added somewhat recently would also give a handle to further slice and dice the dag.

stable ocean Jun 30, 2023, 2:51 AM

#

Let me paste a snippet of a possible implementation of an env

radiant jasper Jun 30, 2023, 2:51 AM

#

An approach is to expose a way to ask what the “layer” is for a particular “result” (artifact, tool, shell) has, and then expose this for debugging

stable ocean Jun 30, 2023, 3:23 AM

#

Ok here comes a snippet @radiant jasper

#

package main

import (
    "dagger.io/dagger"
)

func main() {
    env := dagger.NewEnvironment().
        WithWarmup(
            func(ctx dagger.Context, area string) error {
                Biome.Sync(ctx)
            },
        ).
        WithTool(
            "progrocker",
            "Dead simple CLI tool to develop progrock",
            Generate, Test,
        ).
        WithTool(
            "biome",
            "Utility for managing Progrock's Nix development container (for advanced users only)",
            Biome, NixBase, Nixpkgs,
        ).
        WithShell("biome", Biome).
        // Add env-specific data to the graph,
        // for scripting, inspection and composition
        WithGraph(
            Biome,
            Generate,
            Nixpkgs,
            NixBase,
            NixDerivation,
        }.
        WithCheck("go", "Go test suite",
            func(ctx dagger.Context) (dagger.Check, error) {
                return new(goTestSuite), nil
            },
        ).


type goTestSuite struct {
    exitCode int
    stdout string
}

// Implements dagger.Check interface
func (t *goTestSuite) run(ctx dagger.Context) error {
    ctr, err := Biome(ctx).
    Focus().
    WithExec([]string{
        "gotestsum",
        "--format=testname",
        "--no-color=false",
        "--json-file", "result.json"
        "./...",
    })
    stdout, err := ctr.Stdout(ctx)
    if err != nil {
        return nil, err
    }
    exitCode, err := ctr.ExitCode(ctx)
    if err != nil {
        return nil, err
    }
    t.stdout = stdout
    t.exitCode = exitCode
    return nil
}

func (t *GoTests) Result(ctx dagger.Context) (*dagger.TestResult, error) {
    if err := t.run(ctx); err != nil {
        return nil, err
    }
    return (t.exitCode == 0), nil
}

func (t *GoTest) SubChecks() (dagger.Check, error) {
    // FIXME: parse report file to break out each go test as a subcheck
    return nil, nil
}

func Generate(ctx dagger.Context) (*dagger.Directory, error) {
    return Biome(ctx).
        Focus().
        WithExec([]string{"go", "generate", "./..."}).
        Directory("/src"), nil
}

func Biome(ctx dagger.Context) *dagger.Container {
    return Nixpkgs(ctx, Flake(ctx),
        "bashInteractive",
        "go_1_20",
        "protobuf",
        "protoc-gen-go",
        "protoc-gen-go-grpc",
        "gotestsum",
    ).
        WithEnvVariable("GOCACHE", "/go/build-cache").
        WithMountedCache("/go/pkg/mod", ctx.Client().CacheVolume("go-mod")).
        WithMountedCache("/go/build-cache", ctx.Client().CacheVolume("go-build")).
        WithMountedDirectory("/src", Code(ctx)).
        WithWorkdir("/src")
}

func Code(ctx dagger.Context) *dagger.Directory {
    return ctx.Client().Host().Directory(".", dagger.HostDirectoryOpts{
        Include: []string{
            "**/*.go",
            "**/go.mod",
            "**/go.sum",
            "**/testdata/**/*",
            "**/*.proto",
            "**/*.tmpl",
        },
    })
}

func Flake(ctx dagger.Context) *dagger.Directory {
    return ctx.Client().Host().Directory(".", dagger.HostDirectoryOpts{
        // NB: maintain this as-needed, in case the Nix code sprawls
        Include: []string{"flake.nix", "flake.lock"},
    })
}

#

This is an extrapolation from @flat seal 's actual Daggerized dev environment for his repo progrock. Basically a go dev env based on Nix.

The bottom half is real: Generate, Biome, Code, Flake, all this works today on Dagger 0.6.

The top half (main function + goTestSuite type) is made up: an imaginary backend implementation of the graphql schema above

#

But it should give a general idea

#Zenith GraphQL schema, draft 2