#Zenith GraphQL schema, draft 2

1 messages · Page 1 of 1 (latest)

stable ocean
#

A few updates:

  • Tests become Checks + API improvements
  • Fleshed out Artifact API
stable ocean
#

FYI @oak matrix @misty sapphire @keen glen @rigid mango

keen glen
rigid mango
#

What's the intention of the source/workdir split in Environment? Is the idea that the source is the "input" and the workdir is updated in newly-generated versions of the Environment?

#

nvm saw @stable ocean's other message about this

stable ocean
radiant jasper
#

What are checks, artifacts, tools, tool commands, and shells and how do these actually differ?

stable ocean
#
  • A check is something that can fail or succeed. For example you could render an environment's checks as a test matrix with green/red cells. Or you could export it as a spreadsheet, etc
radiant jasper
#

Another random thought here is could this be used to export one or more thinks like a devcontainers?

radiant jasper
stable ocean
#
  • An artifact is a piece of data which can be downloaded, archived, signed, uploaded to a OCI registry of some kind
#
  • Tools are command-line tools. Something that you can interact with from the terminal. Takes arguments and flags, has output and input streams, etc. Maps roughly to a shell script, npm script etc
radiant jasper
stable ocean
#
  • shells are programs that a user can interact with on a terminal. Could be bash, or nix-shell or a sql repl
stable ocean
#

let me cross out the "OCI" part

radiant jasper
stable ocean
#

Bottom line: those entrypoints to an environment are meant to be familar, and map to well-understood concepts. We are looking to standardize interactions with an environment, with just the right amount of abstraction

radiant jasper
stable ocean
stable ocean
radiant jasper
#

Also where would a “notification”/“timer” fit in?

radiant jasper
stable ocean
#

We have been discussing a future "event" or "trigger" entrypoint 🙂

stable ocean
#

So in the context of robin hood hat: if you're using an environment to develop your app; and your app repo contains a script; then that script is just a file in the environment's working directory. What you can do with that script depends on what capabilities your environment has

#

Can I ask what that script is for? It would help me make a more realistic answer

#

Most environments are likely to have at least one shell available. We could make it a convention that a shell should have the environment's working directory mounted in. So with dagger shell you would have an interactive shell, running in a container, with your script available on the filesystem. What you do next depends on the project

radiant jasper
#

Suppose it is some experiment that a scientist wrote that will collect some benchmark results, and then produce a plot. However the program doesn’t quite crash, but the output is otherwise nonsensical (eg the code with 5 iterations of a task took more time with the one with 10 iterations of the same exact task) - what do I do to “get under the covers” with a debugger or some other interactive inspection shell on this task? Or attach a profiler or a bpf program to something that would otherwise be a “tool” or an “artifact” in this parlance.

#

The other part that I’m not sure of is how does this schéma handle failures, does this provide enough to extract the container stage when the dag “failed” to allow extracting and debugging it? I don’t speak enough graphql (yet) to know 🤣 :

stable ocean
#

Ideally you're using a "python-science" environment that gives you all those capabilities via custom tools, shells etc. For example a specific debugger + specific configuration could be exposed as a custom shell. dagger shell -s debug or something like that.

#

If the environment falls short, then you can extend it yourself by adding your own tools, shells, checks etc. Which are really just code, using a Dagger SDK

radiant jasper
#

As long as that extension story is seem less, I think a lot of my concerns are assuaged. I could see it getting complicated if the scientist wrote their code in go, and I know say rust or python or visa versa.

stable ocean
#

To be honest, we haven't completely figured out the UX for that

radiant jasper
#

My initial intuition is/was (I’m not sure now) that tools, artifacts, and shells is an artificial distinction. Under the hood everything is a container layer. It produces a shell, a command, and one or more artifacts. However this discussion has convinced me that you can provide a better UX by labeling things.

stable ocean
#

Yes exactly, it's all the same thing in the backend. But today Dagger lacks a good frontend, to be directly usable by developers. These are all frontend/UX concepts.

radiant jasper
#

In this simpler model, it’s just a matter of committing layers at appropriate times and exposing them or portions of them at appropriate times

#

In this case you just provide a mechanism to spawn a shell at a particular point in the dag perhaps stealing some intuition from content adressable systems

#

Things like execution groups added somewhat recently would also give a handle to further slice and dice the dag.

stable ocean
#

Let me paste a snippet of a possible implementation of an env

radiant jasper
#

An approach is to expose a way to ask what the “layer” is for a particular “result” (artifact, tool, shell) has, and then expose this for debugging

stable ocean
#

Ok here comes a snippet @radiant jasper

# 
package main

import (
    "dagger.io/dagger"
)

func main() {
    env := dagger.NewEnvironment().
        WithWarmup(
            func(ctx dagger.Context, area string) error {
                Biome.Sync(ctx)
            },
        ).
        WithTool(
            "progrocker",
            "Dead simple CLI tool to develop progrock",
            Generate, Test,
        ).
        WithTool(
            "biome",
            "Utility for managing Progrock's Nix development container (for advanced users only)",
            Biome, NixBase, Nixpkgs,
        ).
        WithShell("biome", Biome).
        // Add env-specific data to the graph,
        // for scripting, inspection and composition
        WithGraph(
            Biome,
            Generate,
            Nixpkgs,
            NixBase,
            NixDerivation,
        }.
        WithCheck("go", "Go test suite",
            func(ctx dagger.Context) (dagger.Check, error) {
                return new(goTestSuite), nil
            },
        ).


type goTestSuite struct {
    exitCode int
    stdout string
}

// Implements dagger.Check interface
func (t *goTestSuite) run(ctx dagger.Context) error {
    ctr, err := Biome(ctx).
    Focus().
    WithExec([]string{
        "gotestsum",
        "--format=testname",
        "--no-color=false",
        "--json-file", "result.json"
        "./...",
    })
    stdout, err := ctr.Stdout(ctx)
    if err != nil {
        return nil, err
    }
    exitCode, err := ctr.ExitCode(ctx)
    if err != nil {
        return nil, err
    }
    t.stdout = stdout
    t.exitCode = exitCode
    return nil
}

func (t *GoTests) Result(ctx dagger.Context) (*dagger.TestResult, error) {
    if err := t.run(ctx); err != nil {
        return nil, err
    }
    return (t.exitCode == 0), nil
}

func (t *GoTest) SubChecks() (dagger.Check, error) {
    // FIXME: parse report file to break out each go test as a subcheck
    return nil, nil
}

func Generate(ctx dagger.Context) (*dagger.Directory, error) {
    return Biome(ctx).
        Focus().
        WithExec([]string{"go", "generate", "./..."}).
        Directory("/src"), nil
}

func Biome(ctx dagger.Context) *dagger.Container {
    return Nixpkgs(ctx, Flake(ctx),
        "bashInteractive",
        "go_1_20",
        "protobuf",
        "protoc-gen-go",
        "protoc-gen-go-grpc",
        "gotestsum",
    ).
        WithEnvVariable("GOCACHE", "/go/build-cache").
        WithMountedCache("/go/pkg/mod", ctx.Client().CacheVolume("go-mod")).
        WithMountedCache("/go/build-cache", ctx.Client().CacheVolume("go-build")).
        WithMountedDirectory("/src", Code(ctx)).
        WithWorkdir("/src")
}

func Code(ctx dagger.Context) *dagger.Directory {
    return ctx.Client().Host().Directory(".", dagger.HostDirectoryOpts{
        Include: []string{
            "**/*.go",
            "**/go.mod",
            "**/go.sum",
            "**/testdata/**/*",
            "**/*.proto",
            "**/*.tmpl",
        },
    })
}

func Flake(ctx dagger.Context) *dagger.Directory {
    return ctx.Client().Host().Directory(".", dagger.HostDirectoryOpts{
        // NB: maintain this as-needed, in case the Nix code sprawls
        Include: []string{"flake.nix", "flake.lock"},
    })
}
#

This is an extrapolation from @flat seal 's actual Daggerized dev environment for his repo progrock. Basically a go dev env based on Nix.

The bottom half is real: Generate, Biome, Code, Flake, all this works today on Dagger 0.6.

The top half (main function + goTestSuite type) is made up: an imaginary backend implementation of the graphql schema above

#

But it should give a general idea