#philosophical question that s on my mind

1 messages · Page 1 of 1 (latest)

potent hemlock
#

I initially assumed the ideal state for Dagger was for everybody on the team to use it, directly: everyone learns the SDK (possibly in their language of choice), and makes the mental shift to writing builds in lazy/functional style.

#

(I think demos so far tend to assume this)

#

But there's another approach that's been kinda-sorta discussed, which is that Dagger is mostly a tool for creating high-level build abstractions that team members consume -- maybe without having a really strong mental model of what's going on under the hood. In this world devops engineers and extension authors use the SDK; most people don't go quite that deep.

#

I'm pondering this because having spent about six months as my team's official "developer experience guy" now, I have a fresh appreciation for the value of meeting people where they are. I want to guide my team toward a world where builds are mostly self-service, and memoized and parallel by default...

#

...but what I've found is that most people don't really understand the full scope of what Docker/buildkit can do. Most engineers on my team don't know what multi-stage builds are! I kind of want to teach them because it's extremely cool, but I wonder if I'm dragging them down a rabbit hole when I could do them more good by giving them lines to color inside of. Not really sure. 🤔

junior basin
#

That's sort of what I'm using it for.

I made a declarative tool that uses dagger. You define inputs/outputs in a step, and give it an anonymous function for that step, which is expected to provide the output/consume the input.

The value for me here is that everyone wants the shorter feedback loop and confidence of running everything locally and consistently with your CI environment, but most people outside of a devops role aren't going to interact with it beyond copy/pasting existing chunks and changing a few things, whichis why having something higher level and declarative is really nice for me.

#

so I like giving people lines to color inside of (I like the analogy lol) but there's also a lot of value in reducing boilerplate and making the whole thing a little more readable that way too.

potent hemlock
#

@junior basin what language(s?) do your users write those anonymous functions in? Is there any kind of type safety/validation for the inputs and outputs?

junior basin
#

For the record I'm not saying this is the right solution for you or anything, just that it's something I'm experimenting with and I'm hopeful that other people at my org are receptive to it. We all use Go.

There's runtime safety but no compile-time type-safety.

Basically it looks like this. this is just some pseudocode:

var (
    ArgumentCompiledBackend  = state.NewFileArgument("compiled-backend")
    ArgumentCompiledFrontend = state.NewDirectoryArgument("compiled-frontend")
)

func actionBuildFrontend(ctx context.Context, opts pipeline.ActionOpts) error {
    path := filepath.Join(os.TempDir(), "frontend")
    if err := os.MkdirAll(path, 0755); err != nil {
        return err
    }

    return opts.State.SetDirectory(ctx, ArgumentCompiledFrontend, path)
}

var stepBuildFrontend = pipeline.NamedStep("build-backend", actionBuildFrontend).
    Requires(ArgumentGoDependencies, pipeline.ArgumentSourceFS).
    Provides(ArgumentCompiledFrontend)
var PipelineBuild = Pipeline{
    Name:     "build",
    Provides: []state.Argument{ArgumentCompiledBackend, ArgumentCompiledFrontend},
    Requires: []state.Argument{ArgumentGoDependencies, ArgumentNodeDependencies},
    Steps:    []pipeline.Step{stepBuildBackend, stepBuildFrontend},
}

var Pipelines = []Pipeline{
    PipelineBuild,          // Defined elsewhere for brevity
}

func main() {
    s := {packge}.NewDaggerWrapper()
    defer s.Close()

    s.AddPipelines(Pipelines)
}

Longer gist example: https://gist.github.com/kminehart/96ee896f7e1cea07e8078e04ef378193

At runtime it'll verify a few things, like if you're trying to use arguments in the state that haven't been provided yet by another step or if your function is done running and it declared provided arg but didn't set it in the state.

#

this is probably a stronger indicator of where most of our errors are since 90% of our pipeline failures are from people updating where files or folders are placed (on accident of course) and us not finding out about it until it actually runs in CI. but yeah this is just one example of where using dagger as a library to provide a more guard-railed experience could be really useful

potent hemlock
#

I'm curious, did you ever look at Earthly? If so, how would you compare their guard-railed experience to the one you cooked up? I kind of liked their model for end-users, but felt like it was pretty confining on the devops side compared to Dagger.

#

I'm idly wondering if eventually the Dagger community will want to have a popular high-level abstraction with a more opinionated framework-y feel that can be extended by individual teams. like, React:Dagger :: NextJS:?

junior basin
#

that's a good question. Personally i want that but i also acknowledge that what i want is probably different than what other people want so maybe an opinionated framework isn't for me.

I've never heard of Earthly but I'll have a look now.

normal sluice
#

The benefit from dagger which i hope I can utilize in my org is the fact that I can produce guard rails and docs in a language the devs are familiar with. Dagger is not gonna solve the problem with them not knowing how to use buildkit or docker, but at least I can wrap it in a syntax they understand, and provide clear comments on the files.

It will also vastly simplify our toolstack, as we now have a mix of shell scripts, executables, and so on. Which we could quite easily consolidate in dagger or rather go =D.

Another benefit is that we can rely on the language' own package manager for distribution of scripts and tools, which is super cool!

#

Also 99% of our devs probably wont ever touch our ci directly, as that is a service we provide, if they need something extra like adding a custom c lib for cgo or something, then they may need to get hands deep. But a lot of it is also for us, platform engineers, to have a better toolstack for developing these workflows. A lot of the time I am sitting waiting for CI to build to check that my tool changes work (bash is a finicky mistress). I'd also really like to more easily be able to test or CI pipeline and so on. In general bringing the general software engineering stack to platform engineering.

potent hemlock
#

I have a vague suspicion that many organizations could do with very similar guardrails, with org-specific customization around the edges by PEs.

#

But probably it's overengineering to look for those commonalities before I even understand my own org's ideal design.

#

I just hate the idea of duplicating work... 😕

heady sequoia
#

I think this is a great question to ask.

In my experience, people reject complexity (on any level) when it's not absolutely necessary for their job to understand.

Coming from this angle, I absolutely believe that most people should not interact with Dagger SDKs directly.

I had a discussion with someone the other day and I compared Dagger to GitHub Actions: most of the time, you just use ready-made actions in your pipelines.

When necessary, you build a JS or Docker based action, but most people don't do that.

#

The extensions feature will obviously make this much better (ie. you can reuse features across languages)

normal sluice
#

that is exactly the tricky thing. Abstraction is super difficult.

The way we've chosen to tackle it is to have a common opinionated abstraction for all of our tooling. As such if we at some point which from our in house solution to dagger, the user wont really have that much of a different api to interact with.

ie.

orgtool run build
orgtool run ci
orgtool run generate_swagger

each of these actions map into a project, such that js may use mocha, dotnet xunit and so on.

This works again for 99% of projects, but we still leave the option to override these in projects if so desired, because their project might not be compatible with our approach. Usually though we just upstream their changes so they can get back to mainline.

That is also something I want to solve to a degree with dagger. I want ownership in teams if they've got an ultra specific feature in the ci system they should own it, this especially goes for other platform squads, whom may not usually be responsible for developer tooling.

Otherwise the developer experience team would end up owning all configuration which isn't sustainable either.

#

using this approach we can get a service from creation to prod in under 5 minutes. if the reviews get in on time

potent hemlock
heady sequoia