#Daggerizing without boilerplate

1 messages · Page 1 of 1 (latest)

vague thunder
#

🧵

#

@ocean quartz creating a wrapper is an antipattern, but I understand why you feel the need to create one. We have a design in the pipe to get you the best of both worlds: removing the need for tedious boilerplate, without sacrificing the native composition model. I know in the beginning of a wrapper it doesn't feel like you're sacrificing anything because everything is fresh and clean, but sooner or later the exponential complexity catches up with you. Anyway, a wrapper is an acceptable stopgap as long as you keep your eyes open to the fact that it's a stopgap- the day you fall in love with the wrapper is the day you're in trouble 🙂

Separately, I strongly advise against the "loose versioning" via targeting a tag/branch from the CLI instead of pinning. Use dagger dependencies for this. I understand the urge not to, but you'd be injecting unreliability at such a low level in the stack; when later you wish you had more repeatability it will be very hard to retrofit it in. This is orthogonal to the boilerplate issue. You can use dagger.json as a receptacle for dependencies without any code boilerplate. It beats inventing a separate meta-config file that will reinvent the same concepts

ocean quartz
#

Hi Solomon is it possible to have a dagger.json without any SDK?

#

Would be interested in that.

#

Loose versioning yes... I get where you are coming from here. Thanks for the thoughts I will take them to the team

#

It's a tricky line of decision making here as there is a school of thought that it might be acceptable for certain modules. Not the builder module, but modules executing higher level CI patterns such as putting a comment on a PR.

#

But yes aware that it is going against the grain

high bay
#

you can then use dagger update / remove / etc to control which version you want to pin it to

ocean quartz
#

Excellent, just tested that and it works

#

To solve the issue of having a seperate file to contain the builder module, it would be ideal to use the dagger.json for this as well. If we were to do this however, we would need to somehow pass the builder module into the ci module's interface type.

#

This would most likely require a local dagger module I imagine.

#

Unless there is a way to fetch the module dynamically using the dag.Module() api and do something similiar to what was doing previously

#

Here is what I have right now in my "CI" module ( which wants to accept the builder module ):


func resolveBuilder(ctx context.Context, ref string, source *dagger.Directory, sshSocket *dagger.Socket) (builder Builder, rerr error) {
    ctx, span := Tracer().Start(ctx, fmt.Sprintf("resolveBuilder: (ref=%s)", ref))
    defer telemetry.End(span, func() error { return rerr })

    mod := dag.ModuleHelper().
        ResolveModuleGit(
            "git@github.com:nine-digital/dagger-go-builder",
            ref,
            sshSocket,
            dagger.ModuleHelperResolveModuleGitOpts{SubDir: "module"},
        )

    return asGolangServiceBuilder(ctx, mod, source, sshSocket)
}

func asGolangServiceBuilder(ctx context.Context, module *dagger.Module, source *dagger.Directory, sshSocket *dagger.Socket) (Builder, error) {
    client := querybuilder.Query().Client(dag.GraphQLClient())

    err := module.Serve(ctx)
    if err != nil {
        return nil, err
    }

    name, err := module.Name(ctx)
    if err != nil {
        return nil, err
    }

    var id string
    err = client.Select(toCamelCase(name)).
        Arg("source", source).
        Arg("sshSocket", sshSocket).
        Select("asGolangServiceBuilder").
        Select("id").
        Bind(&id).
        Execute(ctx)
    if err != nil {
        return nil, err
    }

    return LoadBuilderFromID(dag, BuilderID(id)), nil
}
#

I then call resolveBuilder from the constructor

ocean quartz
#

I guess the setup I am describing is something of what peerDependencies in npm give you.

vague thunder
ocean quartz
#

Thank you for all your responses guys. Let's keep the discussions going. I am meeting with my team now and will generate some more questions for you soon 🙂

vague thunder
#

thanks for all the details it's super useful

vague thunder
#

This motivates me to improve our support for centralized platform modules. I believe we should make them a first class citizen in the API.

$ git clone github.com/my/app
$ dagger init
$ dagger install --platform github.com/my/platform
$ dagger functions
platform-thing: do a platform thing in the current context
other-platform-thin: do another platform thing in the current context

A platform module would simply be a dependency with a special flag. The flag changes how the dependency is loaded: instead of being sandboxed in its own context, it would be loaded in the parent's context. So +defaultPath would load files from the parent's repo.

Besides this special loading behavior, dependency management would work exactly the same.

ocean quartz
#

That would be good. Yeah no worries Solomon

#

Yeah just got off a call with my team and there is definitely a desire to have something like this peer dependencies / platform module thing available. Obviously we haven't started rollout and we don't know for sure we need it, but it does seem useful.

high bay
#

@vague thunder mind opening an issue given that you have the broader picture here?

vague thunder
#

I think we wrongly assumed dev teams would love developing pipelines in their language & being able to run them locally. But overall they don't care. Dagger solves a pain for platform engineers, so the added complexity of learning Dagger iis worth it (to a point!) but devs don't experience that pain so have less patience for the extra complexity.

The natural reaction for platform engineers is to centralize dagger development. We have been in denial about this IMO, and as a result Dagger doesn't make this centralized pattern super easy. So platform engineers resort to wrappers. Which makes dagger appear even more complex.

So it's time for us to embrace centralized daggerization and meet platform teams where they are - starting with native support for platform modules.

Thank you for attending my ted talk 😛

charred spruce
#

A platform module would simply be a dependency with a special flag. The flag changes how the dependency is loaded: instead of being sandboxed in its own context, it would be loaded in the parent's context. So +defaultPath would load files from the parent's repo.
@vague thunder So instead of being sandboxed in the normal dagger.json or git repo context boundary, it would have a vantage point of one directory above that and be able to "look down on the project and itself"?

vague thunder
charred spruce
#

I was thinking of the "one level above" thing for the case where your context boundary needs to be "moved one directory further out"
#1372555736147562507 message

ocean quartz
#

A feature like this would be amazing for our use case here. You are absolutely right that as a platform team we are trying to offer CI/CD as a service to our developers and therefore minimal boilerplate is desirable, simplifying the experience and reducing potential future drift. The concept of "platform modules" would really help with this. This would remove the need to supply the --src parameter everytime, getting rid of one extra flag.

As an added bonus if said modules were accessible from the context of another module that didn't directly import them (like a peerDependency), this would be quite powerful. Yes it does away with some of the feedback you get in your IDE... right now if module A expects interface X, then if I try to pass module B to it, I would get a compile error if B doesn't correctly implement X. If this feature were to exist it would probably need some disclaimers like "use the normal module pattern unless you really have the following use cases ...". The advantages of this feature for us however would be that we can ship two types of modules:

  • CI componentry modules: concerned with processes that aren't really going to be executed on the local dev machine - we would pin these to a branch or "channel" (stable, edge) in the repo. Processes include: deployment to an external system, security scans, putting the build result on the PR ( for PR builds )
  • Builder componentry modules: concerned with producing artifacts from the source code - we would hard pin these in the repo. It would do things like: run tests, run the linter, build the release image

You could have a dagger.json like this ( just a wild stab 🙂 ):

{
  "name": "main",
  "engineVersion": "v0.18.4",
  "dependencies": [
    {
      "name": "ci",
      "source": "github.com/nine-digital/infrastructure-dagger-pipelines/golang-service",
      "platform": true,
    },
    {
      "name": "builder",
      "source": "github.com/nine-digital/infrastructure-dagger-builders/golang-builder",
      "pin": "<sha>"
      "platform": true,
    },
  ]
}

developers could run common tools through the builder module

dagger -m builder call test
dagger -m builder call build
dagger -m builder call lint

developers could also run the CI modules commands locally too (if they want to before they raise their PR):

dagger -m ci call check-pull-request

Our "golang-service" CI module could perhaps "see" the "golang-builder" module so it can call into it and run the functions it needs to test, build, lint when check-pull-request is run.

#

This might be a bit ambitious and probably more thoughts and considerations are required to do this in a safe way. Making a module globally accessible to all other module contexts might pose security issues. For example if a malicious dependency module decided to call your builder module and somehow obtain stuff it shouldn't have access to... There might need to be controls around something like this!

#

But yeah I guess the main point I am trying to make here is not the "how" but a bit about the "why"/ use case we have. Implementing it would require more thought 🙂

vague thunder
#

@vagrant tendon for when you're around on UK time.. I took a stab at implement this "platform module" feature, but it's not working the way I was hoping 😭

I think the problem is I hijack the context directory at the wrong time:

  • I need the platform module's context directory when loading its own functions
  • But then I need the original context when resolving default path etc.

The diff is actually quite small... Any insights would be very appreciated https://github.com/dagger/dagger/pull/10442 🙏

GitHub
Platform modules by shykes · Pull Request #10442 · dagger/dagger

This is a WIP PR for implementing &quot;platform modules&quot; as discussed in this discord thread
NOTE: it doesn&#39;t currently work...
I think the problem is I hijack the context dir...

brave grotto
#

thanks @vague thunder for pinging me, I'm definitely interested in this topic, because it's clearly the main issue we see with dagger adoption at the enterprise level here (ubisoft).

As you said, apart from the early adopters, most devs don't care about the details of "how" the build/CI/... works, they just want an easy way to define what they want at the end: a binary uploaded there, a container image pushed here, etc.

We already have an internal tool that was built a few years ago, on top of docker and using a declarative (YAML) kube-like syntax to produce container images, helm chart, and deploy them. And people like it for a few reasons: similar syntax as kube manifests (these are people used to deploy stuff on kube), a consistent way to declare where these images will be pushed (so you can open another repo, and you know where to find that information), and no need to care about details such as how to run the "docker buildx ..." cmds.

This is what I'm trying to solve with https://github.com/vbehar/mason - even if I totally agree with you about the dangers/challenges of writing a wrapper. The good news is that for the moment I'm still more in love with Dagger than with Mason 😉 And I'd be happy to drop it the day Dagger natively supports this "use-case" of scaling the dagger adoption without introducing too much boilerplate (and maintenance work) on every repo.

GitHub
GitHub - vbehar/mason: Declarative build tool based on Dagger

Declarative build tool based on Dagger. Contribute to vbehar/mason development by creating an account on GitHub.

vague thunder
hardy ember
#

apart from the early adopters, most devs don't care about the details of "how" the build/CI/... works, they just want an easy way to define what they want at the end: a binary uploaded there, a container image pushed here

This is so true in my org too. I've tried my best to make modules that hide a ton of the behind the scenes boilerplate and logic but it's still a hard sell for most devs. Something about having to write code instead of adding a config file and running commands turns them off. Not everyone though, there are folks that love writing dagger code, but that's not the majority. Platform folks absolutely love it but hard to get adoption if dev teams don't either.

hallow fjord
#

I have dozens of project repos that have my dagger build repo checked out as a git submodule, which i'm not sure how I feel about. but each repo can opt-in to receiving updates to the dagger build repo with git submodule update.

plain sigil
#

Hey! Very intersted in this too. As mentioned before, dagger adoption past the early adopters is tough for our engineers (Fastly, 500+ engineers). We're a platform team of 6 with plenty of in-flight projects, one of them replacing our Jenkins setup (~1,000 jenkins pipelines) with a new dagger based CI platform.

We've gone through a couple of iterations (wrapper -> ditch wrapper for good re-usable building blocks -> initialize all the boilerplate). We found the hard way having a wrapper is a limiting factor for a heterogenous ecosystem. Maybe if your company has a well defined, homogenous ecosystem I would pursue this approach.

What has worked in the past for our previous CI system and we're about to give a shot, is a dagger module (obviously) that does all the boiler plate to get you to a working starting point without having to write a line of code. Sort of templatized examples. That way, the learning curve flattens a bit, as you have a working example to start from

The natural reaction for platform engineers is to centralize dagger developmen
This is quite true for us. We develop MOST of the dagger code (common functionality, building blocks) and engineers will take those building blocks to create their pipelines. That is still a lot of work (learning how to write dagger, install deps, learn our APIs, ...). Lowering that friction would be amazing (Did I hear platform modules with access to the parent context... chefkisschefkisschefkiss )

ocean quartz
#

Thanks for the quick response on this Solomon this is very cool! I love this feature and we would definitely use it. Gave it a very quick test on my machine and it works

vague thunder
#

Platform modules, take 2. This time as a flag on dependencies, which means there can be more than one... and they are namespaced.

Unlike take 1, context dir is correctly loaded. The core functionality works.

https://github.com/dagger/dagger/pull/10453

GitHub
Platform modules: take 2 by shykes · Pull Request #10453 · dagger...

This is a second take on the &quot;platform module&quot; feature. (First take: #10442 )
In this version, we add a platform: bool field to dependencies, instead of a top-level platform: { .....

jolly sphinx
#

Does this mean that export will work, for platform modules?

#

Or could work?

vague thunder
vague thunder
#

@marble falcon @hardy ember 👋 let's bikeshed here 🙂

#

There's a question about the name "platform module". Perhaps not the right one?

#

But what are alternatives? Here's what I asked chatgpt yesterday

#

It gets a little confused, but a few options emerge:

  • embedded
  • inline
  • resident
  • surrogate
  • in-situ
marble falcon
#

I tend to call them "high level modules", but that's not a great name

hardy ember
#

I am on the same page as Mark. "Platform" doesn't sound right.

#
  • managed
  • central
marble falcon
#

I wonder if it's the module that needs a name. I mean, platform modules are just regular modules, aren't they. So it's not the module that is special, it's the way it's used. Central/managed/platform, those are just classifications for those modules.

So maybe it's the feature that needs a name.

Module wrapping?

vague thunder
#

Yeah it can be applied to the dependency

#

a "platform dependency", "inline dependency"

#

"surrogate module" also works because the word "surrogate" inherently defines a relationship. (It's intuitive that it's surrogate for another)

jolly sphinx
hardy ember
#

this is a tough one. I don't think I understand why it's a surrogate? The's the same module but it behaves differently based on how you consume it.

vague thunder
#

The app team initializes a module in their app. But, instead of implementing their own types and functions (because it's too much boilerplate), they use a module developed for them by the platform team. It's a surrogate module because it's executed on their module's behalf, as if they had copy-pasted its code into their own module.

vague thunder
#

I think "inline" might be the most technically accurate

ocean quartz
#

yeah I think inline might make the most sense.

ocean quartz
#

Thanks for your fast response on this Solomon. Can definitely see that second option coming in handy 🙂
The benefits would be the namespacing and the ability to supply multiple modules as "inline modules".

#

The first option's benefits are that if you want to have a "central module" which encapsulates the entire tooling for a repo, you can just drop it into the platform field and then the dagger command works without any -m flag... very nice

hardy ember
#

Sounds to me like its more of a managed module from the standpoint of a layman developer who knows nothing about dagger and may not care to know about it unless they are exploring deeper. I guess every centrally hosted moule is a managed module though and managed modules can themselves be "inline". As I say this, it makes me think this complicates things more.