#Blueprints 🟦 + Toolchains 🛠️

1 messages · Page 1 of 1 (latest)

sage thicket
#

Hey @azure bronze just been messing around with blueprints and toolchains. A kind of cool composability use case has come up and I wanted to share it with you.

#

The context is we want to install a blueprint module in repos which will serve as an "orchestrator", running various tasks

#

Those tasks would include, building the code, running the linter, pushing an image to the registry etc.

#

We plan on loosely pinning this module to our repos so we can roll out updates to them without having to continually update each one.

#

However this last point introduces a caveat which is that now we cannot guarantee a build will work the same twice...

#

So naturally we were leaning towards somehow hard pinning tasks that build the source and run the linter, so that they had the same behaviour each time...

#

I can see this working with toolchains.

# 
{
  "name": "toolchains-test",
  "engineVersion": "v0.19.6",
  "blueprint": {
    "name": "go-service",
    "source": "../daggerflows/go-service"
  },
  "toolchains": [
    {
      "name": "go-lint",
      "source": "../modules/go-lint"
    }
  ]
}
#

Please ignore the relative references... imagine the first one (blueprint) was loosely pinned to main and the second one was hard pinned to some specific tag.

azure bronze
#

I really don't think you want any form of loose pinning at runtime anywhere in dagger

sage thicket
#

In this setup developers can run the linter directly via toolchains - very handy:

dagger call go-lint check
#

Well this is where the sort of benefits of both might work for us...

azure bronze
#

(I understand why it's tempting to want it)

sage thicket
#

I discovered that the blueprint module can actually see the linter as a dependency module.

#

Using a bit of graphql I can make the blueprint go-service call the linter module:


func (g *GoService) linter() *querybuilder.Selection {
    client := querybuilder.Query().Client(dag.GraphQLClient())

    return client.Select("goLint").
        Arg("linterVersion", "0.173.0").
        Arg("goVersion", "1.24.3").
        Arg("ssh", g.SSH)
}

func (g *GoService) Build(ctx context.Context) error {
    return g.linter().
        Select("check").
        Arg("since", "asdf").
        Arg("src", g.Src).
        Execute(ctx)
}
azure bronze
#

Can you describe your criteria for a valid use of loose pinning? What has to be true for it to be worth losing repeatability?

sage thicket
#

There are certain things we want to be able to distribute to our developers without them really knowing or caring. For example security scanning, integration with pager duty, integrations with DX

azure bronze
sage thicket
#

Yeah this is great.

#

One of the advantages of toolchains is that you don't need to proxy access to dependencies which can actually perform some task for the end user.

azure bronze
#

Also this pattern 👆 seems relevant maybe?

azure bronze
sage thicket
#

What is dang ?

azure bronze
# sage thicket What is `dang` ?

It's a dagger-native scripting language that requires no codegen, and as a result is ultra-fast. Part of our "need for speed" roadmap.

Note: the blueprint-for-vendoring pattern itself is not specific to dang 🙂

sage thicket
#

Oh very nice.

#

Toolchains should help us out a lot. We want to install things like the linter and the compiler as seperate toolchains in repos.

#

And I found out that the blueprint module can also see toolchain modules and can invoke them over the graphql api...

#

Which suits our hacky not-recommended loose pinning strategy

#

hahaha

#

Perhaps we need to have more of a chat about that so you are aware of what we're doing.

azure bronze
azure bronze
sage thicket
#

Yeah thats understandable and is the reason I wanted to share with you in the first place

#

I felt I was going a bit offroad, and we want to be supported no matter what so yeah

#

If you have some time it would be good to have a meeting with my team at some point

azure bronze
#

And I want to make sure our design takes your actual use case into account

#

Don't want to take any toys away unless it's to trade them for better toys 😄

sage thicket
#

Sounds good

azure bronze
#

@muted comet what's your timeline on this? The sooner the better?

#

By "this" I mean your overall dagger project. Are you under time pressure for certain milestones, or more of a continuous improvement?

sage thicket
#

Can do next week

#

Right now the rollout is only to 3 teams in the company.

#

We have collected a lot of feedback and now are trying to "finalise" the design of the CI.

#

Next year will be crunch time.

#

So ideally we would like to have the kinks out by the end of the year

azure bronze
#

OK so in addition to learning about the details of your rollout, we should also show you the details of what we're shipping next 🙂 So you can take that into account (nothing breaking don't worry, except maybe that graphql backdoor 😛 )

sage thicket
#

haha!

#

all my graphql hacks no!

#

all good

#

How do I book you ?

azure bronze
#

(sent you a DM)

wind glacier
#

This is an interesting use case that we will also have. We distribute CI code to our developers all the time without hard pinning so we can actually perform necessary updates/upgrades any time we need to. The devs can of course hard pin if they want to, but they are on their own at that point.

Essentially this is how our jenkins shared libraries work. The default most folks use is a soft pin. I'm not saying that's right for dagger. But we would probably still have some toolchains that we want to distribute with that model. Mostly liniting and security related.

azure bronze
#

It's not like you have to open a PR and wait for the dev (who doesn't care) to approve it. You can just direct-push to git. It's just as intrusive, gives you just as much control, but you get a proper audit trail (and repeatability)

#

But, I guess you could also get the audit trail from the trace if we allowed soft-pinning

wind glacier
#

like force push an update to the developer's main? That would probably be a very difficult task to achive given enterprise access guardrails

azure bronze
#

Or, what about allowing the client to auto-update the pin on run? So they would get an uncommitted diff when they run if it happens to update the pin

wind glacier
#

people are very ansy about central teams touching their repos

azure bronze
wind glacier
azure bronze
#

What about the dependabot pattern?

#

You probably have a system in place, dev-tolerated and security-approved, for every other central dependency getting pushed to devs?

wind glacier
wind glacier
#

basically it's the soft pin

azure bronze
#

Let's say we enable soft-pinning (perhaps as an opt-in flag). Aren't you worried about the downsides? Ie. lack of end-to-end repeatability

#

Is there a scenario that leads to "well Dagger isn't reliable" because of soft pinning

wind glacier
#

There is no reliable way to use something like renovate for dagger (yet) so pinning all docker images we pull to a SHA is in of itself a security issue.

azure bronze
wind glacier
azure bronze
#

Actually a lockfile could make it more ok to have softpinning I suppose

wind glacier
azure bronze
#

What's missing on our end to work with renovate?

#

I guess develop a renovate plugin?

wind glacier
azure bronze
#

Suddenly soft pin seems so much more appealing 🙂

wind glacier
wind glacier
azure bronze
#

so it would be perfectly repeatable but on your terms

#

and btw the lockfile also captures runtime dependencies like container pulls from third parties etc

#

so it makes not just the build of your dagger module repeatable, but it's execution also

wind glacier
#

That makes sense!

wind glacier