One call to rule them all

[simple shell]

I think we’re very close and there are people doing this today. The only major thing that’s changed in recent times is using “dagger call” instead of “dagger run” - especially if you’re working on a large cross language or cross provider team there are many benefits to using dagger call that make things extremely portable.

[worthy hamlet]

This is what I'm trying to do right now, but constructing in such a way that there's a high level function I can target with dagger call which invokes everything else necessary to achieve the entire pipeline. Is that the typical pattern?

Or do people still leverage e.g. Gitlab stages and do distinct dagger calls in each?

[simple shell]

Yes there’s normally a single function that calls other functions — a decent example in one of my own sample project here: https://github.com/levlaz/snippetbox/blob/a61b1eb0ae92c66509f6d78089345c70d21e161a/ci/main.go#L35

There’s also a Gitlab example in that repo.

[cyan lagoon]

@worthy hamlet I am still trying to figure out how I can have a reliable porting from the GitlabCI pipeline to a full Dagger module / set of funcs.
@simple shell thanks for the example

[austere moon]

see here for a single-function approach in github actions: https://www.youtube.com/watch?v=JVyb7KGEVsE&t=380s&ab_channel=Dagger

[simple shell]

Either way it’s not all or nothing, the main thing you lose when you’re putting everything into a single call is some visibility, but dagger cloud and our new traces feature aims to resolve that.

The beauty is that it should be a very simple and reversible decision. If you write your functions in a way that makes them easy to compose then you can call them one at a time or all together without needing to do anything special. You also get a CLI for free every time you write a function so you can compose things via the CLI as well

[worthy hamlet]

I suppose there are only a few CI platform-type features I would want in addition to the Dagger engine. I wonder if there's room for a very minimal self-hostable platform designed just for Dagger, that just pulls in observability things from Dagger Cloud and provides a few tools for things like scoping environment variables, stopping/retrying parts of a CI run, etc.

[austere moon]

i might sound crazy but kubernetes. perfect fit for thinking in terms of containers only

[simple shell]

Yeah for sure, I think one day there will be something like that provided by dagger and/or dagger cloud but still building a way that is provider agnostic.

When it comes to things like environment variables though, we’ve seen a lot of success using tools like infisical along with dagger. I think keeping configuration and secret management outside of a CI provide is a good practice because it’s one more thing that allows you to built a portable pipeline

[worthy hamlet]

I have plenty of experience with k8s, but don't fancy maintaining a cluster to get CI. It would be fine to host such a tool on, but I would prefer if it could be run off a single node, since Dagger really does great container orchestration by itself.

[austere moon]

ah well then there's really no reason not to just boot up a cloud instance and run dagger on it

if you don't need more than one node i'm pretty sure you don't need anything else

[worthy hamlet]

Haha, I have a call with the Infisical guys today for a demo! I was already planning to integrate this into my PoC, and I'm glad I'm not the only one.

[worthy hamlet]

Maybe only the ability to invoke/cancel/retry individual bits of the pipelines via GUI, and grab stored artifacts and other managerial bits you might want anyone to do. I think it'd be quite small.

[simple shell]

That’s great to hear. They are an awesome team 🙂

[barren shoal]

See #1214729466920706088 for more inspiration 🙂

[worthy hamlet]

https://www.youtube.com/watch?v=xcWcR4rZy44 This also looks super relevant!

[barren shoal]

oh yes! absolutely

The former is a good gateway drug to the latter 😁