eager glade
Hello everyone!
I am trying to deploy Dagger for my monorepo. I want to have a single Dagger setup at the root and be able to call my Dagger functions for all my apps located in /apps/appname.
My monorepo includes apps written in Go, TypeScript, and some using Bun. I am looking for a way to configure my dagger.json file to achieve this, but I haven’t found any answers on the web.
Do you have any ideas on how to do this, or perhaps an alternative approach?
knotty wasp
Hello everyone!
I am trying to deploy Dagger for my monorepo. I want to have a ...
hello! what kinds of functions? build? all identical across all your apps?
eager glade
I will use three stacks: one for Go, one for pnpm, and the last one for Bun, but all with Go SDK. I want to use something like a dictionary with the app name to select the appropriate stack. but yes same process, test, build, and publish on github package repo
mental canopy
I have a similar setup and its been pretty easy with dagger. Docker is our runtime for everything, so for the most part dagger just does a docker build with a specific target/args. So just a list of apps/packages get looped over, run a docker build, run a docker publish, etc. Similar thing for runtime configs as needed.
Think about your dagger functions being of two kinds: 1. basic functionality for one service (docker build, docker push, etc.) and 2. functions to put together a "pipeline" by taking a list of apps and running all the functions needed in 1 for that particular pipeline. This is useful for building pipelines up from smaller pieces you can debug individually, but we also use both types of functions locally for developers to debug and to setup a local environment.
knotty wasp
I've personally seen 2 patterns for this:
eg:
dagger call build-go-app --source=./apps/foodagger call build-node-app --source=./apps/barExample:
cd apps/foo; dagger call buildcd apps/bar; dagger call buildThe second option is the nicest to use IMO, and takes fully advantage of Dagger. In exchange for a little bit of Dagger boilerplate, you will save a lot of custom abstraction.
My recommendation would be 2. But you can also start with 1, and gradually upgrade to 2 later.
Extra benefits of option 2:
you can start with identical boiplerate for each app, but it leaves the door open for each app to customize (and customization in the build & test environment always comes faster than you expect it).
each app boilerplate can be written in the most appropriate language. Typescript for a node app; Go for a go app. This increases the chances of devs getting involved in improvement, debugging and maintenance of their Dagger functions, which is what you want.
You can model the dependencies between components of your monorepo as Dagger module dependencies, cross-platform. For example a frontend module can depend on a backend module. The docs module can depend on the api module (for auto-generated API reference docs). So you can start modeling your software supply chain as a dependency graph of Dagger modules, which is very nice
slow cradle
What is the best way to express cross-function dependencies in Dagger builds? I feel like they only work good for plain, linear builds. It's weird that Dagger runs aren't... DAGs of Dagger functions, and Dagger doesn't provide APIs or primitives to build such a DAG.
Because of that:
anyio.Event for synchronization (I'm using Python) This is necessary because some functions have to be awaited, and this would block the main program flow. So I await them in separate "mini-orhcestration" glue functions and rely on anyio.Event to start downstream functiuons (in their own launchers). If not doing this in Python, I would have to do it outside of Dagger - going back to the old and ugly yaml glue of a typical CI system. Ideally I just want to have a single Dagger function to run so that I don't have to write any GitHub Actions code whatsoever.I really enjoy Dagger as a build and execution engine, I absolutely love the existing APIs and concepts, but the lack of orchestration capabilities makes it quite painful to use it for a semi-complicated monorepo setup.
Maybe I'm doing something wrong? Should I just call multiple small Dagger functions separately (IMO that's bad because yaml glue)? Are there better ways to do orhcestration from Python?
I'm thinking about creating a Dageter job inside my Dagger code just to make expressing dependencies in Dagger eaiser.
obtuse oar
@slow cradle can feel you. I have broken up our whole monorepo with different services and technologies into a single graph of dagger primitives (files, directories, containers), dodging functions entirely just using the plain typescipt sdk. For us, there are no "linear" piplines but subgraphs of the dag that need to be materialized depending on who asks (just deployable image digests, published and pinned in a helm values file vs, linted, tests executed, CVE-scanned and checked by sonarcloud, with source map and and and - these different targets depend on the stage or scope who asks for something). This and the tui output/traces were somewhat usable until the pipelines api got removed from the sdk. I have tried to migrate to functions and modules but it just did not really work yet (also because functions returning nested complex types with lazy properties (again like File, Directory)) seem to be evaluated eagerly.
There need to be something better than calling the cli with different arguments. A way to compose different subgraphs, a way to control partial failures (evaluate as much as you can vs abort on the first failure). Hiding this behind the dagger cli call syntax somewhat feels like hiding buildkits capability behind dockerfile :/
slow cradle
Right. Exactly. I'm using Dagger for around 2 days and was wondering if maybe I didn't get something from the docs. But this seems to be it.
I'm seriously considering gluing Dagster and Dagger together to orchestrate these builds properly.
knotty wasp
dagger functions are how you compose the dag dynamically. Objects are lazy by default.
I need to dig into your specific implementations more, but would be surprised is the issue was really a fundamental limitation of dagger here
obtuse oar
Ill try to provide a minimal reproducer to better explain the issues that i was facing
knotty wasp
that would be great, thank you
slow cradle
Objects are lazy until you need to await something, right?
Let's say one step depends on an output from another step. In this case it's necessary to await it. This can't be done in the main function body since it will be blocking the function from building the rest of the graph - is this correct?
Therefore, these await s must be wrapped by some "orchestration" functions. The functions need to have a way to share state and events.
For example, my deploy step might want to know whether the test step has succeeded or not. Then it decide whether to proceed with the deployment depending on the current deployment environment (prod and review may have different rules and conditions).
As there orchestration functions grow it becomes more and more complicated to understand the code. Which steps depend on a particular step? Where will this event be awaited? Etc.
Maybe I just don't udnerstand things correctly. Some examples of monorepo setups with cross-function dependencies might be useful.
I'll also share some examples later on. I'm also open for a call.
eager glade
I've personally seen 2 patterns for this:
1. Call centralized functions with th...
thanks for your help ! I am working with one but go to two maybe soon !
wheat fractal
@slow cradle - Yeah, I don't think orchestration of tasks is what Dagger is meant to do. It's more about the (mainly CI) tasks themselves or rather whatever you can accomplish within a container. Just like a container should only hold a single service, I think the same goes for Dagger functions. They should only do one thing, and do that one thing well.
You can maybe parallelize a lot of similar work, like building for different platforms, in a CI task with Dagger, but it's not going to solve a whole CI/CD workflow for you.
In fact, because I never saw Dagger as a solution for this, I've been experimenting using Dagger in combination with Temporal.io. I haven't yet gotten far with it, as I'm still struggling a bit with Dagger and my own newbness, but I believe even for a proper CI workflow, which is durable, has retries, has good failures compensation, etc., I need to wrap Dagger functions inside Temporal tasks and use Temporal for the actual workflow orchestration. Dagger simply won't do all that for me, without me having to write a whole workflow framework myself.
You even mention CD (in your example your "deploy setup") and Dagger isn't even a deployment tool IMHO. It's output is mainly the artefacts ready for a deployment. The deployment process itself must be ran with other tools. Dagger won't help much there, unless you program your CD tasks and run it in a Dagger function and there too, you'll run into the missing features you'd need with proper workflow requirements.
So, again. Dagger isn't a complete workflow tool. It is a task runner for the tasks, which are steps within a workflow. At least that is how I see it. I might also be completely wrong. So, wait for someone else to jump in and either substantiate or correct what I said. 🙂
knotty wasp
Well the issue is that "orchestration" and "workflow" are big words that can be used in many different ways.
Features that Dagger doesn't have:
Typically you rely on CI or something like Temporal for the above 👆. From the point of view of those systems, Dagger is absolutely just a tool to run one task at a time.
BUT those tools are wrong.
Dagger actually does orchestrate workflows made of many tasks. It has a sophisticated scheduler, that routinely orchestrate thousands of function calls and the dependencies between them, in complex DAG patterns, every time CI asks it to run one "task".
I recommend you think of Dagger as a "micro-workflow" orchestrator. It doesn't dispatch events and jobs across your whole infrastructure - but within each job it can orchestrate a lot of your workflow's actual logic. And as Dagger gets better it will eventually be able to orchestrate all of it.
Here's the ideal architecture we want to enable for our users over time:
slow cradle
@wheat fractal
You even mention CD (in your example your "deploy setup") and Dagger isn't even a deployment tool IMHO
I agree. But my deploymet step is a very simple one - a third-party SaaS CLI call (it triggers the actual depoyment on Kubernetes). So I decided to just do it from Dagger.
So, again. Dagger isn't a complete workflow tool. It is a task runner for the tasks, which are steps within a workflow.
Agree. I guess I just had this misconception of treating Dagger as a complete replacement for traditional CI/CD systems.
I understend Dagger's scope better now. Similar to you, I am trying to orchestrate Dagger functions from Dagster (might not be the best tool for this job but I know it really well so it's easy to try).
@knotty wasp makes complete sense!
wheat fractal
I recommend you think of Dagger as a "micro-workflow" orchestrator.
As I've noted above, I've been thinking of Dagger mainly as a task runner for anything that can run in a container, with the idea being, this takes out any kind of tech-in-the-middle worries about the infrastructure on any particular compute node (be it a server in the cloud or a local laptop) i.e. the reproducibility Dagger wants to guarantee to its users.
Another way I've been thinking about Dagger is, it turns Docker inside out, by allowing Docker to be ran imperatively, instead of declaratively i.e. with code instead of config.
Dagger actually does orchestrate workflows made of many tasks. It has a sophisticated scheduler
What I don't yet get is how Dagger can, other than via looping in a module function, run anything in a "workflow" manner. Any workflow I believe that can be gotten via Dagger currently is only available through the code I write, not via anything Dagger offers me directly (currently). I've dug into the SDK docs and I just don't see any methods offered, which can give me workflow capabilities.
The "sophisticated scheduler", that you mentioned Solomon, isn't documented (that I could find). In other words, the documentation does not explain what can be expected of Dagger in terms of any kind of "scheduling" or "orchestration" of tasks. In the docs, there is this:
Dagger Functions are the fundamental unit of computing in Dagger. Dagger Functions let you encapsulate common project operations or workflows, such as "pull a container image", "copy a file", and "forward a TCP port", into portable, reusable program code with clear inputs and outputs.
Those examples are tasks and no mention about how they can be "orchestrated".
I saw the section with chaining of functions in a Dagger CLI call. That isn't workflow. It's a combining of tasks. For instance, what happens when the first task fails? Are there retries? Are there ways to fallback? It's not workflow to me.
In fact, there is a whole section on "Integrating with CI" in the docs. If there are workflow capabilities in Dagger, these integrations wouldn't be necessary. I believe? Theoretically, if I ever get further with Dagger working with Temporal, I can offer a PR to add a section to describe how to do that too. 🙂
I guess I'm missing the "workflow" offerings in Dagger and sorry if this sounded somewhat smartass-ish. I know I tend to write that way and can't turn it off. But, I'm really just trying to understand where Dagger needs to fit in my tooling. I thought I had a handle on it, but your comments are throwing me off some. Thanks for your understanding.
wheat fractal
Let me explain my use case with Dagger and maybe it might explain where I want Dagger to be or rather, where I believe it fills in the square.
We have an RDE (remote development environment) setup, which is the basis to a k8s dev platform. We don't wish the developers, however, to have direct access to Dagger inside the system. That is simply "too much power". So, I'm developing our own CLI that will call an API service, which entails Dagger functions to do certain things in a very controlled way.
So far, so good. In fact, part of this work is done.
These same tasks, however, will also be needed in a webhook API for the CI workflow kicked off by a Github app. For instance, when a PR is merged, it will kick off a number of tasks, including some the dev wouldn't be doing, like generating the final container and promoting it, doing security scans of the repository and its dependencies, specialized linting for platform security, etc. All this needs to be automated and away from the dev. For this and other webhook workflows, we will be using Temporal and in the activities in Temporal, we will be running our Dagger functions.
For CD workflow, we will be using Kargo and ArgoCD.
Is this a proper plan of attack?
knotty wasp
Sounds good @wheat fractal . You're definitely on one extreme of the freedom / control spectrum in terms of how much you wrap Dagger and hide it from the end user, but that's a function of your own constraints. Seems like a very valid use of Dagger.
One question would be: why have CI trigger a webhook service managed by Temporal that calls Dagger, when you could perhaps have CI itself call Dagger?
wheat fractal
@knotty wasp - Good question. It's because we want guarantees (Temporal calls it - durability) that the CI steps work 100% to completion. Temporal allows for retries, for instance. Or, if after X failures, we wish to have the means to revert back any changes made to the repository to an initial state. We will also be running multiple hook/ workflow endpoints and activity (task) runners and Temporal will allow us to scale these out too fairly easily.
knotty wasp
I think it would be beneficial for Dagger to have an official Dagger/Temporal integration, so you can do this kind of setup once, in a way that is not too tied to your particular workflow (because that workflow will evolve over time)
I even think eventually we might expose some of those Temporal features from the Dagger API - essentially making it possible to use Temporal (or an interchangeable equivalent, there are many competitors) as a middleware for Dagger, instead of a wrapper
wheat fractal
That's an interesting take. I do believe though you'll be hard pressed to find standardized APIs between the workflow tools to allow for this kind of plug-in-ability or middleware integration. In the end, I do believe any workflow solutions will have to wrap Dagger. But, me saying this might come from my own lack of understanding, cause I am still feeling like I'm missing something with Dagger. 😛
For instance, I'm not sure I understand what you say here:
so you can do this kind of setup once, in a way that is not too tied to your particular workflow
Let me explain my train of thought and understanding about Temporal's and Dagger's "places" in the scheme of how the workflows should work with Temporal.
With Temporal, Dagger functions will be found only in Temporal Activities.
Activities are called inside Temporal workers, which pole a task queue in the Temporal server to see if any work must be done. They are "dumb" in the sense they have no clue about workflow.
On the other hand, the tasks in the queues are generated via other Temporal workers running the workflow code (which have no Dagger code in them). When a worker with workflow code is "kicked off", say via a webhook, the workflow puts one or more tasks into queues and waits for a "reply" that the tasks were successfully accomplished. Then the next steps in the workflow are carried out (more tasks for Dagger or other work) or the workflow is finished, as Activities can be ran in parallel.
That's a rough and quite simplified explanation of the Temporal process, but I hope it can help you understand my confusion of what you noted. Dagger in Temporal would have no knowledge or reference to any workflow. 🤔
Theoretically, Dagger can run a number of the steps or tasks for a CI workflow itself. We'll need to contemplate how much should be done in "one shot" with Dagger. At this time, we wish to keep all tasks fairly simple and have Temporal control accomplishment.
slow cradle
So I finally had my "aha" moment with Dagger when I stopped trying to orchestrate multiple high-level pipeline steps (build, lint, test, publish, deploy) inside a Dagger function and just made a separate Dagger function for each step. Proper caching is really all you need.
I also figured I could just parse the uv.lock file in my monorepo to get cross-project source code dependencies from there.
This resulted in a very clean and simple Dagger pipeline and I am really happy about it. I now truly appreciate being able to just write Python code inside Dagger pipelines. I can't imagine doing something like this with other tools.
I published my approach here --- hopefully other people find it useful (it is the monorepo thread after all)
slate storm
So I finally had my "aha" moment with Dagger when I stopped trying to orchestrat...
cc @ember trout for content material
ember trout
So I finally had my "aha" moment with Dagger when I stopped trying to orchestrat...
Thanks for sharing @slow cradle !
obtuse oar
Ill try to provide a minimal reproducer to better explain the issues that i was ...
Here comes the reproducer @knotty wasp . Sorry for taking that much time. For context again, I would have expected that custom return types in Typescript containing dagger-objects (in this case Files) would be evaluated lazily, but it turns out, returning a custom return type would just return a lazy discoverable type, but turns out, the all fields on that custom type seem to at least be materialized (synced), even if noone asked for it. In this demo, I am asking for "out" but "slow-out" is materialized as well.
https://github.com/danielwegener/dagger-examples
slate storm
Here comes the reproducer <@488409085998530571> . Sorry for taking that much tim...
cc @mossy oracle not sure if this one has/deserves an issue
mossy oracle
Here comes the reproducer <@488409085998530571> . Sorry for taking that much tim...
At first glance, out and slowOut are defined as state in NestedLazyThings, instead of functions. When that object is returned in foo it needs to serialize both properties (out and slowOut) before passing on the result to the API. If you turn the NestedLazyThings type into a class decorated with @object, and out and slowOut into methods decorated with @func(), you should see the effect that you're expecting.
So to sum up, here's a few basics that help understand this:
type instead of class is meant only for passing state, not functions (see https://docs.dagger.io/api/custom-types)Breakdown of what's happening:
dagger call -m ...: The engine asks the Typescript SDK to construct your module's main object (DaggerExamples). The SDK then serializes it (empty {} in this case), and returns that result to the API.... foo: The engine asks the Typescript SDK to invoke your module's DaggerExamples.foo function, using the state from the previous call result. Since this call returns an object with state, the SDK serializes (i.e., resolves) both out and slowOut and return their results as a JSON object ({"out": "<file ID>", "slowOut": "<file ID"}). The state is saved in the server.... out: Module is not called because API server has the value, so it's returned directly (a FileID in this case).From there onwards it's the core API that's being interacted with.
slate storm
At first glance, `out` and `slowOut` are defined as state in `NestedLazyThings`...
thx for the thorough explanation Helder, makes total sense. I completely forgot the Class state vs functions 🙏