Feedback on CICD tool idea

[simple mural]

Hi Pedro! Yeah it's fine here, here's a thread to continue the discussion! What's your tool idea?

[vague lion]

Hey everyone, I'm working on a tool related to CICD pipelines and wanted to ask experts here:

Elevator Pitch
The tool is specifically for monorepos with lots of libraries, services, test-suites, and the like. I'm starting from the assumption that teams only want to test and deploy what they need to - if I just modify a test suite, I want to run that test suite. If I modify a library, I want to test the library, the services that the library uses, and the re-run the test suite that touches the library.

I'm making an OSS project that makes it so teams can have a dependencies.toml at each node (service, lib, test suite, etc...). Then a command is ran in the CI which computes the DAG of nodes that affect each other and uses the git diff to output the nodes you need to do stuff with (could be testing, deploying, etc...).

The idea is that distributed teams can just tune their dependencies.toml to point to other services and the configuration doesn't have to be set up by a "devops team" 3 timezones away and 12 hours later - the CI can evolve independently.

It's like Bazel's graph system but without Bazel.

Questions

• Does this make sense? Am I making false assumptions or is the idea sound? If you can poke any holes in it or completely destroy my idea I'd greatly appreciate it!
• Do you know of any tools like this already?
• Is there any tool like this in Dagger?

Very small example: https://github.com/taco-xyz/dependency-cascade-testing/tree/main/example

[sturdy shard]

@neat beacon the pitch seems similar to Nx? https://nx.dev

Just wondering if you've checked it out and how your tool would differentiate

[vague lion]

Yep! That's like Bazel too! My gripe with that is that they include the build system too, I just wanted to make the graph part and make it agnostic to CICD pipelines.

My comparison is that they're doing the house with the kitchen-sink approach and my tool is just a small OSS tool to chain into other tools. It can be a module in the daggerverse, a custom GitHub action, a jenkins action, etc...

It's actually just a Rust CLI with two commands 😅

[sturdy shard]

I see..well that makes a ton of sense. It's also something that I do see a lot of value in Dagger as well which Dagger doesn't currently solve in the best way possible

so you could potentially use Dagger as your first guinnea pig project to see how useful could be to integrate with other build tools

we're happy to try something together if you're looking for testers 🙌

[vague lion]

I made a custom GitHub action and am using it in my other OSS projects, it works pretty well. We're actually gonna switch to Dagger soon so I'll take the time to make something custom for Dagger :)

[sturdy shard]

that sounds great. We're around if you run into any blockers and/or need some Dagger extension points to get to a working state in your tool

[vague lion]

That's amazing, thanks a lot! I'll keep you posted.

[sturdy shard]

btw, just checked your game and it looks awesome! I know @simple mural has a really good relationship with the https://cu.bzh/ folks if you're into CI/CD for games 🙂

[simple mural]

I think it will be very hard to make a system for declaring a DAG of dependencies without also providing a system to actually run tasks on the nodes of the DAG. At which point you will also be doing the kitchen sink like Bazel, Nx and others. I would recommend focusing on that problem early.

Otherwise you may have the problem where users don't want to bother duplicating the information from another tool (whether Nx, Bazel or Dagger) into yours, and you end up being out of sync

[vague lion]

I see what you're saying, but every time I read threads on HN from both small startups and bigger companies I see people having Bazel PTSD because of how much set up it requires. I can see why a bigger fully integrated solution makes more sense for big companies, but I was aiming more at projects with 5 to 50 dependencies, not 500 or 5000.

E.g. https://github.com/dorny/paths-filter this tool is specifically for running actions depending on file changes and it seems to be used by a fair amount of people

[vague lion]

Ideally this is just a step in your build pipeline that tells you the names of the dependencies that were touched by your PR and you can decide what to do for each of those on your own. I don't see why people need to clone information into my tool, they just state what their dependencies in a .toml file:

[module]
name = "example_app"

[dependencies]
example_lib = { name = "example_lib" } # Our imaginary app uses the example_lib

If you use Nx or Bazel already it's redundant, but if you're just an agile project without 500 services I do think it adds value. Your opinion is very valuable ofc though. With this in mind, do you still think it's redundant in all cases, or do you think it's worth at least trying?

Another reason I built this is that every time there is an HN thread people talk shit on build tools like Nx and Bazel: https://news.ycombinator.com/item?id=32828584

I just wanted the benefits of the DAG while being able to keep my current very simple pipeline that will probably be good enough for the next 3-4 years

[vague lion]

Thanks! It's quite unfortunately quite old at this point and it's on ice cause gamedev tooling is the worst :(. When I get back to it I'll keep that in mind :)

[frank dome]

that will probably be good enough for the next 3-4 years

That may be true in your case, but if I had a million dollar for every time I heard that, I'd be richer than Jeff Bezos, Bill Gates and Elon Musk combined. 😄

If you have an extremely clear scope which you are sure will not change for the forseeable future, this might be a great idea.

Also, you can learn a lot from building something like this (and from the feedback you receive from users).

But there is a reason why build systems like Bazel ended up being the giants they are and it's easy to start going down that path.

If I were building something like this, I'd make sure to keep it an extremely light wrapper that is easy to move away from to something else.

My 2 cents.

[simple mural]

If the tool could be called in a stateless way without having to write a config file, I could see myself using it - there's definitely the need for a primitive that easily tells you which code change touched which parts of your repo - in a simple to consume and platform-agnostic way. Then you could layer a config file & task specification system on top, although I personally wouldn't use that since I have my own tooling (hint it's dagger :-P) . The layering would broaden your audience - like git has plumbing and porcelain commands, some people want an opinionated standalone tool, others want to embed it in their existing tooling. With a tool like this you could get both audiences.

[vague lion]

That's amazing feedback, thanks guys!

One thing I think I could do would be:

• Start with obligatory config files for unsupported languages
• Slowly add support for new languages (starting with Rust :D ) to infer the information of my config files during build time until no more config files are needed
• Only require dependency config files for things that have absolutely no direct relation with the things they depend on (like an end-2-end test that touches different microservices), Even then this can be added as additional metadata in Cargo.toml or pyproject.toml files when available.

Thoughts?

Also @simple mural I'm quite new to Dagger, can you point me towards the features in Dagger that allow it to do this innately?

I don't want to do the task specification thing btw, just the automatic dependency detection, specifically for cross-language jumps like two microservices that depend on each other where my cargo test doesn't trigger my py pytest.