#Caching, Evaluation Context, Performance

1 messages · Page 1 of 1 (latest)

verbal blaze
#

Hey there,
I'm trying to evaluate if Dagger could be the CI tool for us for the next >5 years. For that, I'm trying to plan out what I would need to build to accommodate our devs.
The journey so far has spawned more and more questions, and I would be happy if I could get them answered before I start building.
Generally, for all my questions, the crux is about "is it possible?". The effort required doesn't matter.

  • From within a module, is it possible to extend the context that is pulled in from the host?
    • I don't want to build a custom application, but directories that need to be ignored need to be dynamic
    • It is okay if there is one expensive operation dagger call -m my-ci init per directory that produces some artifact that I check into vcs, would be nice if it could be avoided though
    • The directories to ignore contain dozens of GBs that I don't want to pull in
  • Are there benchmarks on eval time for huge DAGs?
    • I plan to dynamically build DAGs that will create hundreds of thousands of nodes
    • A vast majority will end up receiving inputs that will get invalidated yearly, some even never
    • Devs want subsecond evaluation to iterate quickly
  • How could remote caches be made possible?
    • CI needs ephemeral nodes and needs to make use of the aggressive caching
      • Is there some flag to ignore all existing caches for an invocation?
      • Is there some way to ignore a function cache when invoking it from another function?
      • Does the GC evict based on last access?
      • Can the TTL of 7 days be increased?
    • Devs need to be able to download existing artifacts, rebuilding would take them out for half a day
    • Devs still need to use their local cache primarily
  • Can one pipeline be distributed across multiple nodes?
  • Tests need to connect to qemu. Can we make qemu/kvm available without walking each dev through a custom dagger setup?
  • Is the concept of test cases planned? I assume Dagger Cloud will want to display test statistics one day
silk bronze
verbal blaze
#

Ofc! I didn't expect to get everything answered immediately, and some stuff might require more context anyway 🙂
Thanks!

silk bronze
#

First if you can add some details on your stack it would be very useful. I understand if you can't share everything, but even general information like - monolith or not; current CI; language & toolchains; main problems for the current stack; etc

verbal blaze
#

The stack is a complicated mix of Go, Rust, C, TS, posix shell and init scripts. Tests run on a custom test runtime that additionally uses qemu or connects to remote machines (hardware tests)
Monolith or not is not that easy to answer, in the end it's a monolith split up across many separate processes, it's mostly a monorepo, mostly
current CI is a mix of makefiles, shell scripts, go scripts, rust scripts, docker scripts, and gitlab-ci. this is also the reason I'm looking for a replacement

Most devs are unhappy with current iteration times. Most of that is due to caching. The important part for CI turnaround currently takes 1.5h, it already uses caching. With great caching this could be reduced to <10min
For local iteration, turnaround times are in the ballpark of 15min, and aggressive caching and a proper DAG could reduce this to 1-2min
I ask about subsecond eval time because I want to replace the homegrown solutions that every dev built for themselves over the years, and that is a requirement for adoption

I've tried to find every CI tool out there to compare them and the shortlist was comprised of Nix and Buck2. Nix is out because with a sufficiently large DAG, evaluation time is measured in minutes. I define evaluation time as the time it takes to figure out what needs to be done. Buck2 ticks all boxes, but if I build a big Starlark stack now, then someone needs to maintain that, and no one knows Starlark. Additionally, there are more rough edges that I would like to avoid.
Dagger was only on my mind because of the approachable telemetry. After enough frustration with the other tools, I started to look seriously into Dagger at the start of this week.

#

In regards to possibly tens of thousands of DAG nodes: To achieve the caching goals, every dependency needs to be cached separately, from source to build artifact. There are two dozen active branches around at all times, it needs to be granular for this to work out without busting caching limits. The hundreds of thousands of nodes would be that each test case would get a separate node. That's optional and likely to change, but it will be hard to model dependencies otherwise. I'm most open about this part, as I've seen that there is the concept of dagger checks, although I don't understand yet what exactly it solves. It sounded to me a bit like Earthly target wildcards

verbal blaze
#

small bump, I would still be very happy about any questions that could already be answered 🙂

silk bronze
#

Re: producing artifacts to check into vcs: that is a common pattern and dagger has facilities for it. Also we're adding sugar for it dagger generate with interactive diff apply prompting (optional)

#

Dozens of GB I don't want to upload

Very common and we have robust filtering to handle that. Just not dynamic (yet) but each function argument can be filtered individually which gives you lots of flexibility in practice

verbal blaze
#

Thanks for getting back to me!
Is there some way to discover the wip features that are not so well documented yet? I didn't stumble upon dagger generate or dagger checks yet and only learned about them from keeping up a bit in this help forum

For the other questions: is there any more info that you need from me?