Testing modules (part 2)

[unique barn]

Hello! The "dagger-validated" approach to testing modules is still being defined. The most valuable contribution you can make on this topic, is to participate in the design process, which you are doing right now by thinking about it, trying things out in a real world deployment, and sharing your thoughts - so thank you

Back when Dagger was only available as a library (pre-modules), the answer was simple, just use the Dagger client library in your regular test suite. Now with modules, you are effectively extending the Dagger API, which means Dagger itself is the platform. This complicates testing, and also debugging.

My approach is to assume there are 2 different best practices:

1. Current best practices, based on pragmatic trial and error on the current version of Dagger.

2. Future best practices, based on everything we learn this year + future improvements to the platform. These cannot be defined right now.

I will focus on current, since that can be defined and adopted quickly, with the understanding that it may evolve.

My best guess for current best practices, is to follow @cosmic girder 's lead, and have a Dagger module to test the Dagger module. Mark calls it tests, we started calling ours dev.

The unresolved issue for future best practices, which we're trying to resolve right now, is to connect 2 best practices:

1. A dagger module to develop your project (where you define your CI/CD pipelines etc)
2. A dagger module to test your dagger module (what we're talking about here).

To me, those two should be the same. After all, a Dagger module is a software project. Shouldn't we use the same best practices to test all our software, whether they themselves are test pipelines?

It's sort of a "turtles all the way down" situation, and we don't have a definitive answer, for example @lean sky was suggesting that the module itself could define special functions to test itself. Perhaps that will emerge in the future best practices (which require changes to the platform).

Anyway, that's the frontier in the design process right now

We recently started an experiment to combine both patterns in our own repo: instead of a dagger.json at the root of the repo + source directory in ci/, we switched to having a dev/ module at the root. You now need to dagger call -m dev. This brings consistency with eg. sdk/python which is a dagger module, and has its own dev module at sdk/python/dev.

We'll see how it works out.

Paging a few people who I know have opinions on this 🙂 @static wing @quasi heath @pine flax @sand stratus

[fossil lantern]

Thanks for the update and your thoughts Solomon. I will jot down some of my current thoughts here.

I used Mark's way to perform some tests for my modules and they do feel smooth and intuitive but I also felt something is lacking or could be made better. Here are some of my thoughts.

1. It feels awkward to not use the Go testing framework. Which is a big part of the go ecosystem.
2. Creating additional modules and maintaining them is also not hugely fun. Adds a ton of files to my source control. Makes it tougher for reviewers etc.
3. There are a bunch of repetitive tasks that are required to set up a tests module which can be automated with a Justfile/Taskfile/*file but I don't want to use another tool. I want to use dagger for that kind of stuff.

I didn't know the dev folder meant tests within dagger. Thats an interesting convention. I see a few submodules that perform "ci" for dagger but I don't see anything resembling tests. Maybe I am missing it. I would love to know more about the thinking behind the switch from ci to dev for sure.

Finally, being able to easily and comprehensively test dagger code (modules or otherwise) will put Dagger several steps above any other similar tool. It will be a big selling point for sure!

[unique barn]

I didn't know the dev folder meant tests within dagger. Thats an interesting convention.

To be clear it's 2 days old 🙂 It used to be called ci/ (but was a source directory, not a self-contained module, ie. it didn't have its own dagger.json).

It's also not authoritative, the team is tolerating some churn in how we use Dagger ourselves, to help us explore best practices. But that doesn't mean the current way we use Dagger is final, or that everyone on the team agrees with it. It's definitely still work in progress.

It contains all our build, test and release pipelines. For example dagger call -m dev test all will run our e2e integration test suite.

It feels awkward to not use the Go testing framework. Which is a big part of the go ecosystem.

Yes I agree. I would love for us to find a solution to that. I believe @static wing had some thoughts on that topic.

[fossil lantern]

It contains all our build, test and release pipelines. For example dagger call -m dev test all will run our e2e integration test suite
Ah! Are any of those integration tests dagger modules? The are testing the dagger Go code by using go test right?

[unique barn]

yes but to be clear, the software being tested is the dagger engine, CLI etc, not another dagger module (with the exception of the SDKs)

It's just that (IMO) there is a clear parallel between 1) how to use Dagger to test any software, and 2) how to use Dagger to test Dagger modules.

[fossil lantern]

Agreed

[unique barn]

My guess is that we could add a test hook for SDKs. So each SDK could define how to test the module's code (eg. Go SDK would use go test), and the engine would know how to run that in the usual containerized environment. Then we expose that to the user with something like dagger -m FOO test and voila. As a Go dev, you could write regular foo_test.go files, and they would "just work". You wouldn't be able to run go test directly in the dev environment though. But that's the nature of the beast when testing an extension to the Dagger engine IMO.

[cosmic girder]

I’ve given some thoughts to calling it dev, but tests is more obvious to newcomers IMO.

[unique barn]

@cosmic girder if/when modules have a way to self-test (via dagger test and an SDK hook, or equivalent), will you still need a tests module, for other things? Will you feel the need to have a CI/CD pipeline for your CI/CD pipeline? 🙂

[cosmic girder]

I guess not. 🙂

My tests are self contained: they download any dependencies or load test files from the module source

[drowsy yacht]

My guess is that we could add a test hook for SDKs. So each SDK could define how to test the module's code (eg. Go SDK would use go test), and the engine would know how to run that in the usual containerized environment. Then we expose that to the user with something like dagger -m FOO test and voila. As a Go dev, you could write regular foo_test.go files, and they would "just work". You wouldn't be able to run go test directly in the dev environment though. But that's the nature of the beast when testing an extension to the Dagger engine IMO.

One implication I see with this approach is that you now have to test your modules the way Dagger team identifies is the canonical way to test modules? Unless you were allowed to define your own hook in place of the existing one. Like if I am some person (I'm not this person 😉 ) that uses nose to run my all of my tests in Python except for these which must use pytest. I think there's pros and cons to that, for sure. I like golden paths as long as there is an escape hatch

[unique barn]

In this scenario, the escape hatch would be customizing or forking the SDK I guess

(Dagger itself wouldn't have an opinion on how to test - only the SDK would)

[drowsy yacht]

seems reasonable if patching the SDK is a straightforward process. Providing an interface that can override the hook would probably be the most ideal from the UX perspective as a stream of consciousness thought

(I have no idea if such a thing would be viable with the way dagger builds SDK's 🙂 )

[unique barn]

The SDK is just a module, so we have the building blocks to do any of these things, I think

[static wing]

Not fully formed, but I like the idea of keeping language-native testing. Perhaps some helpers in the SDKs to help with testing

[somber gate]

Creating additional modules and maintaining them is also not hugely fun. Adds a ton of files to my source control. Makes it tougher for reviewers etc.

@fossil lantern I am wondering how much of this gets better once we ship the ability to have private modules (which is imminent)

I imagine a big part of the pain for you here is wrangling all the folders you may need to pull down

[lean sky]

https://github.com/dagger/dagger/issues/6724
^ The idea there, that I still like, is to support language-native test frameworks by writing the support as just another module. There's more sugar to be added, but we do support passing modules themselves as arguments to other module functions, which is what should make this possible

[fossil lantern]

While private module support would solve a lot of other issues I don't know if it will make module testing any better. If I'm testing my daggerverse modules I'd already have the right context pulled in the same way I have it set up today I think. I would first pull the daggerverse repo and then run tests on it. While running it by pointing to a repo is nice, it still doesn't solve the creation and management part of the tests.

On a separate note, I think this kind of testing will be more common for platform teams that manage a daggerverse. I am not sure how much testing will go into contextual type repos (eg: CI for a single app). But if Dagger makes it super easy to test, why not?

[somber gate]

Yeah for sure, my question was a bit tangential. Specifically making private modules is cumbersome I’d assume your test modules would also be private.

[fossil lantern]

Right the test modules will be private. Maybe I misunderstood. The DX of creating private modules is similar to public modules IMO. It's accessing them that's tough.

[green sail]

I’ve given some thoughts to calling it dev, but tests is more obvious to newcomers IMO.
I prefer tests since just their name refers exactly to their main purposes, test your module.

Creating additional modules and maintaining them is also not hugely fun
I agree with @fossil lantern . I'm also using the tests/ submodule pattern. It works really well, but I'd prefer to use the native toolchain of the language that I've chosen (mostly Go these days). It also reduces the cognitive load.

In general, I'm more inclined to an approach where just adding something_test.go (vanilla Go tests) and running dagger test -m . just works! (I think it'd be a huge improvement in the Developer Experience); however this approach is mostly for unit testing (checking individual functions exposed in the Dagger module), on the other hand I see that the self-contained tests/ module also provides value by testing the integration that happen when a certain Dagger module composes/uses another (dagger install -m <another-module>). In my opinion, it's an important reliability point since it tests how the module is supposed to be used by another module, and it can spot version incompatibility problems, and configuration issues that trascend the module's code and logic.

[pine flax]

There's one current limitation for testing in the same module. We need to support self-bindings first. Today, when you codegen a module, you only get the dag.... calls for dependencies. You won't see dag.MyModule() there, so something_test.go won't be able to make those calls from a consumer pov. That's why the tests submodule was necessary. But we do intend on supporting self-bindings, for several reasons. When that comes, yes each SDK could implement a Test function to use the language's most popular test suite or something like that, and hook it into dagger test -m ... in a unified way.