Native toolchain testing with generated clients

[agile flower]

Thanks Tom! Is this the official way to do it? I know it's not documentd but I'm afraid to write code if things are going to change soon. Will this be promoted as the way to test Dagger modules?

[jolly heath]

It's not official but I think it should be a pretty standard way to test a module code.
Maybe @tranquil hemlock has some opinions on that topic too?

[jolly heath]

The current work we are doing on the client generation is simply to make the generated file as native as possible, the test pattern shared in that repo shouldn't be impacted by the future changes.

[agile flower]

Cool, let me test that pattern and provide feedback 🙂

[jolly heath]

That would be amazing!

Note that we have a PR open to improve the integration with existing go project, maybe your setup will require this PR to work: https://github.com/dagger/dagger/pull/11768

[agile flower]

Gotcha, yes my modules are all Go

[jolly heath]

Normally it should work for your case if it's just dagger go modules, but if you find any issue please ping me 🙂

[fluid linden]

@agile flower I'd be interested in your experiences with this. I tried it late one afternoon, got errors, and didn't go back to it

[agile flower]

I saw your messages and got discouraged 😄 So didn't test myself haha

[jolly heath]

Did you post a message on discord? Sorry if I missed it ^^

[fluid linden]

We're still using a test submodule, which loses the junit output but gains the same portability of Dagger modules - it works identically locally and in Gitlab

[fluid linden]

I don't recall, it's not a priority for us currently, junit reports are just a nice-to-have. We run a test submodule before a new version is published, that suffices for now

[jolly heath]

Alright!

[agile flower]

This is a good point. I remember early on, there was a design which let you write the tests in Go but run them via dagger. Like a dagger call tests

[agile flower]

Hey @jolly heath , I spent a little time testing this. The good, it runs! I was able to go test -v ./... and get it to run.

However, I was not able to get coverage with go test -coverprofile cover.out -v ./.... It seems to be looking for coverage of the dagger.gen.go and not my actual code. How do I get a coverage report out? The coverage report is the most important thing I need as that's a measurement we record and share.

2. Since this is basically integration testing, how do I pass credentials to a test? For example, I want to test my git module. I'd have to set a PAT somewhere. How do I do that?

[jolly heath]

Hey,

Thanks for your feedbacks!

1. The generated client use the engine to query your current module, so we would need a way to record the coverage of the code executes inside the engine. I’m not sure how doable that is.

2. You can use an environnement variable, your tests are executed locally so you can access your env.
   Or if your PAT is stored on a secret manager, you can install a dependency like 1Password to get that secret and use it in your tests.

/cc @tranquil hemlock @pseudo granite

[agile flower]

Thank you Tom! If I can't generate coverage, I don't know if there's much benefit to using the native tooling in that case. I'd rather stick to creating a test module because that's running in Dagger ecosystem. If I run the native tooling this way, it feels disconnected from actual Dagger as I need the go tooling outside of dagger. If I can create coverage reports though, that would probably tip the scales.

Even with env vars, it feels strange to manage them in a different way just for tests when dagger does it natively. I wonder if dagger could help mock some of the APIs for testing.

It would be super cool to have the ability to use native tools though! When I introduce it to people the first thing they realize is that it's real code so we can test it

Today we can even write unit tests for Jenkins groovy and generate coverage. So they are going to question why that can't be done with Dagger

[fluid linden]

The submodule for tests is too convenient to give up. As a first step, adding a @test decorator and reporting the results in junit would be great, and isn't a huge task. Coverage is a larger task.

[jolly heath]

Yeah that’s definitely something we need to figure out!
Since the tests successfully include the module’s traces, there might be a way to also link some coverage here

[jolly heath]

Why is the submodule more convenient than having your tests running using the native tool chain with a generated client to call your module?

/cc @tranquil hemlock I think that discussion is very interesting, maybe you have already some idea regarding the coverage

[fluid linden]

Not everyone has Go installed, but all the engineers writing/using Dagger have that. We don't have a standard way of running Go anything, but we do have a Gitlab component to run Dagger.

I'm not against a native method, so long as it's as simple as dagger -m tests call all 😅

To the earlier point, we provide any inputs to tests with a .env with the usual MODULE_METHOD_PARAMETER keys, I don't want to have to provide test inputs in a different way either

[jolly heath]

That makes sense! So a more native approach may not be the right solution for you

[fluid linden]

I'd happily switch to a more native approach if it was a bit easier than it is now. I tried the current suggestion and hit errors about Dagger clients very quickly, which isn't going to pass the smell test over dagger -m tests call all which, thanks to it being a Dagger module, just works