SDK native test framework support

[alpine lantern]

It's been on the radar to improve support for language native test frameworks like go test. However, I can't find an issue tracking that. It would be a great enhancement/enabler nonetheless.

[winter ocean]

Note that our tests are not for dagger modules, but for the dagger engine itself.

[knotty mica]

Yeah, that seems to make a difference. Adopting your method quickly hit problems

[winter ocean]

And yes, agreed it would be great to have better integration with native go tests

[winter ocean]

Oh yeah that test suite is not generalizable at all 🙂 A lot of dagger-in-dagger is involved

[knotty mica]

I copied this method of writing tests ages ago: https://github.com/sagikazarmark/daggerverse/blob/main/helm/tests/main.go, but from a business perspective Junit reports go a long way

For now just having tests is an improvement over yml pipelines, but we have Junit -> reports for other code repos

[alpine lantern]

And those coverage reports feed into tools like Sonar for further analysis which is also a requirement at the enterprise level.

[knotty mica]

That's precisely the use case here. That and https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifactsreportsjunit. The ability to write tests for Dagger modules that can't be replicated for yaml pipelines is a huge point in Dagger's favour, just need to be able to wire it up to existing expectations

[winter ocean]

@neon seal we were just talking about this 🙂

[neon seal]

I have an idea with a potential GoTestSDK combined with client generator, I'll try to hack on that on my free time, it can be pretty cool

On another side, I implemented my one go test runner so I can directly call it from a dagger function without creating a container (it's super hacky but it works)

Example here: https://github.com/TomChv/dagbench/blob/main/.dagger/integration_test

[knotty mica]

You're not wrong about it being super hacky! I do wonder if there's a different approach: rather than trying to fit Dagger into each language's testing package(s), allow the Dagger module's struct type to be defined as a test module, and expand that into generating junit

E.g. a module called tests has type Tests struct{} currently. If that was instead:

type Tests struct{
  Test: true,
}

The module returns a junit XML containing the result of each function in the test module? Something like that, making this up off the top of my head

[neon seal]

Hey there @knotty mica @alpine lantern

I found a way using generated clients to create native test for dagger Go modules, here's an example: https://github.com/TomChv/dagger-native-test-example/tree/main/go

I added an example with a guide in the go directory, it makes testing so easily, you can simply run go test to test your module and the setup is super minimal.

I plan to do tests to also show example on TypeScript using jest or mocha.

It's not possible in other languages for now because it requires standalone client generation (but now we have one more reason to implement that asap 🚀)

/cc @kind saffron @flat haven @low oar ^^^ it's a pretty cool discovery so I'm pinging you 😉

Solomon mentionned that we could generate that as part of dagger init so users have by default tests setup with testctx etc 😛

[winter ocean]

https://tenor.com/view/oh-yeah-nice-gif-14153667

[low oar]

https://tenor.com/iIotiel3lRJ.gif

[knotty mica]

I'll take a look at this today, thanks!

[neon seal]

Np! Feel free to ask question 😄

I also pushed an example for TypeScript Bun: https://github.com/TomChv/dagger-native-test-example/tree/main/typescript-bun in case you are interested

[knotty mica]

Glanced over this, looks great. I presume I can gitignore the generated client and run dagger client install go ./integration/client in my pipeline here. I'll rework some tests today and see how I get on 🙌

[neon seal]

You can run dagger develop to regenerate the client, or you can commit it if you want

When you do dagger client install, it will update the dagger.json to store the client path & generator so you can regenerate everything from a single dagger develop 😉

[knotty mica]

Bit bumpy on the generated client @neon seal.

Code in test submodule:

account := dag.
        Aws(dagger.AwsOpts{AccessKey: ak, SecretKey: sk, SessionToken: st}).
        AccountInfo()

Code in main_test.go:

res := dag.
        Aws(
            dagger.AwsOpts{
                AccessKey:    accessKey,
                SecretKey:    secretKey,
                SessionToken: sessionToken,
            },
        ).
        AccountInfo()

Result:

# dagger/dagger-aws/integration/client/dag
integration/client/dag/dag.gen.go:59:20: undefined: region
integration/client/dag/dag.gen.go:59:28: too many arguments in call to client.Aws
    have (unknown type, []"dagger/dagger-aws/integration/client".AwsOpts...)
    want (..."dagger/dagger-aws/integration/client".AwsOpts)
FAIL    dagger/dagger-aws/integration [build failed]
FAIL

Region is an optional argument to Aws that defaults to eu-central-1. When using this module in another module as a dependency the function signature is:

func (r *dagger.Client) Aws(accessKey *dagger.Secret, secretKey *dagger.Secret, sessionToken *dagger.Secret, opts ...dagger.AwsOpts) *dagger.Aws

My editor doesn't find the integration package (because there isn't one) and doesn't have autocomplete on dag.Aws or dagger.AwsOpts which makes it trickier

The generated client does appear to be missing region:

func Aws(opts ...dagger.AwsOpts) *dagger.Aws {
    client := initClient()
    return client.Aws(region, opts...)
}

[neon seal]

Hmmm do you have a open source repository that I can look at?
It seems there's a little problem in your configuration or in the setup

Normally there should be an integration package, that you create to hold your client and your test files

[knotty mica]

Not open source no, i can either try and produce a minimal repro or a quick call next week

[neon seal]

A minimal repro is perfect, so I can clone it and see what's going on 😄

Generated clients are still experimental so I'm happy to fix any bug that we may find 😄