#Is there recommended pattern to test dagger modules?

1 messages · Page 1 of 1 (latest)

teal tusk
#

Hello Team 👋

I am looking for recommded way to write unit test or somehow integration test for module itself?

balmy pivot
#

Sometimes the module dev will create a sub-module called test or dev, and do dagger call -m ./tests test or something like that. The trick is that the sub-module can import the main module, to call its methods. It works pretty well.

The other approach is to just add self-test methods directly in the main module. Works well too, it just kind of pollutes the namespace, kind of like seeing the devtools always on in a browser I guess 🙂

From a technical point of view, both work.

In the future we'll probably have something like dagger test with built-in support from the Dagger SDKs, so that you can just write native test suite in the module source code, and have those tests run in the containerized dagger environment

#

cc @formal rune who has led a lot of these discussions 🙂 He is on the "tests submodule" camp

teal tusk
#

Thanks for quick response @balmy pivot
then, for time being i should create a submodule inside my module and run that sub-module public function(which eventually calls my module function) from dagger cli

balmy pivot
formal rune
#

As always, there is a story behind why I went down this path:

I used to have a single module called "ci" for all my modules. As the number of modules grew, running CI became slower and slower, because all of the modules had to be installed for every CI run.

After splitting up the giant CI module (and running tests for each module in parallel), build time went down significantly.

Lately, I've been writing example modules. Basically most of my existing tests are already e2e tests that demonstrate how my modules work. Why not make them examples out of the box? The nice thing about examples, that they can be executed the same way as my test modules.

Here is such an example: https://github.com/sagikazarmark/daggerverse/blob/main/helm/examples/go/main.go

GitHub
daggerverse/helm/examples/go/main.go at main · sagikazarmark/dagger...

My Dagger modules. Contribute to sagikazarmark/daggerverse development by creating an account on GitHub.

formal rune
#

Couple other thoughts:

  • There seem to be a recent change that breaks my submodule pattern. I have exclude rules to avoid importing unnecessary files during runs, and the way exlcudes work has been recently broken. AFAIIK @pearl totem is working on some changes that may potentially make them work again, but I haven't followed up on that discussion for a while now.
  • The examples feature is nice, but I'm not sure how stable it is (as in: is it going to break?). I started writing examples where I thought they made sense, but I wouldn't invest too much into them just yet. maybe @cerulean ermine has more information
balmy pivot
teal tusk
#

so which approach should I use test as a submodule or as a separate module, as you mentioned it might have startup cost?

formal rune
#

If it’s for a daggerverse repo, I’d go with the submodule path.

The startup cost is not gonna be huge BTW, unless you have a lot of modules.

teal tusk
#

we have internal private daggerverse monorepo in gitlab, so for each module I should add test submodule, is that what you are suggesting?

subtle hornet
#

Hello, I need support to execute a dagger function that interacts with sonarcloud and veracode in the GA pipeline

cerulean ermine
eager swan
# formal rune That could certainly work, but some of my test modules have dependencies that th...

There seem to be a recent change that breaks my submodule pattern. I have exclude rules to avoid importing unnecessary files during runs, and the way exlcudes work has been recently broken. AFAIIK @Erik Sipsma is working on some changes that may potentially make them work again, but I haven't followed up on that discussion for a while now.
Do you have a link on where i can follow that discussion ?
I am currently building a Gitlab hosted private daggerverse repository in my company using the same structure and patterns as your daggerverse (because i find it clear and optimized), but as you said, it's not working well with v0.14.0 or v0.15.0 .

I can pin the dagger engine to v0.13.1 as you do in your flake, but the private git repository support for HTTP/HTTPS refs from v0.14.0 is something we would love to use as well 😅 , i am currently looking for a way to make it work but if there is something comming "soon" it might not worth the time on this depending on that mentioned discussion 😛

formal rune
#

Chances are removing the test module from excludes would solve the problem. But I haven’t tried it yet

eager swan
#

Thank's for the quick answer !

teal tusk
#

@formal rune I am using your test examples as insipration to write mine, when run in parallel one of the test fail randomly, but same works sequentially, my guess, somehow dagger function step over each other, sound like not thread safe somehow

this is failing randomly

func (m *Tests) All(ctx context.Context, token *dagger.Secret) error {
  p := pool.New().WithErrors().WithContext(ctx)
  p.Go(func(ctx context.Context) error { return m.Pass(ctx, token) })
  p.Go(func(ctx context.Context) error { return m.Fail(ctx, token) })
  p.Go(m.IncorrectParameter)
  p.Go(m.InternalError)
  
  return p.Wait()
}

And this works sequentially

if err := m.Pass(ctx, token); err != nil {
  return err
}
if err := m.Fail(ctx, token); err != nil {
  return err
}
if err := m.IncorrectParameter(ctx); err != nil {
  return err
}
if err := m.InternalError(ctx); err != nil {
  return err
}
formal rune
teal tusk
#

there is not such error, but testa are randomly just failing due to assertion, looks like somehow execution from one test was considered by assertion in another test

formal rune
#

I can't see anything in the code above. Without seeing the original code, I can't tell you what's wrong. One thing I'd add: I usually pass in dependencies in a module constructor, not the all function.

teal tusk
#

you mean utlizing construtor in parent module and using all argutments as part of constructor not function?

formal rune
#

no, utilizing a constructor in the test module

#

That way your all function can stay uniform across test modules, in case that's a factor for you

teal tusk
#

i am not sure if that would make any difference

formal rune
#

it's up to you, it's not mandatory

#

but no, it wouldn't make a difference from the error's perspective

teal tusk
#

also, one more interesting thing, i have optional bool flag, that is part of XOpt type when i set this value from test module it does not get reflected while exeucting parent module it's stil using default value but on the other hand other optional parameter were able functioning correctly

also running parent module indepedantly work with optional bool flag correctly, only with submodule bool does not seem to be set somehow

teal tusk
#

I will try to produce a sample code, my guess, I think calling same module async more than once and if exeuction time is variable , somehow dagger session mixing result from one other

teal tusk
#

@formal rune
here is the sample code
parent module

func (m *Module) Sleep(ctx context.Context, token *dagger.Secret, sec int) int {
    if data, err := token.Plaintext(ctx); err != nil || data == "" {
        return -1
    }
    dag.Container().From("alpine:latest").WithEnvVariable("CACHEBUSTER", time.Now().String()).WithExec([]string{"sh", "-c", fmt.Sprintf("sleep %d", sec)}).Sync(ctx)

    return 1
}

test module

func (m *Tests) SleepAll(ctx context.Context) error {
    // parallel run
    /*p := pool.New().WithErrors().WithContext(ctx)
    p.Go(func(ctx context.Context) error { return m.TestSleep1(ctx) })
    p.Go(func(ctx context.Context) error { return m.TestSleep2(ctx) })

    return p.Wait()*/

    // sequential run
    if err := m.TestSleep1(ctx); err != nil {
        return err
    }
    if err := m.TestSleep2(ctx); err != nil {
        return err
    }

    return nil
}

func (m *Tests) TestSleep1(ctx context.Context) error {
    actual, _ := dag.Module().Sleep(ctx, dag.SetSecret("token", "12312"), 0)
    if actual != 1 {
        return fmt.Errorf("❌ TestCase: Sleep1: expected 1 recieved %d", actual)
    }

    return nil
}

func (m *Tests) TestSleep2(ctx context.Context) error {
    actual, _ := dag.Module().Sleep(ctx, dag.SetSecret("token", ""), 1)
    if actual != -1 {
        return fmt.Errorf("❌ TestCase: Sleep2: expected -1 recieved %d", actual)
    }

    return nil
}
  1. this will run sequentially first, everything will work
  2. uncomment parallel run code and comment sequential code, it will fail for one of the test randomly , if you run multiple times

somehow state is being shared may be due to dagger session is same
also if you run this code even sequentially but set sleep sec value to 0 in both tests case, it will fail again due to race condition

i don't know whether its a bug or dagger feature 😄

formal rune
teal tusk
#

Yeah that trick did the job. Seems like within same session secret set via apis is not unique per test function
so its technically, a bug right?

formal rune
#

I guess so

coarse gate
#

Secret names are unique per session, not per test function, AFAICT. And that's by design, not sure that's a bug 🤔
We've had our fair share with this.

Do not use generic keys for your secrets would be my recommendation.

On the testing topic:
We also have a monorepo with ~20 modules that we provide to the rest of engineering. We followed the same approach mentioned above.

Our modules have a submodule, that depends on the parent.

All of the test modules have the same entrypoint, an All() function.
Our CI for the monorepo will discover tests modules and call the All() method on the submodules on every PR

teal tusk
#

Secret names are unique per session, not per test function, AFAICT. And that's by design, not sure that's a bug
I disagree, What if you want to test module for static validation, like empty secret and invalid format or invalid token?

coarse gate
#

That has nothing to do with the name you set your secret to, does it? Maybe I miss understood your case?

You can always pass an empty secret, or an invalid formatted one... as long as you name them differently.

teal tusk
#

what if token key remain same for multiple calls to same function in same session?

#

I expect func to work even if key is same