#How can a module test and deploy itself?

1 messages · Page 1 of 1 (latest)

viral wraith
#

🧵

#

How can a module test itself?

#

How can a module test and deploy itself?

#

Inviting @stoic summit @nimble dust @sour verge and anyone else interested in this topic 🙂

potent ermine
#

Oh yeah, high level outline of what I was imagining:

You should be able to use existing test frameworks for your language. That's more familiar and saves us from requiring each SDK implement a test framework (which is way beyond something we have the bandwidth for, even in the long-term IMO).

Using Go as an example, if I had module code in main.go:

func (m MyMod) Foo(s string) error {
  if s != "foo" {
    return errors.New("not foo")
  }
  return nil
}

I should be able to create main_test.go:

func TestFoo(t *testing.T) {
  // you can just use `dag` here the same way you do in module code (requires Self bindings support, but we want that anyways)
  require.NoError(t, dag.Foo(ctx))
}

Then to run tests you could just execute dagger test.

The sort of grand idea here is that SDKs don't implement their own test framework, they just wrap other existing ones. dagger test in this case would see your module is using the Go SDK and the Go SDK would wrap go test to actually execute your tests.

So the Go SDK and all other SDKs that support testing would need to have a new Function (in addition to the existing ones like Codegen and Runtime) like:

func Test(ctx context.Context, modSource *ModuleSource) {
   // invoke "go test" on the provided module source
}

I imagine each language could have a "default"/"preferred" test framework. For Go it's extremely obvious since "go test" is essentially standard.

However, other languages are less standardized obviously and have a zillion different test frameworks out there.

So the question then is how to support other test frameworks so users can choose which one they want. E.g. Python has pytest, behave, etc.

This is still an open question, plenty of options, but I was imagining that this could itself be a module. So you could imagine that the signature for Test above could instead be:

func Test(ctx context.Context, modSource *moduleSource, framework TestFramework) {
   // TestFramework is an interface that has functions wrapping some external test framework, can just provide the relevant directories from modSource to it
}

Then, with a little CLI sugar you could imagine supporting something like:

// use pytest
dagger test --framework github.com/foo/daggerverse/pytest
// use behave
dagger test --framework github.com/foo/daggerverse/behave

Where github.com/foo/daggerverse/pytest and github.com/foo/daggerverse/behave are modules that implement the TestFramework interface

#

The end result is:

  • Users get to use test frameworks they already know, SDK developers don't have to implement anything from scratch
  • Test framework support can be implemented by modules that just wrap existing external test frameworks
  • To run tests users just use this new dagger test command
#

As a side note: it might be cool if dagger test wasn't strictly needed and you could just do e.g. go test directly.

  • This is technically possible, though a kind of large effort (talked to @livid prism about this a few weeks ago)
  • I would not want this to be the only way because it would impose requirements that users have certain host dependencies pre-installed. So I think dagger test would be the place to start and we could consider supporting this in the future
viral wraith
#

Wouldn't this break regular go test?

potent ermine
#

You can't use regular go test on module source code anyways, so not really breaking anything

viral wraith
#

Ah. right 🙂

#

If we later invested the work to support go test (like you said, possible but lots of work), would it require going back and breaking the module layout, or would it be a drop-in improvement from the user's POV?

potent ermine
# viral wraith If we later invested the work to support `go test` (like you said, possible but ...

Hypothetical at this point, but I could imagine it would be back-compat. The idea is that we'd need some magic to trigger different behavior depending on whether the code is running direct on the host or in a module container.

The hard part is that when it detects it's not in a module container, I think it would need to essentially just load itself as a module in the engine and then run there. Otherwise we'd break sandboxing and you'd get dramatically different behavior.

viral wraith
#

OK this dagger test idea looks pretty sweet

#

What about other "meta-functions"? For example I have this CI config generator. I don't know where to put it

#

It has the shape of a codegen use case (certainly it has the same UX problems as all codegen-on-dagger use cases, for example the awkward -o PATH part)

#

but not sure how much it generalizes to codegen. It might be more of a unique snowflake

#

At the moment I tried daggerizing .github 😛

#

My workflow:

  • Edit .github/main.go (pretty fun, like my github config has source code)
  • Call dagger -m .github generate -o .github . This writes files to .github/workflows/. Insert codegen UX problems here (can't remove files, etc)
#

Where would that go?

potent ermine
#

I don't have any grand unified theories here, but my impression is that this sounds like just a DX problem with file export from modules.

There's plenty of use cases for exporting files that aren't in this "meta"/"codegen-esque" category which would also have the same issues of A) command is too verbose and B) inability to delete files.

I'd want to see if we can just squash those DX problems for all use cases of exporting files/dirs. If we did that, then I don't see why your use case shouldn't just be a plain old function.

#

I'm not fundamentally opposed to a more grand idea here, I'm just not sure yet if it's necessary

viral wraith
#

yeah it might be 2 distinct problems

#

one is codegen

#

the other is the meta aspect. I don't want the function to generate ci for my module, to show up at the top level of my module api

#

that second problem is not codegen-specific. It's not test-specific either, so I'm in search of a solution

#

it's different from "codegen is cumbersome in my daggerized project" because there's the specific additional parameter "...and my project is a module"

nimble dust
#

👍 to that approach in general though, handing out to go test seems neat

#

for other meta stuff, go build tags spring to mind - that's kind of how _test.go files work in go, i could totally imagine a _example.go-type file as well

#

but not sure how well that approach extends to other languages

viral wraith
#

@nimble dust any ideas for my "generate ci config" use csse? 🙏

nimble dust
#

honestly 😛 i might just have it live in .github/.dagger or something fun like that

or do you want a module to generate it's own ci?

viral wraith
viral wraith
#

maybe could be dagger -m .github generate?

Is there a possible dagger generate that would be equivalent to dagger test described above?

nimble dust
#

could we not do dagger call -m .github call generate -o .? unless you want that as a shorthand?
i'm sure about having special entrypoints, but i'm generally kinda unopinionated

#

though if we're having special verbs like generate and test at the top level, it is a bit weird to have core and maybe in the future core types like container there as well

viral wraith
#

That's what I do today and it's very brittle.

#

First it's verbose. And second, that export path has to change any time my workdir changes

#

But at least, if we feel good about making .github a sub-module (ie we think it's a clean solution rather than a hack) it solves half of my problem. The other half is a straightforward "how to daggerize codegen pipelines" problem.

nimble dust
#

i think having generate as an alias for call ... -o same/level/as/dagger.json makes sense - but i don't neccessarily think this is the same as test, which would want to interact with the sdk code in a fundamentally different way, not like call today

stoic summit
#

mayhaps we could consolidate dagger develop into dagger generate and allow you to extend it? had a similar thought in the past around generate being worth promoting into a special feature, and we kind of already have it

#

low conviction on that idea, mostly pattern matching to see if a good idea falls out

#

maybe it's dagger generate [path] and there's some way to map that to a generator(s)

viral wraith
nimble dust
viral wraith
#

@stoic summit that definitely tickles something in my brain. I've been thinking that a path-oriented API might be useful, kind of like the "routes" model in web frameworks, but for directories in your project. It's been a very vague idea so far. But it came out of earlier codegen issues also (specifically how entangled our own repo is with dagger module codegen)

potent ermine
#

Yeah I do like the idea of extending dagger develop for this (alongside possibly renaming to dagger generate or smoething). The unified idea here being:

  1. dagger test -> Test function in SDK that accepts optional interface arg that implements custom test framework
  2. dagger develop/generate -> Codegen function in SDK that accepts optional interface arg that implements custom codegen implementation (or other args like path, etc.)

Or something like that. Basically just make SDK functions invokable from the CLI and extendable via interfaces when needed.

potent ermine
nimble dust
viral wraith
#

An important note on dagger test. If I run dagger test in my daggerized project, I expect that to run the project's test in dagger. If it only runs the "self-tests" of the module itself, that's weird

potent ermine
viral wraith
#

But then what happens to the existing dagger call test?

#

In other words all the "regular" dagger functions that have to do with testing your project. Do they still show up in dagger functions, are they still introspectable the usual way, or do they kind of "disappear" from the schema?

potent ermine
#

That's a good question. Thinking out loud: the TestFramework interface I mentioned above could have a function that returns the list of tests (I think that's pretty much a table-stakes feature for test frameworks), which would allow the tests to become Functions in the actual schema too. Then we could also more easily support running particular tests, etc. too. Would probably also make integration with telemetry+traces easier?

I'd need to think more about exactly how it'd fit into the schema. Maybe this gets back to past conversations about how dagger call and Functions are primitives that we progressively build abstractions on top of (dagger test being a first one). Sort of like the unix "everything is a file" model, but for us "everything is a function".

#

Need to think about it more

viral wraith
#

Yeah there's something there for sure

#

I want to tie that to my proposal of a Check interface, which is a superset of tests in my mind. Anything green/red. Maps to Github checks. So possibly that would lead to a dagger check command instead.

#

Then we would need some way for:

  1. The module to expose one or more checks. For example in our repo we have engine test all, engine test important, engine lint, sdk python test, sdk python lint, etc. etc. So how do you aggregate all of that into a unique dagger check (same question if it's dagger test, just being optimistic and hoping you'll like the check idea 🙂

  2. The SDK to add more checks, via native tooling integration, ie. test_xxx.go, but why not also add builtin support for .golangci.yaml etc, in the case of a broader check interface

  3. The resulting schema to be exposed to callers