Switch checks to `@check` pragma, improv...

[dull orchid]

This is amazing 🙂 Loving it so far!

I'm trying to wrap my head around a few use cases we have for our tests.
We have modules with a test submodule that can benefit immensely from the //+checks pragma.
Some of these modules have a requirement for an input parameter in the constructor for the tests to run.

Is it possible to pass an input to the constructor when retrieving a module through source?

mod := d.Src.AsModule(dagger.DirectoryAsModuleOpts{
    SourceRootPath: path,
})
cg := mod.Checks().Run()

Or call the Checks() API from a module implementation:

dagger install my-modulet/tests
...
dag.MyModule(param).Checks()

[dull orchid]

Oh, the message from a check report is not yet implemented ... 😄
I was so excited 😛

[narrow geode]

what do you mean by "not yet implemented"?
I think that's the goal to have this kind of output.
If the check is passing, then we have a very minimal UI, and if it's failing we can display the errors like in other commands.
If you want a full detailed output maybe in this case it's best to run as a classical task, with dagger call for instance.

[dull orchid]

If i get the report output when calling checks through the API (not the CLI), i get a nice markdown table, with a "missleading" column called "message"

Which is empty and no code to populate from anything. I was just playing with this feature and found this

I was inspired by some of the PRs to try to refactor our monolithing approach to testing. It may be that checks are not suited for what we're doing as a platform team (module testing through submodules) - I was trying to avoid reimplementing some of this existing functionality 😛

[jagged vessel]

Hey !
I am really interested in the check feature.
But where is the info in the documentation about how withCheck is working ?

[narrow geode]

yes, true 🫣
I'm not sure what this field is about, wondering what you would like to have inside a message?

[dull orchid]

the error returned from the check was my first thought... But that could become a large blob soon enough

[narrow geode]

withCheck is simply flagging an exposed function as a check function. It's something only the SDK implementation really care about.
At the dagger module level, what really matters is the check pragma, like +check or @check as you can see in https://github.com/dagger/dagger/pull/11374

[narrow geode]

yeah, the thing is the error or the output can be difficult to read. Especially what's running is a dagger function, so it can be a lot of different steps

[dull orchid]

Probably best to remove that message from the table 🙂 And rely on the output of the check run so there's no confusion

[languid sierra]

@dull orchid that message field is a remnant of the initial API where we intended to return the error, like you assumed. But that turned out to be not useful- the real useful information is in the logs. So we removed the message field and rely on telemetry instead. We will enventually wrap telemetry collection in a convenient function like Check.logs(): File but didn't get around to it yet.

[dull orchid]

Tx for the explanation. I got excited trying the checks pragma and ended up refactoring our ci approach.
I couldn't use the Checks through, as they seem to be more for a simpler use case (rather than running a test suite against modules)?
Annoyingly, I need an env var passed in many of my test functions...
We implemented a similar approach with an interface so we can call checks without knowing which ones need to be called, but looking at the devex of this functionality I wish I could use it 🙂

is there any way this would eventually be exposed through the module API?

dag.MyModule().Checks()

[languid sierra]

@dull orchid for inspiration, we're switching our entire CI to checks + toolchains 🙂 So very much not intended for only simple use cases

--> https://github.com/dagger/dagger/pull/11373

[languid sierra]

You can take advantage of user defaults. If you set some arguments with a local .env, those will be taken into account for checks.

[dull orchid]

I tried briefly when calling through the sdk/ not CLI. I’ll revisit it. Because that would be amazing …

I’ll check your refactor too. Tx!!

[languid sierra]

FYI @dull orchid we're also implementing automatic scale-out of checks. So you can naively run dagger checks against one engine, and that engine will automatically scale out some of the checks to other engines in the cluster

That will allow us to ditch Github Actions completely for our CI (and also kubernetes)

[dull orchid]

True, dagger checks picks up the .env file!!! (never doubted you :P)

Not picking the default values if I call the checks through the API 🤔

func (m *Modules) Test(ctx context.Context, path string) (string, error) {
  mod := m.Src.AsModule(dagger.DirectoryAsModuleOpts{SourceRootPath: path + "/tests"})
  cg := mod.Checks().Run()
  return cg.Report().Contents(ctx)
}

That seems to not pick up the .env defaults

I may have to go deeper into the refactor and see if I can skip altogether the api, and call checks through the CLI directly 🙂

[languid sierra]

When calling through the API, are you sure your client has access to that .env?

[dull orchid]

I would think so. I can list the contents of the path from the API and see the .env file, and when calling the function i can see
user default: debian-tests(token="env:FORGE_TOKEN")

But the checks fail with the missing required argument

[languid sierra]

The most likely suspect IMO is your client's workdir. Are you sure the process serving the API is in a workdir covered by that .env?

to be sure you can run host().findUp(".env") and see what it returns

[dull orchid]

Thanks for the insights 🙂
I couldn't make it to work through the API so i ended up moving to the CLI with the checks command.
I still liked the idea of having the markdown report in the summary, but I can live without for now
The .env file works as long as it's a toolchain installed and the .env lives in the outer directory

I may revisit it in the future, but we're in a much better place now 🙂

also, what did I miss with:

implementing automatic scale-out of checks. So you can naively run dagger checks against one engine, and that engine will automatically scale out some of the checks to other engines in the cluster
other engines in the cluster

??? Are engines aware of each other ??? is this your dagger cloud engines? Or I have missed a serious feature hidden somewhere? 😄

[languid sierra]

it's still wip, not released yet. I would love to show you guys a demo when it's ready 😁 Perhaps while I'm in France week of Dec 14?

[dull orchid]

Sure thing. Always happy to hop on a call and be amazed