Did v0.11.7 introduce breaking changes?

[left cradle]

dagger -m github.com/fluent-ci-templates/trivy-pipeline call \
  fs \
    --src ./ \
    --output-file trivy-report.out \
    --format table \
    --exit-code 7 \
  export \
    --path ./ --allow-parent-dir-path

[sonic trail]

That module has not been updated for some time and relies on a custom Deno SDK that has not been updated for for a couple of months.
https://github.com/fluent-ci-templates/trivy-pipeline/tree/main
I'd reach out to the author via an issue, perhaps, and see if he is continuing to develop on these.
https://github.com/fluent-ci-templates/trivy-pipeline/blob/main/dagger.json#L3-L4
https://github.com/fluentci-io/daggerverse/tree/main/deno-sdk

There are alternatives in the Daggerverse as well.
Could one of them work for you?

[blissful sierra]

@signal vigil @copper veldt flagging the Greenwich team, do we know what's the root cause of this breaking change?

[left cradle]

Could one of them work for you?
I think it would be faster to build ground-up from a bare wolfi *Container. Assuming THAT will still work...

However, you answered a question I didn't ask 🙂

Allow me to rephrase. Is this a blip in v0.11.7 that will soon be reverted and I'll be able to continue using my trivy pipelines as-is, or is this a deliberate change in dagger's code and that module will no longer work (unless updated)?

This raises another question. Since it seems dagger updates can introduce breaking changes ruling daggerverse modules incompatible, how do I tell which daggerverse modules I can use with the most recent dagger version? Or.. which modules require which dagger version?

[blissful sierra]

@left cradle very sorry about this. These are valid and important questions. I don't have the answer. But flagged a few people who will, in the UK timezone so should be online soon.

It's definitely not the intent to break modules already published. Overall backwards compatibility is something we take seriously. We don't promise to break nothing, but breaking conpat should always be the exception not the norm

To answer your question, so far we haven't tracked module/engine version mappings, because we haven't (or hadn't?) broken modules in the way you're seeing yet.

[left cradle]

thank you. I feel safer now 🙂 I mean, there are pieces of soft that announce loud and clear that they will most likely make breaking changes with new versions, e.g. kubernetes. Hence the question about dagger's back-compat,

Thanks for explaining 😉

[signal vigil]

I had a quick skim and I couldn't spot anything obvious in https://github.com/dagger/dagger/compare/v0.11.6...v0.11.7 . Will continue looking.

Anything that looks suspicious to you @tepid sigil @copper veldt ?

[left cradle]

I tried running this (#1250340115050664029 message) in a dind container, in isolation, after installing a fresh dagger binary. Got the same error.

Other (not all; some, not trivy-related) pipelines are failing as well. Didn't notice a common denominator yet.

[trim kiln]

Is the issue related to tini ? I just generate module with deno runtime (dagger init --sdk=github.com/fluentci-io/daggerverse/deno-sdk@main hello-deno) and got this error:

✘ ModuleSource.asModule: Module! 24.7s
! failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Error: failed to generate code: input: moduleSource.withContextDirectory.withName.withSDK.withSourceSubpath.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.GeneratedCode: input: container.from.withExec.withMountedCache.withDirectory.withFile.withMountedDirectory.withWorkdir.withNewFile.withExec.directory resolve: process "/dev/.buildkit_qemu_emulator /tini -- docker-entrypoint.sh sh -c codegen --module . --lang deno --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stderr:
[WARN  tini (7)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
Error: load module config: failed to read local config file: open dagger.json: no such file or directory

I also tried it with 0.11.6 and it's not working as well 😦

[left cradle]

I also tried it with 0.11.6 and it's not working as well 😦
I can't confirm. If I pin v0.11.6 the pipeline passes OK. Ran it just now.

[tepid sigil]

I checked all the commits related to the SDK runtime and nothing is related to such error :/

However, I can also see this interesting error

Error: load module config: failed to read local config file: open dagger.json: no such file or directory

This is curious, but this is using the custom deno SDK so maybe it didn't adapt to a change in the path that happened mainstream?
I fixed this issue in https://github.com/dagger/dagger/commit/433435319c6ce862378fa67308e29f61a99214fb for the TS SDK, I'm not sure it's related but might be

[left cradle]

so maybe it didn't adapt to a change in the path that happened mainstream
would this explain why it works with v0.11.6?
I literally pinned the dagger version to 0.11.6 in all my pipelines and they are all passing now. That's the only change I made

[tepid sigil]

Yeah probably, it could also be https://github.com/dagger/dagger/commit/c819459f28dd465bbfeeffc7c3cee196c0ec0067 since the Deno SDK and your module hasn't been updated for a while

I'm glad it works with 0.11.6 though

[copper veldt]

No this isn't it

This PR doesn't introduce any breaking change except for an error message - that's not in the screenshot, so it's not this

[tepid sigil]

Hmm

I do not see which commit would be related, it's not related to any runtime change, it must be from the core or CLI but we don't have much changes on that in v0.11.7

[copper veldt]

Could it be https://github.com/dagger/dagger/pull/7549

Are you doing anything with the module runtime API? We changed a function signature here

(it's also noted in the patch notes)

[tepid sigil]

this is a very small but technically breaking change to the SDKs-as-modules interface, as noted above, so just FYI. Let me know if there's anyone else I should
Might be it yep

[left cradle]

Attaching as TXT. I figure the image is not very searchable for text 🙂 But it preserves formatting and coloring

[copper veldt]

Oh that's likely it, a module runtime is trying to json unmarshal the file id

[tepid sigil]

unmarshal introspection json: invalid character 'C' looking for beginning of value
Error: unmarshal introspection json: invalid character 'C' looking for beginning of value

[copper veldt]

Do you have a custom module runtime @left cradle?

[tepid sigil]

He uses the deno SDK runtime

It's not part of our repository, it's a custom one https://github.com/fluentci-io/daggerverse/tree/main/deno-sdk

[copper veldt]

Right so that needs a change like in the above

We did intentionally break backwards compatibility here (improved perf/caching/etc) - but I'm not sure how we would have improved signalling, it is right at the top of release notes, under breaking changes.

[signal vigil]

Thank you both for digging into this! I kept thinking about this particular problem and I think that it would make sense to enforce a compatibility check for the module runtimes too, same as we do for the Engine & CLI.

In this case, the Deno Module Runtime would need to explicitly specify which Dagger Engine versions it supports. In this module's case, the max version would have been 0.11.6. A clear version incompatibility will save future debugging sessions like this one.

Does this sound reasonable in principle?

While I could not find an existing issue for this, these would be worth mentioning when creating it:

• https://github.com/dagger/dagger/pull/7031
• https://github.com/dagger/dagger/issues/7594

[copper veldt]

I'm not 100% sure about that - if each module needs to define explicitly which ones it supports, then it means that upgrading dagger versions needs all your deps to upgrade

which means the ecosystem becomes very hard to upgrade

what we really should have done in this case is use the engineVersion here to have the engine determine what API to present

but we didn't, i think that was a mistake

[left cradle]

Just thinking it out loud... separate versioning for dagger (engine/cli) and API/SDK? Following SemVer, SDK users could bind to particular API major versions (or their ranges). If the API changes, it will likely be a breaking change for the API user, so it's easy to fail-fast at boot-time and say "module/runtime doesn't support this dagger API version. Update your module or use --force-allow-unsupported-version to override". This way dagger engine/cli and API can live separate lives, as far as versioning and compatibility are concerned.

Or follow SemVer more closely and only introduce breaking changes with Major version releases. This would make it easier for SDK users to commit to particular API (major) versions.

the ecosystem becomes very hard to upgrade
but if the release contains breaking changes, the ecosystem most likely HAS to upgrade anyway. It's just that with a separate error message and refusal to execute by default, the ecosystem would detect incompatibilities sooner. Especially if the RELEASE notes have a separate section for SDK/API breaking changes.

But then again, I have no idea how these runtimes and engines work with each other, so there's a good chance all of the above is nonsense 😄

@signal vigil

While I could not find an existing issue for this
should I raise an issue for this thread?

[signal vigil]

@copper veldt Not the module, the SDK.

While https://github.com/fluent-ci-templates/trivy-pipeline/blob/8eaf27441a7349e010bb778f63986b966ad6974f/dagger.json doesn't require a specific engineVersion, it uses the Deno SDK @main which brings us to this: https://github.com/fluentci-io/daggerverse/blob/ecfeba3285723b7442386797548aba9488631a6f/deno-sdk/dagger.json engineVersion: v0.9.9

So in this case, we would:

• check the Engine version that the Deno SDK advertises
• if there is a breaking change between this version and the Engine that the client connects to, we fail with a message

So basically we extend https://github.com/dagger/dagger/pull/7031 to SDKs also, not just the CLI / Engine.

If this sounds roughly correct, I would be keen to create the issue so that we have something to track for our releases.

[copper veldt]

right, the sdk is a module though

[signal vigil]

Yes, a special module.

[copper veldt]

i still think the right general approach would be to use engineVersion to determine the correct api to serve to modules
if we can't do that, we can extend 7031, but note, it's not quite the same, since now the engine has a version constraint on the engine version that a module requires

and then it becomes a maximum version constraint as well

[signal vigil]

• When using multiple modules with different versioning requirements, would you serve multiple api versions?
• Would we need to maintain a mapping of which engine version supports which api versions?

There are a few more questions here, but I think that now would be a good time to create a GitHub issue so that we can start capturing this context there.

Do you want to do it @jed or should I?

[copper veldt]

yes to both of the points

we will occasionally need to break backwards compat

i've got some context in my head, i'll write the issue

[signal vigil]

I am not sure about separate versions - it adds a significant maintenance burden on everyone, as we have discovered after separating the Engine/CLI & SDK versions. We used v0.8.0 to align all these versions and that worked fairly well in practice since. Past context:

• https://github.com/dagger/dagger/discussions/3456
• https://github.com/dagger/dagger/discussions/5374

It might be worth adding more context to the issue that @copper veldt will open. Also one to track in terms of progress.