#I'm currently working on implementing

1 messages · Page 1 of 1 (latest)

violet forum
#

So my idea is to have a common and shared description of a user module that all SDKs can share. The moduleTypes function would return this description (like a json file) instead of dagger.Module (and it will also removes the need to return an intermediate container as we do currently)
I'm currently missing what the difference in work would be. Why is "generate a json file" less work than "construct an ID for a module"? I know I'm missing some important detail but on the surface that sounds like the same work, just a different format in the end.

Something that is a bit sad with the json document is it's at the end very close to the graphql query. The thing is as we have to deal with ids, we can't express it in one single query. That makes things more complexe.
If we could have one single query to describe a module, that could be a solution. Maybe that's something doable (a new dedicated query)?
That's the thing, when you construct a Module object and return it, you are returning the ID for the Module. An ID is essentially exactly what you're describing, a whole DAG of interconnected queries

uneven thunder
#

Why is "generate a json file" less work than "construct an ID for a module"?
All SDKs basically analyse the source code first, then construct the ID. Also because the analysis is used for other things than the moduleTypes.
So I'm wondering if we could remove the "analysis to ModuleID" phase, especially because we are duplicating a lot of logic there.
To generate a JSON file (for instance) on the other hand is most of the time just to dump the struct created as the output of the analysis phase. But this struct can also be used for the other tasks, especially the function invocation.

But maybe that's not a good idea 🤷
Especially we already had a version that way, version I removed. And while reading the python code for the moduleTypes, I just have the feeling to duplicate some logic, meaning if the API change we have to replicate the same change across multiple SDKs to keep them aligned.

vagrant scaffold
#

To do that it requires a client and it requires to be executed using engine version set by the user module (to be in the exact same case than when we are performing a call).

I don't understand that part. When the SDK is constructing the module object, why does it need the exact same engine version as the module itself will need, later? In my mind those are 2 different runtime dependencies, for 2 different modules.

uneven thunder
#

I'll try to explain this question of versions (and it's a bit long):

#

Let's take this very simple module:

package main

type MyModule struct{}

type Status string

const (
    Active Status = "here"
)

func (m *MyModule) Status() Status {
    return Active
}

With a dagger engine version (declared in the dagger.json) of v0.18.11 or higher:

$ dagger call -q status

ACTIVE

With a dagger engine version (declared in the dagger.json) of v0.18.10 or lower:

$ dagger call -q status

here

Both commands have been executed with a v0.19.3 dagger version.

#

The key point here is this depends on the way the types are exposed to the engine, not on the way the function invocation is made.

As both cases have been executed using a v0.19.3, the way the types are exposed are the same:

dag.Module().
        WithEnum(
                dag.TypeDef().WithEnum("Status").
                        WithEnumMember("Active", dagger.TypeDefWithEnumMemberOpts{Value: "here"}))

But on the engine side, the behaviour will change:

// simplified version to focus on this question
func (s *moduleSchema) typeDefWithEnumMember(ctx context.Context, def *core.TypeDef, args struct {...}) (*core.TypeDef, error) {
    if !core.Supports(ctx, "v0.18.11") {
        return def.WithEnumValue(args.Name, args.Value, args.Description, sourceMap)
    }
    return def.WithEnumMember(args.Name, args.Value, args.Description, sourceMap)
}

WithEnumValue is the deprecated behaviour:

        dagql.Func("withEnumValue", s.typeDefWithEnumValue).
            Doc(`Adds a static value for an Enum TypeDef, failing if the type is not an enum.`).
            Deprecated("Use `withEnumMember` instead").
            Args(...),

        dagql.Func("withEnumMember", s.typeDefWithEnumMember).
            View(AllVersion).
            Doc(`Adds a static value for an Enum TypeDef, failing if the type is not an enum.`).
            Args(...),

So, what we want here is when withEnumMember will be called, the engine version declared in the user module dagger.json must be used to constraint the engine and chose the right implementation.

#

The trick here is we are not calling a function on the user module, that declares the v0.18.10 engine version, we are calling the SDK that can have a higher version (or the engine version in case of go as it's not a module). So what we want is the code called on the SDK, performing the registration of the types exposed by the module, to be connected to an engine with the visible version the one declared in the user module.

Imagine we are in typescript, or any other SDK than go, using a proper module.

#

When the engine is asking for the types, this is the flow we would like to have:

  • moduleSource.runModuleDefInSDK
    • sdk.AsModuleTypes().ModuleTypes() 
      • var module dagql.ObjectResult[*core.Module]
err = dag.Select(ctx, sdk.mod.sdk, &module,
    dagql.Selector{
        Field: "moduleTypes",
        Args: []dagql.NamedInput{
            {
                Name:  "modSource",
                Value: dagql.NewID[*core.ModuleSource](source.ID()),
            },
            {
                Name:  "introspectionJson",
                Value: dagql.NewID[*core.File](schemaJSONFile.ID()),
            }
        },
    },
)
    • inside moduleTypes, the SDK will call its own tool to perform the type analysis and type registration, returning a dagger.Module object.
    • usually this is done inside a container related to the SDK language, for instance inside a python container in python.
    • so at some point there's a withExec inside this container, executed the final code containing a dag client to do the dag.Module().WithEnum(...)
#

The thing here is we are performing this call from the engine, using the engine version declared by the SDK. What we would like is a way to say "call moduleTypes but with this view with this engine version". As far as I know, this is only possible on a withExec. But to call the field moduleTypes is not a withExec. So we are transforming that into a withExec:

  • the moduleTypes instead of executing the withExec will return a dagger.Container with the command behind the withExec declared as the entrypoint
  • the engine will withExec an empty command, using the entrypoint
  • this allows to set some metadata so that the call is seen as coming from the module itself, declaring a specific engine version:
  • // get the container with the entrypoint
  var ctr dagql.ObjectResult[*core.Container]
    err = dag.Select(ctx, sdk.mod.sdk, &ctr,
        dagql.Selector{
            Field: "moduleTypes",
            Args: []dagql.NamedInput{
                {
                    Name:  "modSource",
                    Value: dagql.NewID[*core.ModuleSource](source.ID()),
                },
                {
                    Name:  "introspectionJson",
                    Value: dagql.NewID[*core.File](schemaJSONFile.ID()),
                },
                {
                    Name:  "outputFilePath",
                    Value: dagql.NewString(moduleIDPath),
                },
            },
        },
    )
// call the entrypoint with execMD
  var modDefsID string
    ignoreCtx := dagql.WithSkip(ctx) // ignore some spans as they are internal trick only
    err = dag.Select(ignoreCtx, ctr, &modDefsID,
        dagql.Selector{
            Field: "withExec",
            Args: []dagql.NamedInput{
                {Name: "args", Value: dagql.ArrayInput[dagql.String]{}},
                {Name: "useEntrypoint", Value: dagql.NewBoolean(true)},
                {Name: "experimentalPrivilegedNesting", Value: dagql.NewBoolean(true)},
                {Name: "execMD", Value: dagql.NewSerializedString(&execMD)},
            },
        },
        // ... get the result that has been written on a file
    )
  • execMD is defined that way:
        execMD := buildkit.ExecutionMetadata{
        ClientID: identity.NewID(),
        CallID:   dagql.CurrentID(ctx),
        ExecID:   identity.NewID(),
        Internal: true,
    }
    execMD.EncodedModuleID, err = currentModuleID.Encode()
  • the EncodedModuleID contains the engineVersion as declared in the dagger.json inside the user module.
#

The Go SDK is taking some shortcuts as it's not a module, it's injecting the right execMD directly at the level of the WithExec because we are using a dag.Select that allows us to do it on withExec.

#

to try to summarize:

  • we are in the process of creating the module from its module source
  • during the registration phase of the types, the SDK will call dagger APIs to create the types (dag.Module().WithEnum(...))
    • the engine needs to react based on the version as defined inside the module we are creating, not based on the version of the SDK, because the enum behaviour is set at the registration
  • this is done by injecting some metadata during a withExec (the thing about container and entrypoint)
#

One solution I could imagine that would make things simpler, is the ability to specify the engineVersion when we dagger.Connect(). That way, we could do something like dag = dagger.Connect(versionFromUserModuleDaggerJson) inside the code that is creating the types (then dag.Module().WithEnum(...)). This could simplify and make clearer the SDK API, remove some hacks with entrypoint.

vagrant scaffold
#

Whether it's withEnumMember or withEnumValue that is called should not depend on what engine version the module requires. Because the module code does not actually call that function. It should depend on the SDK's required engine version.

So the SDK should not worry about the module's required engine version when registering types, it should get an engine view based on its own code.