#SDK module bootstrapping

1 messages · Page 1 of 1 (latest)

icy steeple
#

Here's how I bootstrapped:

# on zenith-functions HEAD
./hack/dev

# switch to sdk-modules branch (I have it in a git worktree)
cd ./sdk-modules/sdk/go/runtime

# build Go SDK runtime container, bootstrapping from current goruntime
echo '{goSdk{bootstrap{publish(address:"vito/dagger-sdk-go:bootstrapped")}}}' | with-dev dagger query

#
# change core/schema/module.go installRuntime to use vito/dagger-sdk-go:bootstrapped
#

# now start the sdk-modules engine, using above image for Go SDK
./hack/dev

# bootstrap again, this time using the runtime built previously - same digest
echo '{goSdk{bootstrap{publish(address:"vito/dagger-sdk-go:bootstrapped")}}}' | with-dev dagger query
#

It's pretty brittle in that it depends on a specific commit of zenith-functions as the bootstrap starting point, but it works

analog wedge
#

So the idea is to release sdk runtime container images in tandem with CLI releases?

icy steeple
#

Yeah, something like that, I haven't fully thought through the release/versioning implications yet. Initially I thought we would just publish these at the same time as the SDK clients, but now this is a circular dependency, since not only the CLI but the engine needs to know what version "sdk": "go" should resolve to. Unless we pin it in dagger.json and bump it automatically somehow?

#

We could have the SDK just put the full image ref directly into dagger.json for their chosen SDK, which would put the CLI back at the top of the pyramid where it belongs

analog wedge
#

The func (s *moduleSchema) installRuntime() is pretty great 🙂 That's the sort of thing I had in mind to do eventually. I like the canned functions. I'm wondering if we can ship with a default way to build the runtime image, like a stripped down version of a Go module, without codegen (other than the one already shipped with core), but still supporting loading from a registry for configurability.

#

The version part is tricky. The easiest is to tie it to the CLI release so that the CLI knows which version to pull, from its own version. But that prevents innovation in the SDK library itself, in between engine releases. So pinning in dagger.json would allow that, but also open up to version conflicts. I guess that's the preferred approach. Less surprise if there's a warning of incompatibility since the time you made your module to when you upgraded the CLI, and we can make it easy to update that ref.

#

We know we eventually need to handle different sdk versions in modules since they could be incompatible with new API features. We'll need something like MVS.

icy steeple
#

I would have figured we dodge the MVS complexity by virtue of everything being containerized and isolated to separate schemas, but maybe I'm missing something

#

It's true that there's a dependency between the SDK module's runtime and the API it runs against, though

analog wedge
#

I'm thinking changes in API features. The core api will be the same for all modules running in a session.

#

Or even SDK features, especially breaking changes. Like the way you declare your functions or other future entrypoints.

icy steeple
icy steeple
#

Almost there...

icy steeple
#

If anyone's hungry for more API bikeshedding, here's the delta so far:

extend type Module {
  # The container image ref of the module's SDK runtime,
  # e.g. vito/dagger-sdk-go:bootstrapped.
  sdkRuntime: String!

  # Returns the code generated for the module by its runtime,
  # so that dagger mod sync can simply export it locally.
  generatedCode: Directory! 
}

extend type Directory {
  asModule(
    sourceSubpath: String # already existed
    # A pre-built runtime container.
    #
    # Used for bootstrapping SDK runtimes; they'll build their
    # own image, then load the image as a module, and then use
    # that module to build their image instead.
    runtime: ContainerID
  ): Module!
}

Also a good time to bikeshed the SDK runtime module interface. Right now it's this pretty ugly/ad-hoc schema:

type GoSdk {
  # Build the runtime container for the given module code, running codegen along the way.
  moduleRuntime(modRoot: DirectoryID!, subPath: String): Container!

  # Return generated code and files for the module.
  codegen(modRoot: DirectoryID!, subPath: String): Directory!
}

The repetition there feels prime for an intermediate type.

cc @sacred roost

icy steeple
#

Ah right, need to build + push multi-arch images too... Getting bit by that "Publish vs Build-time platform variants" quirk you ran into @sacred roost - now I'll need to call the module multiple times, since telling the SDK module to publish feels pretty wrong

sacred roost