#How to make optimal build files

1 messages · Page 1 of 1 (latest)

nimble walrus
#

Having this build file (some stuff redacted)


  containerBackend(): Container {
    return dag
      .container()
      .from("sbtscala/scala-sbt")
      .withDirectory("/", src)
      .withWorkdir("/");
  }
  async buildBackend_(backendJar: File, arch: string): Promise<Container> {
    const backendNative = await dag
      .container({ platform: arch as any })
      .from("native-image-community:23")
      .withWorkdir("/")
      .withFile("/a.jar", backendJar)
      .withExec(["native-image",".."])
      .file("/app");

    return dag
      .container({ platform: arch as any })
      .from("scratch")
      .withWorkdir("/")
      .withFile("/app", backendNative)
      .withExec(["chmod", "+x", "/app"])
      .withEntrypoint(["/app"]);
  }

  @func()
  async buildBackend(): Promise<string> {
      const platforms = ["linux/amd64", "linux/arm64"];
      const backendJar = await this.containerBackend()
        .withExec(["sbt", "assembly"])
        .file("backend.jar");

      const containers = await Promise.all(platforms.map(x => this.buildBackend_(backendJar, x)));

      return dag.container().publish("container", { platformVariants: containers });
  }

To make this build optimal in terms of building I would to construct a DAG which builds

  1. containerBackend once
  2. pass on the jar to two parallel jobs which use the graal native image and return the containers
  3. use the returned containers to publish it

This is a typical fan out -> fan in thing. However:

  • Since buildBackend_ returns a Promise<Container> it builds them in sequence?
  • I guess it's hard to constrain/determine that some parts always yield the same result, if that would be the case it would be easier to construct a DAG and optimize and create an optimal build plan.. is there any article how this optimally or recommendations ?
velvet python
# nimble walrus Having this build file (some stuff redacted) ```typescript containerBackend(...

Since buildBackend_ returns a Promise<Container> it builds them in sequence?

no, the promise.all will build each architecture in parallel.

your pipeline seems correct to me from a high level point of you. The only thing I'd add if you haven't done so is the proper cache volumes in the sbt assembly step and the native-image part so when re-running on subsequent builds, things are faster

#

is there anything in particular which you think it's not being currently optimized?