I have a dockerfile that starts like

[knotty prairie]

Yep we actually do almost exactly that in our CI here: https://github.com/sipsma/dagger/blob/bf489d6dae5f8a884403041aa56ff02c6eed6eec/internal/mage/util/engine.go#L489-L489

The distilled gist being:

dag.Container().From("golang:1.21.5-alpine3.18").
  WithDirectory("/", dag.Container.From("tonistiigi/xx:1.2.1").Rootfs())