Are dagger directory IDs persistable?

[ashen dawn]

If we want to ensure we are re-using a previously created artifact, can the IDs be saved and referenced in a later run of dagger assuming the buildkit storage still exists?

    var savedValueFromPreviousRun dagger.DirectoryID;
    savedValueFromPreviousRun = "value loaded from saved state"
    result := client.Container().Build(savedValueFromPreviousRun)

This sort of use case would be for ensuring the same artifact that was built and then tested is used for a release. We wouldn't want to allow a cache miss here and rebuild from the start as we wouldn't be able to guarantee the tests were ran on what is being released or the same artifact is being deployed across different environments (eg. dev/qa/...).

An alternative is to use containers Published externally which makes sense for containerized code but some use cases (eg. terraform code) wouldn't normally be packaged into a container (at least how I've done it).

[solid crown]

yes, the ID is just a base64 representation of the operations and metadata buildkit generates. If you get the and base64 decode it, you can see all this metadata

if you're building and artifact I'd consider returning a sha256 SUM of the artifact though instead of relying in Dagger's ID?

Take into account that ID's will very likely become implementation detail in the near future (https://github.com/dagger/dagger/issues/3558). cc @crisp nebula

[ashen dawn]

is the sha256 value accessible, didn't see that from go sdk?

and how would I resume using the sha256 value later?

[solid crown]

oh! you're looking forward to perform some sort of resumption later. I just thought that you wanted the id mostly as a verification mechanism that subsequent builds are still the same. In that case yes, I'd assume you have to use the ID. Erik has more understanding about this and he'll provide more accurate input

[ashen dawn]

yeah, in our release process would need to pause for manual approvals in many cases

seems like adding the relevant directory to a scratch container and publishing it probably cleaner though and saving the full container ref

[crisp nebula]

No, I don't recommend storing the IDs and using them directly later. In addition to what Marcos mentioned and some other changes we might make in the future, the fundamental problem is that submitting the same ID won't guarantee a cache hit. Even if you are submitting to the same buildkitd, the cache may be pruned by the time you submit the ID, in which case the build will need to run again.

Instead, you should just always re-run all of the code for your pipeline. So then, if there does happen to be a cache miss anywhere, you will guaranteed to also re-run the tests before deployment. But in the more typical case where you get cache hits, the tests don't need to re-run and everything is fast.

[ashen dawn]

re-running everything isn't realistic in this case, compliance requires environmental segregation so a server running a prod deployment can't rerun a performance test etc

but just ending / resuming from a container artifact should fit well with to go sdk