IDs change with each commit to my project

[deep raft]

This was more of a curiosity question.

At one point with dagger go SDK (pre 0.5.0) I came to rely on using dagger.FileID obtained from a go.sum file I had as a cache key.
My goal was - assume this ID was stable and consistent for the contents of a file.

It seems this changed around https://github.com/dagger/dagger/pull/4615 when local git refs got plumbed in as labels.

I just wanted to check, as I am sure I made an incorrect assumption that I could use this ID when I chose to use it for a cache key, that I shouldn't really be using it.

I assume these labels in the progress groups dont go into computing the cache keys for the build graph?
Otherwise, my cache would bust on each commit. I don't think I have observed that.

This is mostly me thinking out loud 😂 just thought it was interesting to share. I don't know if anyone else has attempted to use these IDs in this way.

p.s. i've not invoked a Pipeline in my codebase. I am not sure what that is yet.

And if this is interesting unexpected. I have a small recreation:
https://github.com/GeorgeMac/dagger-cache

When I run this, add an empty commit and run it again the ID changes.
You can un-base64 it and see the git ref.
I saw that I needed to ensure origin was set as a remote on my local checkout of the project and that was enough to trigger this behaviour.

[last ice]

Pipeline is purely annotation metadata for visualization purposes. We’re developing tooling to use it, but not released yet

[blazing flame]

I assume these labels in the progress groups dont go into computing the cache keys for the build graph?
Otherwise, my cache would bust on each commit. I don't think I have observed that.

this is correct. re: using FileID as a cache key and why that changed with Labels, @wispy condor likely has the correct answer here since he worked on the labels support 🙏

[wispy condor]

file IDs do indeed include the pipeline metadata: https://github.com/dagger/dagger/blob/475f19908f90f6a5793caf8fa375e2201dc1b1dc/core/file.go#L32 - so yeah, different labels embedded therein will result in different FileIDs. but yeah, the build graph cache isn't affected.

this is likely affecting you now because there's always an implicit "root" pipeline that contains those git/github labels

which is a recent change (engine v0.4.0, go SDK v0.5.0)

[last ice]

maybe there’s a separate “file digest” field we could add that gives you an actual content digest for the file? I know I could use that for other things

[deep raft]

Thanks for all the details folks! 😄 Makes total sense.
Exposing the files digest would be a neat quality of life improvement 💯

[wispy condor]

interesting idea. theoretically this could apply to anything in the DAG, right? tempting to add something similar for Container, but that might be confusing since digest already has a certain meaning there

[last ice]

yes container is what i need it for actually 😁

for the situation where you push docker images to a tag that includes a git commit ID for immutability; except you don’t want to repush the same image to 100 different tags just because it wasn’t changed for 100 commits

[quiet blaze]

@last ice for the case you describe, I'm using the git object ID. It does directly depend on git though.
git rev-parse HEAD:path/to/dir

[last ice]

that’s great actually 🥇 does it work well for you?

[quiet blaze]

Reducing the number of published artifacts is great. We are still in the middle of fixing up various tooling to change how it understands tags. There exists a deterministic way to determine the tag you want based on a git ref, but only if you have the git repo. I might have to jump through some hoops to make that mapping available in some other contexts.