dagger restarts makes pushing to a local registry a flake

[sharp raven]

Hi folks

I have been investigating all this week on how to store small artifacts in a place between dagger runs ref [previous messgae](#1278620859304640534 message). i finally setled on using a local registry instead of using a mounted cache. but this also is failing at least on wsl. I am not sure what is going on and would appreciate any suggestions

I attached the code i use to start the local registry using go after scouring from posts here. every now and then the dagger engine restarts i have no idea why i just see that it did

docker ps -a 
CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS          PORTS     NAMES
404b3c2e8c04   registry.dagger.io/engine:v0.12.5   "dagger-entrypoint.s…"   9 minutes ago    Up 9 minutes              dagger-engine-7d4613636ea6099e
0cac1be05978   registry:2                          "/entrypoint.sh /etc…"   31 minutes ago   Up 31 minutes             dagger-registry

    _, err = dag.Container().Publish(ctx, "localhost:5000/my-org/my-repo:latest")
    if err != nil {
        return fmt.Errorf("publish sha file failed for %s", shafile)
    }

any ideas on how
Regards

as soon as dagger restarts happen all my builds fail because they cant get a hold of the local registry

is there a way i can make both the engine and registry sync after a dagger restart ? or should i be using a url address that is always garanteed to be valid when pushing/pulling from this local repo

[crisp patio]

@sharp raven it kind of looks like you're trying to reimplement with a local registry what the dagger cache should get you for free. What happens if you just run your dagger functions one after the other, and make no effort at persisting in between runs? normally persistence should happen automatically via caching

[sharp raven]

Hi Solomon. i tried to explain the rationale in the link i have above. there is a small amount of metadata that i need so that i dont hard code the commit sha between my repos otherwise i am reall y surprised at what dagger is doing .. its short of magic really. so yeah i get it but i could not figure how to work around this without more problems

the gist of it is .. map release-name to commit-sha .. i wish dagger already had this <day-dreaming/>

[lament scaffold]

@sharp raven I still believe you shouldn't need a registry for this. I've tried making sense of your previous thread but I'm not exactly sure what you're trying to achieve here. Do you mind sharing a more concrete example about what you're trying to share across different dagger operations?

[sharp raven]

my apologies for not making this clear ..

i have repo {a1, a2} a1 depends on a2
when i am building a1 i need something that converts the string a2.latest which means give me the commit sha for the latest release of a2. this way a1 does not have the commit sha of a2 hard coded

once i have a commit sha for a2 the dagger logic kicks in and makes sure i only build that commit sha once

i really wish this is something that dagger already had because i am not storing a lot of information

the alternative would be for a1 to either hard code the commit sha of a2 in the build logic ...

or when building a1 clone and buildthe latest version of a2 ...

the question is if you do this you are going to build a2 again and again because its not a fixed release ...

i hope this makes sense ?

[crisp patio]

so you want a lookup function to "lock" the latest commit of repona2, and store that commit in a file somewhere in the a1 repo, so that subsequent builds from a given commit of a2 always pull that same commit of a1? And presumably you want a way to update the lock file when you decide to, exactly like you would update a dependency in a regular package manager.

Is that right @sharp raven ?

[sharp raven]

thart is correct !!!

right now the best place so far is the registry.

a1 just pulls that from the registry using a unique name something like localhost:5000/myorg/repo-a2:latest this place is garanteed to have the latest release always

i am storing a json file in there that contains the commit sha for a2

once i have that commit sha i pass it to function that is building a2 and that is automatically cached by the dagger magic since it will always be the same sha until a new release is made

also this scheme is not language specific so it works for any repo that needs to be built without using a package.json for example

[lament scaffold]

the dag.Git call should help you with this.

fmt.Println(dag.Git("github.com/dagger/dagger").Branch("main").Commit(context.Background()))

[sharp raven]

ok but that does not lock it to a release

[lament scaffold]

that will always give you the latest commit for that repo

[lament scaffold]

what do you call a release?

[crisp patio]

@sharp raven there is a stopgap solution and a longer term solution

[sharp raven]

thats not what i want .. it helps to get the sha ... but it will not tell me the commit sha of repo a2 that is the current release

[sharp raven]

i reall ythink this is a useful feature for dagger ... and would love what you folks come up with

[lament scaffold]

what does it mean in your case that a repo has a release?

is that a github release?, a tag?

[crisp patio]

short term: you implement the lock file logic yourself in the dagger module. you will need access to your source repo as argument (to read the lockfils). you can have a separate function that updates the lockfile, which will return a directory (that you can the export with dagger call -o .)

[sharp raven]

so evey now and then a build function is called that builds every repo and does ci . then at the end of that function and when all is successful i store the commit sha for all repos in the registry like i described above

so this is a mapping that maps a repo name to a commit sha .. think of it as if we are using the registry as your go.sum file in a non language specific way

[crisp patio]

longer term, we actually want to support lookup functions as a first class citizen. Once that feature is available, you would mark your function with // +lookup or something to that effect, then dagger would manage the lockfile logic for you, using the dagger.json, or perhaps a new dagger.lock, as the repository. like a general purpose lockfile

another stopgap you can do, if you don't need precise control over the update schedule, and can live with a best-effort time-based update schedule ("update to the latest commit every X hours or Y days, roughly") then you can simplify the implementation and skip the lockfile: instead call your lookup function every time, but with a ttl on the caching. that way the cache expires roughly every X days and the lookup is called again. this is simpler to implement, but less precise

[sharp raven]

yeah i thought about the time based release but gave up on it quickly

frankly all i need is a function from dagger that would act like a key value store ... i give it a string and it returns a string .. simple

[crisp patio]

simple for you, but very hard to implement correctly 😛

[sharp raven]

i hear you !

[crisp patio]

why did you give up on ttl?

the key is your constraints of persistence and precision

based on your constraints we can find the right solution for you

[sharp raven]

because there is no garanty the build of the last hour was successful .. its not precise and deterministic enough as the mapping based approach

[lament scaffold]

I guess this depends on how much control you want to have about when to release a2 in your pipeline. Doesn't sending the commit SHA of a2 should be enough for this? . For example:

dagger call build --a2-sha ABCD

^ the first time you call this, you build a2 with the ABCD commit and then use that result to build a1. Any other subsequent builds that you don't want a2 to be built, you just pass the same SHA.

Now, if you DO actually want to re-build a2 and a1, you just pass any other a2 sha to your build pipeline.

[sharp raven]

thats the crux of the issue .. we are trying to not hard code this i nthe build logic

like you said once you know the commit sha its all dagger magic at play

[lament scaffold]

but someone somewhere has to say: "hey, this commit is a new release". How do you currently express that?

[sharp raven]

so evey now and then a build function is called that builds every repo and does ci . then at the end of that function and when all is successful i store the commit sha for all repos in the registry like i described above

so one part does the garanteed write to the registry

the other just picks it up and pass it to dagger

[lament scaffold]

ok, I see. It's two separate pipelines then. What you're looking for basically is a way to pass artifacts across different Dagger engines

[sharp raven]

the write step is a single write that either is a pass or fail .. we are basically updating the lock file sort of say

[sharp raven]

indeed .. but the only artifact i need is a json file that has a unique non changing name. and a registry seems like the best place for it at least for now

[lament scaffold]

yes, a registry works, gotcha

if you're still having issues with the restart problem, we can check that out tomorrow

[sharp raven]

no problem ! i really appreciate all the discussions and help here. it might have been a wsl issue. now i switched to running my pipelines in a nix machine and i have not seen restarts since then

[crisp patio]

I think storing the sha in a file in the repo would be more practical. lock files are a well established pattern, and you'll carry less infra dependency

[sharp raven]

might be more difficult for me actually. i have to wait unti all repos are build to generate those lock files.

[lament scaffold]

either this or have a latest tag on each repo that you overwrite everytime there's a release

then you can just use latest everywhere

[sharp raven]

might be better than a lockfile but still its a multiple commit accross multiple repos. and the hope is none of them fails. its not a write once on a single lock file thing which the central registry has

[lament scaffold]

gotcha. So you generate 1 lockfile for all the repos. Still, seems like committing that lockfile in git might be better than pushing that to a registry as Solomon was suggesting. That way, you can have a version history about what commit each repo was at for every given release.

[sharp raven]

good point !

so i have release repo already i could commit a single lockfile in the single rel repo

[lament scaffold]

https://tenor.com/bpEm0.gif