Secret Not Found after upgrading to 0.10.0

[dark vale]

Hello, after upgrading to 0.10.0, my pipeline no longer works. It breaks in secret. WHat could be the reason?

 ERROR: secret 3665363537343732363362363133363739613038313464396563373732663935643737386333356663356666313639376334393337313536353363366337313231343432393263356164b613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad: not found

Secret Not Found after upgrading to 0.10.0

my code looks like this

    netrc = client.set_secret("netrc", to_netrc(url, username or "", password or ""))

    clone = ["git", "clone", "--recursive"]
    if ref:
        clone.extend(["-b", ref])
    clone.extend([url, "."])

    container = (
        client.container()
        .from_("alpine/git")
        .with_env_variable("__CLONE_ID__", clone_id or str(time.time_ns()))
        .with_workdir("/src")
        .with_mounted_secret("/root/.netrc", netrc)
        .with_exec(clone, skip_entrypoint=True)
        .with_exec(["git", "rev-parse", "HEAD"], skip_entrypoint=True)
    )

[wicked axle]

This is likely a regression from https://github.com/dagger/dagger/pull/6601, will take a look

[dark vale]

thanks @wicked axle

[wicked axle]

Do you have a minimal reproducer?

Does the example in https://github.com/dagger/dagger/blob/e141b96556714ad8bdc690317b70884ea846a4bc/examples/sdk/python/secrets/pipeline.py work for you?

It's really weird, we do have tests for this behavior so I suspect there might be something else going on here as well

[dark vale]

This one works

[dark vale]

let me try create it

[dark vale]

here you go.

async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        netrc = client.set_secret("netrc", "doesn't matter")
        clone = (
            client.container()
            .from_("alpine/git")
            .with_workdir("/src")
            .with_mounted_secret("/root/.netrc", netrc)
            .with_exec(
                [
                    "git",
                    "clone",
                    "--recursive",
                    "https://github.com/TelkomIndonesia/vector-vault-unseal",
                    ".",
                ],
                skip_entrypoint=True,
            )
            .with_exec(["git", "rev-parse", "HEAD"], skip_entrypoint=True)
        )
        await clone.directory("/src").docker_build().sync()


if __name__ == "__main__":
    anyio.run(main)

[dark vale]

i thinks its the .with_mounted_secret("/root/.netrc", netrc). remove that and the build succeded

[dark vale]

what what's weird is that it fails on the docker build, which doesn't use the secret

[dark vale]

hi @wicked axle , sorry for pinging, but is there any workaround for now,
i would love to use the dagger function on our existing pipeline

[wicked axle]

yeah no worries! sorry, i was off sick the last half of the week 😢

i've managed to reproduce the issue now, and i'll be trying to get a fix together for this week's patch release (which hopefully we'll be running tomorrow)

will let you know if i find a workaround!

[wicked axle]

ok i don't think there's a proper workaround sadly 😢
maybe you could Export the result of clone.directory, and then reload it in? I think that should work.

however, i've worked out a fix, you can track it's progress in https://github.com/dagger/dagger/pull/6809

[dark vale]

awesome, thanks a lot @wicked axle , will keep track on it.

[cursive turtle]

there is a temporary workaround which is using a private cache volume across two steps and then copy the contets of /src/ in user-space to do the docker build. It's not elegant but it does the trick

here's a snippet that should work @dark vale

async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        netrc = client.set_secret("netrc", "doesn't matter")
        cv = client.cache_volume("src")
        clone = (
            client.container()
            .from_("alpine/git")
            .with_mounted_cache("/src", cv, sharing=dagger.CacheSharingMode.PRIVATE)
            .with_workdir("/src")
            .with_mounted_secret("/root/.netrc", netrc)
            .with_exec(
                [
                    "git",
                    "clone",
                    "--recursive",
                    "https://github.com/TelkomIndonesia/vector-vault-unseal",
                    ".",
                ],
                skip_entrypoint=True,
            )
            .with_exec(["git", "rev-parse", "HEAD"], skip_entrypoint=True)
        )

        (await client
             .container().from_("alpine")
             .with_mounted_cache("/src", cv, sharing=dagger.CacheSharingMode.PRIVATE)
             .with_exec(["cp", "-r", "/src", "/app"])
             .directory("/app").docker_build()
        )


if __name__ == "__main__":
    anyio.run(main)

[dark vale]

i see, let me try, thanks @cursive turtle

[cursive turtle]

sure! exporting to the host as Justin said also works and will probably be less code. The only caveat is that you're relying on the user's machine to export and then re-upload files to the build context across steps

[dark vale]

Hi @wicked axle , just want to confirm. I've seen that this PR is merged and available to 0.10.1, but even after upgrading (and making sure the engine is running 0.10.1), i still got the same error for the reproducer.

[wicked axle]

mm no you're right, silly of me to not test the fix on the original repro

there's some internal logic i'm not fully understanding, but this fix appears to work: https://github.com/dagger/dagger/pull/6837

(confirmed on the repro)