Binary secret mounting

[smoky fog]

I have a cert that's password protected and the goreleaser toolchains handle this (cosign/quill).

I ran into the issue of passing through with invalid incoding, and I think the issue is related to: https://github.com/dagger/dagger/issues/5069

Is there any current option I can use to safely mount temporarily the certificate for the tools without concern it persisting? The string(b) didn't work this time around.

To be clear I'm invoking goreleaser-cross to wrap my image builds as I need the C toolchain to properly handle CGO_ENABLED=1 for one verison of the binary.

[half leaf]

You can do something like this in the meantime:
https://play.dagger.cloud/playground/rhYZiDKqJ_k

This won't leak any secrets

[smoky fog]

I'm running mage tasks and this is one minor step to mount in the middle of a call to goreleaser-cross. I'm not quite certain how I'd use that since it wouldn't be in scope of my current job right? Would I use a herestring and invoke graphql directly in Go job for that one file instead? Never used graphql before and in dagger/go sdk either so any quick pointer would be useful.

If you think that fix will be in the next 2-3 weeks I can wait as the release can be delayed.

[half leaf]

👋 I shared the graphql equivalent so you can see how I set a base64 encoded secret and then decoded it to a WithMountedTemp folder so the secret doesn't leak in the cache.

you can achieve the same thing using any SDK

how do you tell goreleaser-cross to look for the cert currently?

[smoky fog]

the goreleaser running is from goreleaser-cross image, so it expects the file to exist at the path of the QUILL_CERT for example.

I'll have to look at mountedtemp and see how this was done. Thanks for helping out 👍

Here's an example of the dynamic loading I'm doing. I have a struct providing a list of variables and the struct designates if it's a file, a secret, or just a value.

i tried pasting code but didn't work so just taking screenshot to give you an idea

So in otherwords, now that I looked more closely, you are saying in the meantime I just base64 encode outside, then base64 decode using inline shell in container? That makes sense 👍

Basically any update to dagger later would just provide a helper to do this to simplify things, but wrap up that basic activity for me correct? I'll just use some good ole elbow grease to getter done.

[smoky fog]

Having discord issues, so posting a couple messages to wrap this up

@half leaf ok, here's what I got...

secretFile := os.Getenv(reqVar.Name)
secretFileBase := filepath.Base(secretFile)
hostSecretFileDir := filepath.Dir(secretFile)
containerSecretsTempDir := "/tmp/secrets"
secretFileFullPathInContainer := filepath.Join(containerSecretsTempDir, secretFileBase)

# READING AND ENCODING TO BASE64
golang = golang.WithEnvVariable(reqVar.Name, secretFileFullPathInContainer)
fileContents, err := reqVar.FileContentsToBase64()
if err != nil {
    pterm.Error.Printfln("error setting file content as base64: %s", secretFile)
    return err
}

Now i'm creating the target TEMP file that will be need to be decoded.
I want to end up with the temp file being outputting to the /tmp/secrets/filename.ext

// mountTempFileName is the temporary file to create in the context of container
// which will then be decoded internally and flipped out to the target file path from the reqVar value provided.
const mountTempFileName = "/buildtmp/secret-to-decode"

// mountSecretTempFile is a secret that is a file containing the base64 encoded contents that the exec statement will decode to the
// final file which is not in /buildtmp, but located in `/tmp/secrets/` and should be safe
mountSecretTempFile := client.SetSecret("/buildtmp/secret-to-decode", fileContents)

Now I think this is close. Build works, but can you confirm this looks to be correct to securely use the secret without any risk of exposure?

golang = golang.
    WithMountedSecret("/buildtmp/secret-to-decode", mountSecretTempFile).
    WithMountedTemp("/buildtmp").
    WithExec([]string{
        "sh",
        "-c",
        "base64",
        "-d",
        "/buildtmp/secret-to-decode",
        ">",
        secretFileFullPathInContainer, // for example /tmp/secrets/mysecretfile.pem
    })

I know that's a lot, but I tried to trim it done to focus on the important piece. Look good?
Haven't read up on the secret caching stuff, so trying to make sure my secret target path doesn't get exposed from /tmp/secrets too.

Sorry for multiple messages but I hope this helps

[half leaf]

👋 about to hit the gym. Will reply in ~1.5h

[smoky fog]

no rush. totally fine to do tomorrow. Just trying to make sure i'm on the right path and staying secure 👍 I'm done for the day

[half leaf]

yes, this seems correct. Let me know how it goes!

[smoky fog]

@half leaf because I'm mounting both the using the secret plus mounted tmp all of that is secure right? I need to go reread the blog or github discusison somewhere and see more on the architecture of how that works. Just right now making sure i don't need to be concerned about the certs/files persisting if using the MountedTemp (seems obvious but just checking) and mountedsecret api.

[half leaf]

yes, everything's secure as you're using the secret and the tmpfs volume

[smoky fog]

Ran into a blocker. It's a password protected file. I think that means I can't read the encrypted content to base64 encode in the first place. I'll have to look at an alternative approach, maybe it's a good time to switch my signing process too 🙂

It's a binary password encrypted pfx12 type file. Do you expect that to work as we chatted about or to have to unlock the file somehow first?

[half leaf]

I'd expect that to work regarldess since base64 works at the byte level, not at the file extension level

if you do base64 <your_file.pfx12> it'll work

[smoky fog]

So I'm stuck still... any help would be appreciated as I'm trying to release. Anyone on team that could help iron this out?

I see changes upcoming in 0.8 but not in a current release.

I just ran upgrade to latest version of dagger dagger.io/dagger v0.7.4 now.....

I have a password protected pfxcert... following the earlier conversation I did:

golang = golang.WithEnvVariable(reqVar.Name, secretFileFullPathInContainer)
            fileContents, err := reqVar.FileContentsToBase64()
            if err != nil {
                pterm.Error.Printfln("error setting file content as base64: %s", secretFile)
                return err

The filecontents to base64 errors out. I contruct the target path with this:

containerSecretsTempDir := "/tmp/secrets"
            secretFileFullPathInContainer := filepath.Join(containerSecretsTempDir, secretFileBase) // example /tmp/secrets/mycert.pem etc. This is a different path in continer than host as it's nested build in container.

I'm just not certain if you can even read a password protected pfx file bytes raw to encode and pass through so the underlying tools would correctly decrypt.

Maybe there's a different approach I should take with mounting the file directly or something 🤔

I won't be back around to this project for maybe 6 months so if it's not a clear answer I'll just move on. Was just hoping to leverage the weeks of work to daggerize today if I can iron out that last error.

[violet epoch]

@smoky fog so you're employing a pattern like this, right?
https://docs.dagger.io/723462/use-secrets/#use-secrets-from-the-host-filesystem

happy to repro.
do you have a good way to generate a password protected pfxcert?

figure I'd then base64 encode it and leave it on the filesystem

then pull it into a container, base64 decrypt it, and then deal with the password protection: how do you plan to do that?

I'll use my windows machine to do this with powershell

[smoky fog]

thank you! i just saw your response... let me look.

Ok... so for prior context... other than my code examples....

In CI i download a password protected PFX file.
I'm using dagger to run the goreleaser-cross build so it's a nested docker type call as I need to handle CGO_ENABLED.

So what I provided above was a work around for the defect on GraphQL not handling the characters for a binary (will be fixed in 0.8.0) .I did the instructed base64 encoding, but using the method I pasted above (see this link #1115404536400584704 message) it still says can't base64 encode the file.

Does that help? I did a lot of work on this in the past and shelved my work for nearly 4-6 months because of this issue and now it's time to release. Ideally the mac signing and such still works doing this method, but the encoding of that cert seems to be the last blocker.

[violet epoch]

welll...0.8.0 is released as of today 🙂

[smoky fog]

@violet epoch and no i'm sorry I don't have a way to reproduce generating one of those.

[violet epoch]

I was able to create a password protected pfx, no prob

[smoky fog]

Well that's cool!
I'm still doing what I was told should solve that so am I dealing with a problem with graphql and base64 still and must do that upgrade or it this a red herring you think?

I think there's a lot more possible breaking changes I don't have time to refactor for 0.8.0... this project is "on ice" so i'm just trying to avoid all that dagger work being unused, but if I can't figure that out i'll have to just ship unsigned and then do it manually afterward 😢

[violet epoch]

Happy to jump on dev audio to sync up

maybe we can figure something out in pairing

[smoky fog]

sure thang!
Someone on my team mentioned that base64 wasn't necessarily url safe so maybe something is conflicting with the call on certain things like + or =.

want to ring me up in DM call here in discord? Might need to restart to make sure I can do screen share

[violet epoch]

Can jump in dev audio or in DM

[smoky fog]

cool. I'll call ya. Might have to restart (due to mac settings) if I screen share anyway

[heady needle]

In 0.8 you can now set secrets from a file in the host. Sorry I don’t have context just point out this is new.

As for the breaking changes they’re mostly deprecation removals so you may not be much affected.

[smoky fog]

Very helpful thanks to @violet epoch for helping show me the primary issue was PEBCAK

I am running into one last issue related to goreleaser-cross + buildkit that is

73: exec /usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mypasswordprotectedcert.pfx
73: [1.01s] [WARN  tini (11)] Tini is not running as PID 1 and isn't registered as a child subreaper.
73: [1.01s] Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
73: [1.01s] To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
73: [1.01s] WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
73: [1.01s] Configure a credential helper to remove this warning. See
73: [1.01s] https://docs.docker.com/engine/reference/commandline/login/#credentials-store
73: [1.01s] 
73: [1.01s] Error saving credentials: rename /root/.docker/config.json337799365 /root/.docker/config.json: device or resource busy
73: exec /usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mypasswordprotectedcert.pfx 
ERROR: process "/usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mypasswordprotectedcert.pfx" did not complete successfully: exit code: 1

Now if this was on desktop I had a custom warning on this step telling me

WARNING  if you get an error about  error getting credentials - err: docker-credential-desktop resolves to executable in current directory (./docker-credential-desktop), remove the value "credsStore" : "desktop" from $HOME/.docker/config.json

But i'm not on desktop, running in ephemeral ci context. If there's an immediate obvious thing I'm missing appreciate any help ,otherwise no problem.

Working on this right now.

I'm going to upgrade to 0.8.0 and see if the new setSecretFile helps with any of this. The decode commands and more added a lot of complexity. Maybe this will help. Crossing my fingers nothing else I'm using is deprecated 😆

[smoky fog]

So I’m almost there. The conflict with the mounting of the docker.config is still failing though. I think this is specific to when I mount the dockerfile and do the rename in the temp file. Not certain yet how to fix that one. Anyone run into that? Pretty sure someone mentioned working with Goreleaser-cross as well.

[sudden kelp]

you’re getting the error about the docker desktop auth helper?

[smoky fog]

I had that problem locally. Right now in the CI system it seems to conflict when renaming the temp file to the target docker.config file. The docker desktop mention was my local debugging error.

So ignore the warning (2nd message) as that was my custom error to help guide a local release failure. The first one with Tini messages was from the build error where Goreleaser-cross. Maybe Monday I’ll dig out some stuff from y’all’s repo as I swear one of you had used Goreleaser-cross and if it was doing publishing I’m sure this had to be solved too.

[violet epoch]

Let us know how this is going and if we can help more to figure this out today! 👋 cc @lapis vortex

Context @lapis vortex that Sheldon is using goreleaser-cross and was hitting some issues. Not sure of current state.

[smoky fog]

Thank you! Still stuck on the error

73: [1.01s] Configure a credential helper to remove this warning. See
73: [1.01s] https://docs.docker.com/engine/reference/commandline/login/#credentials-store
73: [1.01s] 
73: [1.01s] Error saving credentials: rename /root/.docker/config.json337799365 /root/.docker/config.json: device or resource busy
73: exec /usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mypasswordprotectedcert.pfx 
ERROR: process "/usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mypasswordprotectedcert.pfx" did not complete successfully: exit code: 1

Trying to figure that one out. On desktop I'd just delete the creds, but in CI, not sure why this happens. I'll see if I can dig out the other examples of goreleaser-cross from dagger's discussions and compare.

[sudden kelp]

@smoky fog what code is printing that error? Is it a custom script by you, or a third-party tool?

[smoky fog]

That's from running goreleaser-cross. The error specifically to do with the docker config file unable to write. I'm using that decode line.

I can't find other goreleaser-cross examples by the team though I thought something existed. I searched all of github and didn't see anything equivalent (might be something I need to blog on 🙂 )

[smoky fog]

So I read that docker doesn't let you rename a file that's mounted.

Since the code is mounting the secret temporary file and then running sh -c base46 -d /builtmp/secret-to-decode > secretFileFullPathInContainer ... this would be

rename /root/.docker/config.json1907217525 /root/.docker/config.json

And tini's command exec /usr/bin/tini -- /entrypoint.sh sh -c base64 -d /buildtmp/secret-to-decode > /tmp/secrets/mytargetfile.pfx.... I'm wondering if the mounting order for me is wrong somehow. I'll look at the new 0.8.0 secret mounting to see if this helps too.

Ok, so i think the root error was coming from this location

    if ci.IsCI() {
            dockerConfig, err := os.ReadFile(filepath.Join(dockerConfigDirectory, "config.json"))
            if err != nil {
                pterm.Error.Printfln("error reading docker config file: %s", dockerConfigDirectory)
                return err
            }
            dockerSecret := client.SetSecret("DOCKER_CONFIG", string(dockerConfig))
            golang = golang.WithMountedSecret("/root/.docker/config.json", dockerSecret) // using dagger secrets api
        }

Any immediate red flags on that mounting?

following the example (which I'm assuming is updated for 0.8.0 docs https://docs.dagger.io/7442989/cookbook#expose-secret-via-file

Seems other than the format the SetSecret with MountedSecret next is the same approach...

So something is happening after that step when running the goreleaser part and the base64 trying to rename and that I'm trying to dig through my terribly patchwork code and figure out as maybe the first mount is right but then something tries to update it again. not sure. yet

[sudden kelp]

I'm still piecing together all the context, so sorry if I'm asking dumb questions but: this command sh -c base46 -d /builtmp/secret-to-decode > secretFileFullPathInContainer is NOT renaming any file, and I don't see how it can be the source of that error

I'm still confused as to what code actually attempts to rename a file

[smoky fog]

I know me too 😆
This was prior to 0.8.0 i had to handle base64 encoding the secret file contents, mounting temporarily into the container and then have it run the base64 decode.

I'm not if it's an issue with goreleaser-cross or my process right now. I'm stripping out my base64 encoding step to eliminate that extra complexity first as it's fixed upstream and then will see if error repeats.

In rereading the error output I tend to think it has to be an issue with the base64 decode step on my passed in file. I'll remove the extra step here and see what happens. Maybe it will go away 🎉

I think that now I can just mount directly as needed instead of having the interim step. Trying now

[smoky fog]

Error saving credentials: rename /root/.docker/config.json4280309876 /root/.docker/config.json: device or resource busy

Also after the dagger output I got this dozy of a message

Error: input:1: container.from.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withMountedSecret.withSecretVariable.withEnvVariable.withMountedSecret.withSecretVariable.withEnvVariable.withMountedSecret.withSecretVariable.withSecretVariable.withSecretVariable.withSecretVariable.withSecretVariable.withSecretVariable.withSecretVariable.withMountedDirectory.withWorkdir.withMountedCache.withMountedCache.withUnixSocket.withEntrypoint.withExec.directory process "/entrypoint.sh release --clean --skip-validate --release-notes=.changes/v2.0.0.md" did not complete successfully: exit code: 1

That's pretty funny.

Anyway so now the base64 decoding isn't the issue. I think it has to do with the conflict in mounting the docker config when maybe it's blocked by the goreleaser container. I'm looking at the docs again for goreleaser-cross (which mention to do this mount). The busy signal seems to be something others have run into in a general sense because of trying to update a mounted file. Dagger is trying to rename the temp file to the target path so might be the source of that conflict.

[sudden kelp]

It's probably the docker CLI itself: https://github.com/docker/cli/blob/06de1f8d2c56737ea06a547197a5ae3701aa5397/cli/command/registry/login.go#L157

Here's the rename: https://github.com/docker/cli/blob/06de1f8d2c56737ea06a547197a5ae3701aa5397/cli/config/configfile/file.go#L177

It's just a way for the docker CLI to atomically make a change to the config

So what I think happens, is that you run a docker command that changes the docker config file (for example to add the credentials after running docker login?), and that fails because it tries to be a smartass about it and swap in a temporary file, which fails because the file is mounted

[smoky fog]

I'm running in azure pipelines. The beginning of the job has the plugin task for

- task: Docker@2
  displayName: docker-login
  inputs:
    command: login

Later I run mage release:all and this invokes dagger.
I'm checking goreleaser-cross too in case it does something I'm missing.

I see this in the ./entrypoint.sh which I have to call as the entrypoint for dagger.

https://github.com/goreleaser/goreleaser-cross/blob/6b58c993c3a606b4910c17691bb4b0f4a72b9c7c/entrypoint.sh#L30

Maybe I should set an environment variable for DOCKER_CREDS_FILE instead of mounting the file directly?

trying now.... 🤞

I think i'm close...

Error response from daemon: Get "https://ghcr.io/v2/": denied: denied

I'm not using ghcr.io for anything. I think that's the goreleaser image is using and now it got confused on the docker login for docker hub + gchr just for the pull I think

const upstreamimagetopull = "ghcr.io/goreleaser/goreleaser-cross"

gonna flip over to the docker hub equilavent and see if that helps.

Could use some advice... last thing I think.

• ghcr.io is where one of the underlying images in the goreleaser build occurs.
• When I run docker login for docker hub i think I'm good.
• Now though ghcr public image to pull tries to use the docker credentials for the ghcr, which I don't want to happen.

I think the fact I'm passing in GITHUB_TOKEN for pulling packages in the job but not for publishing is confusing the image. I think I need to make sure it's set to empty/null in the dagger call. Checking that now too. 🔨

For simplicity I'm going to NOT use github token right now. The job was supposed to support this for scoop/brew installs, but I'm disabling as it's causing this conflict.

This helped. Only need to finish mounting directory correct and hoping it's done. The removal of the GITHUB_TOKEN is a goreleaser logic issue (very confusing sometimes when you don't use github).... Now it didn't mount my git directory due to using the custom dagger mounting logic. A few more tweaks and hope this is good

So i customized the same approach ya'll did. Maybe I did the expression wrong. I tried to mount the entire .git directory but maybe it's not correct.

[sudden kelp]

@smoky fog sorry I may be way too late on this, and I still dont understand everything going on. But based on my understanding of the problem, at some point docker is executed by Dagger, with a file mounted to /root/.docker/docker.config, and that causes the rename error - correct?

[smoky fog]

ignore prior, i've logged my progress. Current issue remaining is the

65: [1.30s]   ⨯ release failed after 0s                          error=current folder is not a git repository

I showed the custom code I emulated from ya'lls repo to mount, but included the .git directory as I needed it for goreleaser. Not certain why that didn't get mounted

So I backtracked where this all is mounted...

The examples from goreleaser-cross:
https://github.com/goreleaser/goreleaser-cross-example/blob/a5a2d67e191918dbe322589d66586f67e8a66914/Makefile#L16

    @docker run \
        --rm \
        -e CGO_ENABLED=1 \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v `pwd`:/go/src/$(PACKAGE_NAME) \
        -v `pwd`/sysroot:/sysroot \
        -w /go/src/$(PACKAGE_NAME) \
        ghcr.io/goreleaser/goreleaser-cross:${GOLANG_CROSS_VERSION} \
        --rm-dist --skip-validate --skip-publish

Now in my invocation i did this... this is mounting and setting the working directry just like the example. This should mean the contents mount into the right path I'd think. Same repository go code from dagger's repo.

    repo := RepositoryGoCodeOnly(client)
    golang = golang.WithMountedDirectory("/src", repo).
        WithWorkdir("/src")......

[lapis vortex]

Does Repository(c) come from the host or from a git clone?

[smoky fog]

Hmmm. The CI job is running (and does find the .goreleaser.yaml) that is in the source directory. I'm mounting almost identically to the dagger repo's approach. (I just tried glob variation on .git directory so before it was just .git. That would be the client (second image).

[lapis vortex]

I was just wondering if Repository() does a client.Git() to get the directory in your screenshot, since you may need client.Git(repo, dagger.GitOpts{ KeepGitDir: true })

[smoky fog]

I'm crunched on time (this project is about to sit for a while once I release). In the effort of simplicity ijust mounted the client directory and it worked.

Now unfortunately I got no error detail output. Remind me how can I enable better debug output when I'm invoking dagger via mage? Maybe I need to invoke goreleaser with debug logging.

All it told me

62: exec /entrypoint.sh release --clean --skip-validate --release-notes=.changes/v2.0.0.md ERROR: process "/entrypoint.sh release --clean --skip-validate --release-notes=.changes/v2.0.0.md" did not complete successfully: exit code: 1

😢

[sudden kelp]

You can wrap the whole thing with dagger run to get the sweet TUI

(I think it does include more useful error context, but not 100% sure)

[smoky fog]

I might just do that. I'm running purely in CI right now so the dagger run (I've used it) isn't invoked. Instead it's running purely with mage invocation.

I didn't want to run locally cause I'm on M1 and didn't build originally with all the M1/amd64 stuff ironed out so was afraid I'd build wrong. Another thing to troubleshoot 😆