Secrets in the CLI | Dagger | Page 1

sage raven May 23, 2023, 1:09 PM

#

Hey, the built-in secrets exist in memory, so you need to run those queries in the same dagger session. Multiple dagger query commands will trigger new sessions.

sage raven May 23, 2023, 1:58 PM

#

In the meantime you can use curl with dagger run sh ....

little marlin May 23, 2023, 2:00 PM

#

sage raven Hey, the built-in secrets exist in memory, so you need to run those queries in t...

good observation.. this changed with the setSecret feature but I don't think we were aware how it'd affect dagger query. @astral lynx any thoughts here? since setSecret cannot be used now with dagger query

sage raven May 23, 2023, 2:02 PM

#

If query reuses an existing session then you can just run the script with dagger run, e.g., dagger run sh -c 'dagger query ...'.

#

Either way, maybe you have a different source for secrets that doesn't depend on setSecret. We have plans to make this easier by being able to swap the secret store for another (e.g., 1Password), in the future.

#

https://github.com/dagger/dagger/issues/5181

GitHub

dagger query should reuse an existing session · Issue #5181 · dagge...

This goes for other commands since they share the same base for starting a session. This would allow wrapping multiple dagger query inside a dagger run script.sh. Otherwise you can't use setSec...

sage raven May 23, 2023, 2:35 PM

#

@void wing, try this as a workaround: dagger run script.sh, with following:

#!/bin/sh

set -euo pipefail

query() {
    jq -sRn '{query: input}' | \
    curl -s \
        -u $DAGGER_SESSION_TOKEN: \
        -H "content-type:application/json" \
        -d@- http://127.0.0.1:$DAGGER_SESSION_PORT/query | \
    jq -r .data
}

S_APP_ID=$(query <<EOF | jq -r .setSecret.id
{
  setSecret(
        name: "APP_ID",
        plaintext: "1234"
    ) {
        id
    }
}
EOF
)

output=$(query <<EOF | jq -r .container.from.withSecretVariable.withExec.stdout
{
  container {
    from(address: "node:latest") {
      withSecretVariable(name: "APP_ID", secret: "$S_APP_ID") {
        withExec(args: ["-v"]) {
          stdout
        }
      }
    }
  }
}
EOF
)

echo $output

void wing May 24, 2023, 4:56 AM

#

Thanks for the workaround..it works..

void wing May 24, 2023, 6:10 AM

#

what could be the reliable solution in future apart from workaround?

sage raven May 24, 2023, 7:13 AM

#

void wing what could be the reliable solution in future apart from workaround?

Wouldn't https://github.com/dagger/dagger/issues/5181 work for you? You'd be able to remove the query function and replace the calls with dagger query.

void wing May 24, 2023, 10:52 AM

#

sage raven Wouldn't https://github.com/dagger/dagger/issues/5181 work for you? You'd be abl...

yes once #5181 available that would work. meanwhile we are trying to use curl to query dagger with node as suggested. However in js file we are not getting correct values of secrets. can you please help us with that?

#** setup.sh** file for setting up secrets and running test.js in the container

#!/bin/bash
set -euo pipefail

query() {
    jq -sRn '{query: input}' | \
    curl -s \
        -u $DAGGER_SESSION_TOKEN: \
        -H "content-type:application/json" \
        -H "X-Debug:true" \
        -d@- http://127.0.0.1:$DAGGER_SESSION_PORT/query | \
    jq -r .data
}


S_APP_ID=$(query <<EOF | jq -r .setSecret.id
{
  setSecret(
        name: "APP_ID",
        plaintext: "123"
    ) {
        id
    }
}
EOF
)
echo "S_APP_ID: $S_APP_ID"

# scripts directory contains test.js file.
S_SOURCE_DIR=$(query <<EOF | jq -r .host.directory.id
{
  
        host{
            directory(path: "scripts"){
                id
            }
        }
}
EOF
)

output=$(query <<EOF | jq -r .container.from.withDirectory.withWorkdir.withSecretVariable.withEntrypoint.stdout
{
  container {
    from(address: "node:latest") {
      withDirectory(directory: "$S_SOURCE_DIR", path: "/") {
        withWorkdir(path: "/") {
          withSecretVariable(name: "APP_ID", secret: "$S_APP_ID") {
            withEntrypoint(args: ["node", "test.js"]) {
              stdout
            }  
          }
        }
      }
    }
  }
}
EOF
)

echo $output

test.js file for testing the secrets


console.log("Hello world");
console.log(process.env.APP_ID);

This is the output we are getting from the container when we run it using dagger run ./setup.sh. Getting filenames instead of secrets values.

Hello world
package.json setup.sh ...

sage raven May 24, 2023, 11:03 AM

#

Seems to work for me, but may I suggest you change withEntrypoint to withExec? Notice that the command that is run is node test.js node because node is in CMD, which in Dagger it's equal to withDefaultArgs to the entrypoint. Switching to withExec will replace node as the default command to using yours.

#

The this becomes 👇

void wing May 24, 2023, 11:05 AM

#

I tried ith withExec and it returns failed to solve: fork/exec ./setup.sh: exec format error

sage raven May 24, 2023, 11:06 AM

#

void wing I tried ith withExec and it returns `failed to solve: fork/exec ./setup.sh: exec...

I'm using your code as is (just the shell script and test.js, nothing else). What's your dagger version?

void wing May 24, 2023, 11:06 AM

#

dagger v0.5.2 darwin/arm64

sage raven May 24, 2023, 11:07 AM

#

One sec while I test.

void wing May 24, 2023, 11:12 AM

#

how can we add debug in curl? like we do --debug in dagger query in cli..can we do it in curl command?

sage raven May 24, 2023, 11:13 AM

#

You can do it dagger run --debug script.sh.

void wing May 24, 2023, 11:13 AM

#

yes but it is not debugging the query...its just adding connected to engine log and directly running the curl..

Connected to engine a20ef2dfeac9
failed to solve: fork/exec ./setup.sh: exec format error

sage raven May 24, 2023, 11:14 AM

#

The TUI does that: _EXPERIMENTAL_DAGGER_TUI=1 dagger run --debug script.sh

#

Can you show me your script as it's now?

void wing May 24, 2023, 11:17 AM

#

# setup.sh file for setting up secrets and running the container with test.js
#!/bin/bash
set -euo pipefail

query() {
    jq -sRn '{query: input}' | \
    curl -s \
        -u $DAGGER_SESSION_TOKEN: \
        -H "content-type:application/json" \
        -H "X-Debug:true" \
        -d@- http://127.0.0.1:$DAGGER_SESSION_PORT/query | \
    jq -r .data
}


S_APP_ID=$(query <<EOF | jq -r .setSecret.id
{
  setSecret(
        name: "APP_ID",
        plaintext: "123"
    ) {
        id
    }
}
EOF
)
echo "S_APP_ID: $S_APP_ID"


S_SOURCE_DIR=$(query <<EOF | jq -r .host.directory.id
{
  
        host{
            directory(path: "scripts"){
                id
            }
        }
}
EOF
)

output=$(query <<EOF | jq -r .container.from.withDirectory.withWorkdir.withSecretVariable.withExec.stdout
{
  container {
    from(address: "node:latest") {
      withDirectory(directory: "$S_SOURCE_DIR", path: "/") {
        withWorkdir(path: "/") {
          withSecretVariable(name: "APP_ID", secret: "$S_APP_ID") {
            withExec(args: ["node", "test.js"]) {
              stdout
            }  
          }
        }
      }
    }
  }
}
EOF
)

echo $output

Executing _EXPERIMENTAL_DAGGER_TUI=1 dagger run --debug ./setup.sh

█ ERROR ./setup.sh
┻ 
fork/exec ./setup.sh: exec format error

sage raven May 24, 2023, 11:20 AM

#

Still works for me... maybe you have some char in the file that I don't get when copying and pasting? Can you send me the file as an attachment?

#

And also, try updating to 0.5.3. That's what I'm testing on (latest release).

void wing May 24, 2023, 11:23 AM

#

yes i just upgraded version but no luck..yes sending you script as file just a min

#

📎 setup.sh 📎 test.js

#

updated to 0.5.3 dagger v0.5.3 darwin/arm64

sage raven May 24, 2023, 11:26 AM

#

Now I can reproduce 🙂

void wing May 24, 2023, 11:27 AM

#

thats good 🙂

sage raven May 24, 2023, 11:30 AM

#

Ah, you need to remove that first comment. The first line has to be the #!/bin/bash bang.

#

You had that when you last pasted but I unconsciously left it out of my copy.

void wing May 24, 2023, 11:34 AM

#

ohh..yaa now command is running...but are you able to see console logs of test.js in terminal? what is the secret value do you see when you execute dagger run ./setup.sh

sage raven May 24, 2023, 11:35 AM

#

It returns: Hello world setup.sh testscript.

#

Ok, let me check

void wing May 24, 2023, 11:36 AM

#

yes thats the issue...not getting the actual secret value in js...process.env shows env APP_ID but when we print it in console it is printing filenames

sage raven May 24, 2023, 11:40 AM

#

Just remember that dagger's logs will strip out the secret when you print it to avoid leaks, but you should be getting *** instead.

void wing May 24, 2023, 11:41 AM

#

yes thats true...if we run it without TUI dagger run ./setup.sh then it is printing filenames.. and if we run it with TUI _EXPERIMENTAL_DAGGER_TUI=1 dagger run ./setup.sh then it is printing nothing..not even Hello world.

sage raven May 24, 2023, 11:42 AM

#

That's because it's cached, more likely. In the next release it'll show the output.

void wing May 24, 2023, 11:43 AM

#

how we can make sure that js file is getting the correct values? can we debug it or store env in file through js?

sage raven May 24, 2023, 11:43 AM

#

You can use -i with the TUI for the interactive version.

sage raven May 24, 2023, 11:43 AM

#

void wing how we can make sure that js file is getting the correct values? can we debug it...

You can try to assert it.

#

Give me a sec to debug it.

#

I was getting the *** before, there must be something different in this version of your script.

void wing May 24, 2023, 11:44 AM

#

sure

sage raven May 24, 2023, 11:44 AM

#

sage raven The this becomes 👇

As you can see here ☝️

#

I need to put a cache buster in there

#

It's making it harder to iterate 🙂

#

Yeah, I'm getting it now. So it must be your cache.

#

#!/bin/bash
set -euo pipefail

query() {
    jq -sRn '{query: input}' | \
    curl -s \
        -u $DAGGER_SESSION_TOKEN: \
        -H "content-type:application/json" \
        -H "X-Debug:true" \
        -d@- http://127.0.0.1:$DAGGER_SESSION_PORT/query | \
    jq -r .data
}


S_APP_ID=$(query <<EOF | jq -r .setSecret.id
{
  setSecret(
        name: "APP_ID",
        plaintext: "123"
    ) {
        id
    }
}
EOF
)
echo "S_APP_ID: $S_APP_ID"


S_SOURCE_DIR=$(query <<EOF | jq -r .host.directory.id
{

        host{
            directory(path: "testscript"){
                id
            }
        }
}
EOF
)

output=$(query <<EOF | jq -r .container.from.withEnvVariable.withDirectory.withWorkdir.withSecretVariable.withExec.stdout
{
  container {
    from(address: "node:latest") {
    withEnvVariable(name: "CACHE_BUSTER", value: "$(date)") {
      withDirectory(directory: "$S_SOURCE_DIR", path: "/") {
        withWorkdir(path: "/") {
          withSecretVariable(name: "APP_ID", secret: "$S_APP_ID") {
            withExec(args: ["node", "test.js"]) {
              stdout
            }
          }
        }
      }
    }}
  }
}
EOF
)

echo $output

#

I added the "CACHE_BUSTER" env var.

#

This prevents the next steps from getting cached.

#

So, the exec is returning correctly

#

It seems to be the setup.sh that's returning the files.

#

$output may have a special meaning in the shell.

#

Change to TEST_OUT or something like that

void wing May 24, 2023, 11:57 AM

#

I copied your script but in my pc its giving me the same issue...have you just passed CACHE_BUSTER in graphql query or did anything extra in cli? I also tried changing output to TEST_OUT but the same prb

sage raven May 24, 2023, 11:58 AM

#

It's actually this:

-echo $output
+echo "$output"

#

#

console.log("Hello world")
if (process.env.APP_ID === "123") {
    console.log("You got it!")
} else {
    console.log("Oh noes!!")
}

❯ dagger run ./setup.sh
S_APP_ID: eyJuYW1lIjoiQVBQX0lEIn0=
Hello world
You got it!

void wing May 24, 2023, 12:06 PM

#

yes finally if i run dagger run ./setup.sh then it is giving me exact same output as yours...but with TUI I m not getting any console logs like yours..may be it is due to cache. I also tried with changing name of test.js file..but with _EXPERIMENTAL_DAGGER_TUI=1 dagger run ./setup.sh command it is not printing logs in cli..

sage raven May 24, 2023, 12:08 PM

#

I'm showing the interactive TUI... use dagger run -i ....

void wing May 24, 2023, 12:15 PM

#

yes got it with interactive TUI -i... and S_APP_ID is giving same secret id everytime...I am also getting "eyJuYW1lIjoiQVBQX0lEIn0=" for S_APP_ID ..not sure it is the expected behaviour or not..anyway thanks a lot for your time and help 🙂

sage raven May 24, 2023, 12:17 PM

#

Yeah, it's expected, it's based on the key, not the value 🙂

void wing May 24, 2023, 12:18 PM

#

cool!!

#

one more question.. how we should clear the cache periodically in dagger?

sage raven May 24, 2023, 12:19 PM

#

❯ echo "eyJuYW1lIjoiQVBQX0lEIn0=" | base64 -d
{"name":"APP_ID"}

sage raven May 24, 2023, 12:19 PM

#

void wing one more question.. how we should clear the cache periodically in dagger?

What do you mean? Wipe everything?

void wing May 24, 2023, 12:21 PM

#

dagger manage caching in pc right? so what is the best solution to save disk space ? can we clear cached images or files during the dagger run periodically?

sage raven May 24, 2023, 12:22 PM

#

Not selectively. You can periodically wipe the whole cache, meaning the next run will be slower but that would solve your disk space issue.

#

There's instructions for that in https://docs.dagger.io/235290/troubleshooting

void wing May 24, 2023, 12:26 PM

#

okay got it...thanks!!

void wing Jun 18, 2023, 11:08 AM

#

@sage raven hi there?

sage raven Jun 19, 2023, 7:42 AM

#

void wing <@768585883120173076> hi there?

Hey! How is it working out for you?

void wing Jun 19, 2023, 1:01 PM

#

hi there?

#Secrets in the CLI