Dynamically getting secrets within a module | Dagger | Page 1

river lotus Apr 16, 2025, 2:27 PM

#

Is it possible to get a secret from within a module? Or do I really need to pass 10 arguments?

#

I keep getting errors like this:

unable to lookup "op://[REDACTED]]/[REDACTED]/CLOUDFLARE_ACCOUNT_ID": Neither `OP_SERVICE_ACCOUNT_TOKEN` is set nor `op` binary is present

#

I know I can pass secrets as parameters to the function, but I have like 16 secrets that I'd rather not have to pass on every dagger call 😦

#

Or is everyone setting up a separate Makefile/Justfile/scripts to actually use dagger?

fossil heart Apr 16, 2025, 2:36 PM

#

@river lotus Dagger lacks that convenience, for now. It's because we generally prioritize strong sandboxing (which leads to a safer and more reliable platform) over short-term convenience. Then we try to add layers of convenience on top 🙂 Here's the issue for this particular convenience, it's definitely on our radar. https://github.com/dagger/dagger/issues/9584

GitHub

.env support & default secrets · Issue #9584 · dagger/dagger

Overview Dagger should natively support .env files. It's a de facto standard for managing environment-specific configuration in a lightweight, portable way. It is widely supported, included by ...

#

Also Dagger natively supports recursion. So when you call dag.Secret("op://...") from your module, it's actually correctly querying 1password, just from the context of the module's sandboxed runtime. That doesn't solve your problem, but it's useful to understand how it works under the hood and why we designed it this way.

river lotus Apr 16, 2025, 2:43 PM

#

ohh I see, thanks!
and I promise I searched before asking but between discord and multiple GH issues I was getting confused

fossil heart Apr 16, 2025, 2:43 PM

#

No worries, sorry for the confusion

river lotus Apr 16, 2025, 2:44 PM

#

Happy to know it's on the radar - I'm happy to add a Justfile in the meantime to make it easier

#

with Earthly being shutdown in a few months, I'm much more eager to migrate to dagger

fossil heart Apr 16, 2025, 2:49 PM

#

river lotus with Earthly being shutdown in a few months, I'm much more eager to migrate to d...

Did you see our joint announcement with Earthly this morning?

river lotus Apr 16, 2025, 3:02 PM

#

no I haven't!

#

just read - super cool of y'all ❤️

river lotus Apr 17, 2025, 6:54 AM

#

One thing that I found weird with lots of secrets is that it adds a lot of time to the withExec span when it seems to just be fetching secrets - it would be nice if there was a separate span when it spends 8 seconds fetching secrets

#

I also feel like dagger could be a bit more efficient/faster with secret fetching considering that all of these secrets come from a single 1Password item, but (I'm guessing) it's fetching them all individually (possibly with no concurrency?)

#

One way I found to speed it up is to use op cli myself to retreive secrets instead of dagger's 1Password integration

#

that brought the withExec span down to 400ms

fossil heart Apr 17, 2025, 12:05 PM

#

@river lotus would you mind filing an issue? I think it's either a CLI or engine bug - something is resolved serially instead of concurrently

river lotus Apr 17, 2025, 12:15 PM

#

done! https://github.com/dagger/dagger/issues/10192

GitHub

🐞1Password secrets load very slowly when using many secrets · I...

What is the issue? I have a lot of secrets using 1Password and it's causing dagger to spend 8 seconds fetching secrets before running the withExec command: I ended up switching to env secrets t...

nocturne adder Apr 17, 2025, 3:31 PM

#

river lotus One way I found to speed it up is to use op cli myself to retreive secrets inste...

do you know by any chance if the op CLI fetches the secrets concurrently?

#

in any case, that's spot on feedback. Thx for brigning it up! will fix shorty 🙏

river lotus Apr 17, 2025, 4:20 PM

#

nocturne adder do you know by any chance if the `op` CLI fetches the secrets concurrently?

It's very fast so I think so? Although all of my secrets are from a single item in 1P so it might be fetching the one item rather than re-fetching for every secret

nocturne adder Apr 17, 2025, 4:35 PM

#

river lotus It's very fast so I think so? Although all of my secrets are from a single item ...

make sense that the op CLI optimizes that 🙏

#

cc @fringe venture @tired topaz since you have context on secret providers 🙏

fringe venture Apr 18, 2025, 8:41 PM

#

@river lotus I assume this is not using OP_SERVICE_ACCOUNT_TOKEN?

There's 2 implementations for op -- when using OP_SERVICE_ACCOUNT_TOKEN it's using the 1Password API, and when not setting it it's using the op binary if present

#

the CLI invocation is meant for seamless development, but "production" usage should be using the OP_SERVICE_ACCOUNT_TOKEN

fringe venture Apr 18, 2025, 8:44 PM

#

river lotus It's very fast so I think so? Although all of my secrets are from a single item ...

@tired topaz implemented caching as well for vault -- perhaps we should do the same for op?

#

Although the tricky part is that for vault we have no choice to fetch the entire item. Whereas op gives us a way to fetch a single field within an item ...

To support caching we'd have the fetch the whole item rather than just the field (e.g. more than the user requested), which doesn't sound too good security-wise (e.g. you ask for foo/field, and we end up fetching and loading foo/* -- not great)

river lotus Apr 18, 2025, 8:47 PM

#

fringe venture <@85379843826413568> I assume this is not using `OP_SERVICE_ACCOUNT_TOKEN`? The...

I tried with both the service account token and the op binary and both were slow

fringe venture Apr 18, 2025, 8:47 PM

#

Secrets in dagger are "just in time" (they're loaded as they're accessed), so we don't know in advance which secrets the dag is going to use

fringe venture Apr 18, 2025, 8:47 PM

#

river lotus I tried with both the service account token and the op binary and both were slow

I'll give it a try locally

river lotus Apr 18, 2025, 8:49 PM

#

Out of curiosity, why wouldn’t caching work when fetching a single field?

fringe venture Apr 18, 2025, 8:50 PM

#

Oh it would. I suspect the slowness compared to using op directly is it fetches all the fields within a secret

#

you have 8 secrets, coming from the same item, different fields. Correct?

river lotus Apr 18, 2025, 8:50 PM

#

Ohh I see yeah.

river lotus Apr 18, 2025, 8:50 PM

#

fringe venture you have 8 secrets, coming from the same item, different fields. Correct?

Correct

fringe venture Apr 18, 2025, 8:50 PM

#

Ok, reproing here

river lotus Apr 18, 2025, 9:34 PM

#

I think if we could do like 10 concurrent fetches it would go pretty fast

fringe venture Apr 18, 2025, 9:38 PM

#

I've patched the engine to get some more visibility on op calls

fringe venture Apr 18, 2025, 10:08 PM

#

@mental abyss Confirmed that the they get resolved sequentially ...

   1   │ [2025-04-18 15:07:03.748368 -0700 PDT m=+2.131914793] OP SECRET PROVIDER Private/dagger test/s1
   2   │ [2025-04-18 15:07:04.640784 -0700 PDT m=+3.024454460] OP SECRET PROVIDER Private/dagger test/s2
   3   │ [2025-04-18 15:07:05.504448 -0700 PDT m=+3.888232001] OP SECRET PROVIDER Private/dagger test/s3
   4   │ [2025-04-18 15:07:06.426638 -0700 PDT m=+4.810536043] OP SECRET PROVIDER Private/dagger test/s4
   5   │ [2025-04-18 15:07:07.272362 -0700 PDT m=+5.656358876] OP SECRET PROVIDER Private/dagger test/s5
   6   │ [2025-04-18 15:07:08.203619 -0700 PDT m=+6.587719501] OP SECRET PROVIDER Private/dagger test/s6
   7   │ [2025-04-18 15:07:09.102311 -0700 PDT m=+7.486504793] OP SECRET PROVIDER Private/dagger test/s7
   8   │ [2025-04-18 15:07:10.031499 -0700 PDT m=+8.415782835] OP SECRET PROVIDER Private/dagger test/s8
   9   │ [2025-04-18 15:07:10.950931 -0700 PDT m=+9.335299960] OP SECRET PROVIDER Private/dagger test/s9
  10   │ [2025-04-18 15:07:11.916055 -0700 PDT m=+10.300507043] OP SECRET PROVIDER Private/dagger test/s10

Looking into it

#

Also they only get called once, so caching each wouldn't change anything

fringe venture Apr 18, 2025, 10:10 PM

#

fringe venture Secrets in dagger are "just in time" (they're loaded as they're accessed), so we...

I think this is what happens -- each secret is loaded just in time. We only fetch the field that was requested, and each op resolve takes a bit of time

#

Paging Dr @gritty lion -- TLDR: secret attachables get called sequencially in a blocking fashion. I think that's just the way it is with buildkit resolver?

#

Oh no wait -- maybe it's our own mutex

gritty lion Apr 18, 2025, 10:15 PM

#

We're using an RLock here, so theoretically they shouldn't be blocking each other there, unless something else is holding a write lock

fringe venture Apr 18, 2025, 10:15 PM

#

yep yep got it

#

Oh actually no -- it's a rlock, right

gritty lion Apr 18, 2025, 10:17 PM

#

Yeah it looks like buildkit is doing it in serial here https://github.com/sipsma/buildkit/blob/03396bcf635a07d4e0ad7b50dab1a28d71f8f132/solver/llbsolver/ops/exec.go#L554-L562

fringe venture Apr 18, 2025, 10:17 PM

#

we only lock in AddSecret

fringe venture Apr 18, 2025, 10:19 PM

#

gritty lion Yeah it looks like buildkit is doing it in serial here https://github.com/sipsma...

thanks!

#

@gritty lion Follow up to that ... any ill advised reason not to use an errgroup and do that in parallel? and what's the latest with buildkit patches, do we still run a fork?

gritty lion Apr 18, 2025, 10:21 PM

#

fringe venture <@949034677610643507> Follow up to that ... any ill advised reason not to use an...

No looks like switching to errgroup should be an easy fix. Can just open a PR in our fork here: https://github.com/dagger/buildkit

Once we get to porting WithExec dagger native code it'll all be in-house, but for now we're just patching as needed there

fringe venture Apr 18, 2025, 10:21 PM

#

👍

#

@gritty lion is our fork the master branch?

gritty lion Apr 18, 2025, 10:27 PM

#

fringe venture <@949034677610643507> is our fork the master branch?

yeah we merge our changes to master there

#

and just cherry-pick from upstream buildkit as needed

fringe venture Apr 18, 2025, 10:35 PM

#

@gritty lion https://github.com/dagger/buildkit/pull/5

#

Not really tested yet

gritty lion Apr 18, 2025, 10:39 PM

#

fringe venture Not really tested yet

LGTM, if you want to test locally against dagger before merging you can just update the go.mod replace for github.com/moby/buildkit to point to your commit

fringe venture Apr 18, 2025, 10:41 PM

#

would it work to point to to the local FS with the PR checked out?

bk versions are a little intimidating, it's github.com/moby/buildkit => github.com/dagger/buildkit v0.0.0-20250415011403-6f5903130a80 now

#

oh no local it won't work because of the sandbox

#

i'll try go get github.com/dagger/buildkit@<commit>

nocturne adder Apr 18, 2025, 10:43 PM

#

fringe venture i'll try `go get github.com/dagger/buildkit@<commit>`

that's what I do 😛

#

there's even an unglier hack to use local replaces but you probably don't even want to heart about that

fringe venture Apr 18, 2025, 10:44 PM

#

i'll re-try with moby/buildkit rather than dagger/buildkit

#

dagger/buildkit didn't work

#

go: downloading github.com/dagger/buildkit v0.0.0-20250418223454-60c160d09b68
go: github.com/dagger/buildkit@60c160d09b6878d933677a671176bc61c5fd893f (v0.0.0-20250418223454-60c160d09b68) requires github.com/dagger/buildkit@v0.0.0-20250418223454-60c160d09b68: parsing go.mod:
        module declares its path as: github.com/moby/buildkit
                but was required as: github.com/dagger/buildkit

nocturne adder Apr 18, 2025, 10:45 PM

#

@fringe venture try adding this in the replace github.com/dagger/buildkit v0.0.0-20250418223454-60c160d09b68 and then go mod tidy && go build ./cmd/engine?

fringe venture Apr 18, 2025, 10:46 PM

#

the second command failed because i'm on mac ... trying a ./hack/dev to see if it works

nocturne adder Apr 18, 2025, 10:47 PM

#

fringe venture the second command failed because i'm on mac ... trying a ./hack/dev to see if i...

building worked locally on linux

#

so ./hack/dev/should

#

🐧 ❤️

fringe venture Apr 18, 2025, 10:49 PM

#

fringe venture <@418233653592719364> Confirmed that the they get resolved sequentially ... ```...

  14   │ [2025-04-18 15:48:41.179835 -0700 PDT m=+3.581065667] OP SECRET PROVIDER Private/dagger test/s10
  15   │ [2025-04-18 15:48:41.180266 -0700 PDT m=+3.581496084] OP SECRET PROVIDER Private/dagger test/s1
  16   │ [2025-04-18 15:48:41.180318 -0700 PDT m=+3.581548584] OP SECRET PROVIDER Private/dagger test/s6
  17   │ [2025-04-18 15:48:41.180373 -0700 PDT m=+3.581603709] OP SECRET PROVIDER Private/dagger test/s3
  18   │ [2025-04-18 15:48:41.180443 -0700 PDT m=+3.581673501] OP SECRET PROVIDER Private/dagger test/s5
  19   │ [2025-04-18 15:48:41.180498 -0700 PDT m=+3.581728917] OP SECRET PROVIDER Private/dagger test/s2
  20   │ [2025-04-18 15:48:41.180542 -0700 PDT m=+3.581772917] OP SECRET PROVIDER Private/dagger test/s7
  21   │ [2025-04-18 15:48:41.180609 -0700 PDT m=+3.581839084] OP SECRET PROVIDER Private/dagger test/s4
  22   │ [2025-04-18 15:48:41.180901 -0700 PDT m=+3.582131167] OP SECRET PROVIDER Private/dagger test/s8
  23   │ [2025-04-18 15:48:41.181006 -0700 PDT m=+3.582236126] OP SECRET PROVIDER Private/dagger test/s9

They're all in parallel now 🎉

#

#

and the secrets are still there

#

@gritty lion Ok to merge?

fringe venture Apr 18, 2025, 10:57 PM

#

nocturne adder <@707661669819613324> try adding this in the replace `github.com/dagger/buildki...

Now should I update dagger doing the same thing or is there a different procedure

#

?

nocturne adder Apr 18, 2025, 10:58 PM

#

fringe venture Now should I update dagger doing the same thing or is there a different procedur...

yes, you need to send a PR to dagger/dagger with that replace bump

#

that's it

fringe venture Apr 18, 2025, 11:01 PM

#

@nocturne adder how did you come up with the first part (2025041822345)?

#

oh, got it -- did a go get, it failed but it spat out the version in the error messages

nocturne adder Apr 18, 2025, 11:02 PM

#

fringe venture oh, got it -- did a go get, it failed but it spat out the version in the error m...

yep

fringe venture Apr 18, 2025, 11:05 PM

#

https://github.com/dagger/dagger/pull/10211/files

#

🙏

#

@river lotus the fix will be out on the next release

#Dynamically getting secrets within a module