Ah, that's why this doesn't work 😢 | Dagger | Page 1

fast parrot Sep 16, 2024, 5:53 PM

#

what's the example not working?

raw nimbus Sep 16, 2024, 5:54 PM

#

I have a sops module that takes a directory and can load secrets from sops

#

I then have a project config module that passes sops the directory and gets the secrets

#

then each service has its own dagger module that uses the project config

#

Can I screen share? Might be easier

fast parrot Sep 16, 2024, 5:57 PM

#

Would it be possible to send screenshots/code-snippets async? Otherwise it's gonna get lost in an ephemeral screen share 😄

raw nimbus Sep 16, 2024, 6:00 PM

#

OK. So the sops module exists and takes a directory and a privateKey secret. From this, it can load a sops.yaml.

Because we can't return arbitrary objects (GraphQL reasons), I need to return Secret[].

This presented a problem, because .find doesn't support async. Now I could use forEach in every calling module to filter, but I decided this would be better to handle within the sops module itself and to make lives easier for callers.

As such, I tried returning Self and providing a requireSecret function

#

Now, the problem comes when I add another module in the middle. See, I have many services within a single project that share secrets. So I created a Config module that provides common values, but now it needs to delegate secrets

#

    @func()
    async getSecrets(
        @argument({ defaultPath: "." }) directory: Directory): Promise<Sops> {
    }
}

#

Now this doesn't work, because modules can't return types that aren't their own or within Dagger Core

#

So instead, I switched the Config module to return a Directory

#

    @func()
    async secretsDirectory(
        @argument({ defaultPath: "." }) directory: Directory): Promise<Directory> {
        return directory;
    }
}

#

so then my service module trys to use Sops with this directory and that's where it seems to lose the secret store

#

Sharing the context because this isn't the way I wanted this code to work, but it's the only way that it can work with things atm

#

the error is:

┃ Error: resolve: failed to add client resources from ID: failed to add secret from source client z3qg2l1s7bi8n4xlr9835xb95: secret xxh3:d703e97eab42eef9 not found in other store
  ✔ upload /home/rawkode/Code/src/github.com/RawkodeAcademy/RawkodeAcademy/projects/rawkode.academy/technology-service from ju29pc0lsn6qibewl9lpcibzw (client id: uhs5a7zlxd2vm5536hslk6rhl, sessi
on id: 88bkzu4md6vov5g6ccrgspnu3) 0.6s

harsh bridge Sep 16, 2024, 6:11 PM

#

Don't you have to set dag.setSecret somewhere in the chain to be able to retrieve it back? Maybe that's missing?

raw nimbus Sep 16, 2024, 6:13 PM

#

Sigh

#

secrets.push(new SopsSecret(newKey, dag.secret(value)));

#

You're right

#

I'm using the wrong function!

harsh bridge Sep 16, 2024, 6:14 PM

#

ah

raw nimbus Sep 16, 2024, 6:16 PM

#

I think it's time for the pub

harsh bridge Sep 16, 2024, 6:17 PM

#

🍻

fast parrot Sep 16, 2024, 6:21 PM

#

Thank you @harsh bridge! I was reviewing a PR that I thought at first glance might fix a bug that could have been causing this (https://github.com/dagger/dagger/pull/8358), but came back to see it was simpler this time, good eye 🙂

And no worries @raw nimbus, setSecret and its side-effectful nature is inherently confusing and leads to these sort of situations. In the long term we still want to add support for something like a "secret provider" as a first-class citizen that should hopefully make this all easier to think about and manage

raw nimbus Sep 16, 2024, 6:34 PM

#

Damn, sadly I think it is the engine

#

! call function "deployProduction": process "bun /src/projects/rawkode.academy/technology-service/dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1
┃ Error: resolve: failed to add client resources from ID: failed to add secret from source client ilxpy3ughbrjchv7ashadj5bc: secret xxh3:73a624ab9a29f93f not found in other store
  ✔ upload /home/rawkode/Code/src/github.com/RawkodeAcademy/RawkodeAcademy/projects/rawkode.academy/technology-service from ju29pc0lsn6qibewl9lpcibzw (client id: i04bx5s7uqr7zu1dqokrkjum1, sessi
on id: xha6ooys4ugp69snjikx27t16) 0.6s

#

This is after using setSecret

harsh bridge Sep 16, 2024, 6:46 PM

#

You retrieve the secret with their respective name? Somewhere you also have to do an await secret.name(). secret is of type dagger.Secret

#

I spent hours doing this wrong, thinking I could trick dagger 😑

raw nimbus Sep 16, 2024, 6:49 PM

#

Your'e right, I'm over complicating this

#

I don't need to store the Secret objects and pass them around

#

I can just request them.

#

Lets just forget the past 5 hours of my life please

#

It's not worth it 😂

harsh bridge Sep 16, 2024, 6:50 PM

#

Or you could just take

@object()
class Secrets {
    @func()
    secrets: Secret[];

    constructor(secrets: Secret[]) {
        this.secrets = secrets;
    }

    @func()
    async getSecret(name: string): Promise<Secret | undefined> {
        for (const secret of this.secrets) {
            const secretName = await secret.name();

            if (secretName === name) {
                return secret;
            }
        }

        return undefined;
    }
}

#

🤣

#

The full code of my function

import { object, func, type Secret, dag } from "@dagger.io/dagger";
import { InfisicalClient } from "@infisical/sdk";

@object()
class Secrets {
        @func()
        secrets: Secret[];

        constructor(secrets: Secret[]) {
                this.secrets = secrets;
        }

        @func()
        async getSecret(name: string): Promise<Secret | undefined> {
                for (const secret of this.secrets) {
                        const secretName = await secret.name();

                        if (secretName === name) {
                                return secret;
                        }
                }

                return undefined;
        }
}

@object()
class Infisical {
        @func()
        async getSecrets(
                clientId: Secret,
                clientSecret: Secret,
                environment: string,
                projectId: string,
        ): Promise<Secrets> {
                const client = new InfisicalClient({
                        auth: {
                                universalAuth: {
                                        clientId: await clientId.plaintext(),
                                        clientSecret: await clientSecret.plaintext(),
                                },
                        },
                });

                const infisicalSecrets = await client.listSecrets({
                        environment,
                        projectId,
                });

                const secrets = infisicalSecrets.map((secret) => {
                        return dag.setSecret(secret.secretKey, secret.secretValue);
                });

                console.log(`Found ${secrets.length} secrets for the env '${environment}'`);

                return new Secrets(secrets);
        }
}

raw nimbus Sep 16, 2024, 7:05 PM

#

Even you're doing too much work I think

#

I'll share some code with you in 10

#

on a call

harsh bridge Sep 16, 2024, 7:55 PM

#

I'm back in 5

#

the only thing I took away was: "for every dag.setSecret, I'll have to do a secret.name() to retrieve the original name"

raw nimbus Sep 16, 2024, 8:03 PM

#

What I'm wondering is if we need to keep references to the Secrets, can't we just set them then request them anywhere else?

#

Driving home. Will check shortly

harsh bridge Sep 16, 2024, 8:06 PM

#

With the "inconvenience" @fast parrot mentioned, I think we need to keep them as a mapping from the function to the dagger world and vice versa :/

#

Or in other words. It makes it more convenient to keep them for mapping reasons

#

But I'm happy to get rid of unnecessary code 🙂

harsh bridge Sep 16, 2024, 9:08 PM

#

in case you want to have a quick call, I'm just taking the dog for a quick walk

raw nimbus Sep 16, 2024, 10:06 PM

#

@fast parrot Why is it OK for my @func() to call an external function, namely:

convertJsonToSecrets = (obj: any, prefix = ""): void => {
        for (const key in obj) {
            if (obj.hasOwnProperty(key)) {
                const value = obj[key];
                const newKey = prefix ? `${prefix}.${key}` : key;

                if (typeof value === "string") {
                    console.log(`Setting secret: ${newKey}`);
                    secrets[newKey] = dag.setSecret(key, value);
                    console.log(`Now got secrets: ${JSON.stringify(this.secrets)}`);
                } else if (typeof value === "object" && !Array.isArray(value)) {
                    convertJsonToSecrets(value, newKey);
                } else {
                    throw new Error(`Unsupported value type: ${typeof value}`);
                }
            }
        }
    };

but if I embed this within the @object() without a @func() it breaks?

Does every function in the class need to be GraphQL compatible, even if not @func declaration?

fast parrot Sep 16, 2024, 10:07 PM

#

@flat scarab does that sound expected to you ^ ?

raw nimbus Sep 16, 2024, 10:07 PM

#

I'm also adding my secrets to a class property and those aren't being persisted within the engine when I return this and chain a function afterwards

#

I might try and build this in Python and see if its the same

flat scarab Sep 17, 2024, 6:25 AM

#

raw nimbus <@949034677610643507> Why is it OK for my `@func()` to call an external function...

It might be considered as a private property of your object in that case, meaning it will still be serialized but since it’s a function, it’s gonna fail. You should format it as a func instead of a variable (I could also just ignore func inside the state, please open an issue for that)

raw nimbus Sep 17, 2024, 6:39 AM

#

Ok. That helps, I think I understand how this works a bit better.

#

I'll fix it then I'll open an issue

flat scarab Sep 17, 2024, 8:20 AM

#

Sure, there's still a lot to improve on the TS SDK, specially because it's much more syntax free than Go, but I have so much to do already haha.
If you wanna give me some help though and contribute, I can show you where you can fix that problem

raw nimbus Sep 17, 2024, 1:35 PM

#

I'll at a conference this week, but if you can give me a tour of the SDK and how it's built next week; I'm more than happy to help you out

#Ah, that's why this doesn't work 😢