Any form of print output from my module

[charred sail]

hey! would you be able to specify this a bit more? for example, do you literally just want printlns directly from your module functions? or would you also want e.g. the logs from withExec for containers that your function runs?

also: have you tried --progress=dots? it's a much much quieter alternative to plain that focuses on log output, and just prints dots as spans complete so you can tell it's not stuck

[elfin hawk]

Just print output, nothing else. One issue we've had to work around is AWS ECS passwords in stdout and we'd like to prevent that as well.

I have tried dots but I don't recall if it still shows my print outputs?

[charred sail]

dots will show print output, yes - it's literally just log output + dots

The challenge I see for "just print output" is: what do you do when you DO want to see the output of a withExec? Or do you literally only care if it fails? (valid)

For example if I wrote a function that calls a ./hack/generate script that I'm familiar with in my repo, I'd kind of expect to see that output in CI, especially because most of my functions are just calling scripts, and not printing anything on their own, so that would make most of my CI logs empty. I believe you if you say you don't want that output, but it doesn't feel like a sane default to me; seems like we'd need a way to opt-in or detect that case.

Also: are you using Dagger Cloud? One reason this is important is there's likely large overlap; if you care about log output in e.g. GitLab CI, I presume you're not using Cloud's UI, which is where you'd be able to go to find logs if we hide them in the CLI output

cc @dusk walrus in case you have any thoughts on this 🙏

[elfin hawk]

Your first point is what we're after. We'll provide the log output we need, unless something fails in which case show the error

I'll try dots, I was under the impression it still showed quite a lot of Dagger output, will test again

[charred sail]

dots will still show withExec output, yes; it's basically 'show all the logs from all the spans you would normally show me'

[elfin hawk]

No Dagger Cloud yet. The feedback we've had is that the logs are hard to read for engineers that are not using Dagger but just using a templated pipeline (which calls a Dagger module). They want the output from the module, but struggle to find it in the Gitlab CI output (which is very verbose, it's the same as --progress=plain).

--progress=dots is better in Gitlab, I'll need to edit the module and see exactly what this will output before suggesting we switch over, I'll report back shortly.

[charred sail]

there's also dagger --silent which will print literally nothing but the return value of the function

[elfin hawk]

dots is showing exec output. Before I go editing code, it'll also show fmt.Print output?

[dusk walrus]

--progress=dots does show the prints

[elfin hawk]

Great, will test that now

Yeah that's much better. Wasn't aware dots would keep our output, that's good. All that's left is to be able to hide stdout from WithExec. We use a scrappy workaround:

func (m AwsCli) ExecWithRedirectStdout(
    ctx context.Context,
    // Command to run (without "aws") (e.g., ["sts", "get-caller-identity"]).
    args []string,
) (string, error) {
    redirectStdout := "stdout.txt"

    args = append([]string{"aws"}, args...)
    args = append(args, ">", redirectStdout)

    ctr := m.Container.
        WithEnvVariable("CACHE_BUSTER", time.Now().Format(time.RFC3339Nano)).
        WithExec([]string{"sh", "-c", strings.Join(args, " ")})

    output, err := ctr.File(redirectStdout).Contents(ctx)
    if err != nil {
        return "", err
    }
    return output, nil
}

for this command: args := []string{"ecr", "get-login-password"} else the password is plaintext printed into Gitlab logs

[charred sail]

well - there's a side topic there, which is that those secrets will end up in your engine cache in plaintext. maybe this would help in the future? https://github.com/dagger/dagger/pull/11728

[charred sail]

plus this option, depending on your use case that might be what you're really after - it won't show print output, but it will print a string return value from the function

[elfin hawk]

That's not so much a problem, access to the engine is by request and the password is for temporary credentials that expire quickly (but not immediately, so having it in plaintext logs isn't good)

That secrets manager would be useful in general for sure, but not sure it's useful here because each Gitlab job creates temporary credentials for a role and then creates an ECR password to push images - there's no long-term secret to store.

Just glanced through that MR, we'll almost certainly be able to use that elsewhere 👍

@charred sail Quick one, are these aws-sm aws-ps supported in .env files?

[charred sail]

.env files support secret addresses, and that PR adds a new aws:// scheme for them, so that should work yeah (cc @spiral mirage @unborn ember to confirm, not sure if there are multiple schemes)

[unborn ember]

yep, that's correct. This will be supported in .env files once that release is out. And yes, we added two aws schemes. aws+sm for SecretManager secrets and aws+ps for ParameterStore values

[elfin hawk]

Great, happy to see that added

[pale pelican]

cc @spiral mirage @unborn ember do we want that same capability in Dagger? 😁

[spiral mirage]

its a common enough requirement that it makes sense to provide convenience for it, just a question of how. It could be a special URI like aws://ecr-login-password and we handle the aws ecr get-login-password flow in the client secret manager to populate it

[unborn ember]

usually these kind of short-term credentials come in the form of OIDC federation integration. Is that your case @elfin hawk? If so, we're already thinking about an OIDC provider to help with that

[elfin hawk]

Yep. Runner workers get an oidc token that can be exchanged for credentials, and it's done automatically for users if a certain variable is provided

[unborn ember]

perfect. We do have this mapped and we're already thinking of having a similar secretprovider integration. Stay tuned ™

[unborn ember]

SGTM given that this is a very common use case and we still don't have proper support of safe secreat creation from a command (https://github.com/dagger/dagger/issues/10376)

[spiral mirage]

yup, even if we had that, the flow would still look something like: use oidc to generate a temporary key/secret pair with the assumed role (unless the function execution magically gets oidc too?), pass that into the function, in the function use the aws cli to generate an ecr token, turn that into a secret.

Some sort of convenience integration built into the aws secret provider feels like it fits given that every ECR user will need that flow.

[unborn ember]

use oidc to generate a temporary key/secret pair with the assumed role (unless the function execution magically gets oidc too?), pass that into the function,

I was assuming that in the case of Native-CI, this could be part of the pipeline function as well, you just pass the OIDC token and then the function takes care of assuming the right AWS role. AFAIK that's all available in the aws CLI as well which you can invoke within your function