GCP service_account.json to access secret manager api issue

[north jasper]

Hi,
I hope someone can help but it is a use-case that may not have come up before and specific to me and mine. Some context: all of the application and infrastructure secrets I need are stored in GCP Secret Manager. To access them I have a service account (.json) that is used when creating the client for interacting with the secret manager api. Because I want to run dagger in Gitlab I can only supply the serviceaccount.json file through an environment variable because I cannot push the secret file to the repository. A sample of the code for accessing secret manager is shown below:

def get_secret(project_id: str, secret_name: str, sa_json_from_env: str) -> str:
    sa_json = json.loads(sa_json_from_env)
    client = secretmanager.SecretManagerServiceClient().from_service_account_info(info=sa_json)
    name = f"projects/{project_id}/secrets/{secret_name}/versions/latest"
    response = client.access_secret_version(name=name)
    return response.payload.data.decode("UTF-8")

This code works when I run it outside of the dagger pipeline. If I export the service_account.json to the environment variable SA_JSON invoke it like secret = get_secret("my_project", "my_secret", os.environ.get("SA_JSON")) it reads the variable from the environment and creates the client with service account and then returns the secret.

When I run it inside the dagger pipeline something I do not understand happens...

..This is the dagger function I am using, note that it calls the function above:

@object_type
class Sampled:
    @function
    async def simple_check(self, token: dagger.Secret) -> str:
        sa_env_str = await token.plaintext()
        secret = get_secret("my_project", "my_secret", sa_env_str)
        return await (
            dag.container()
            .from_("alpine:latest")
            .with_env_variable("SAMPLED", secret)
            .with_exec(["env"])
            .stdout()
        )

I export the SA_JSON variable to my current terminal and then run dagger call simple-check --token=env:SA_JSON. It loads the dependencies (I can also see that the env:SA_JSON is correctly picked up if i print it) and tries to run the code but for some reason the secretmanager client does not use the serviceaccount.json I have provided. Instead it appears to ignore that entirely and invoke a fallback to a different function, specifically: secretmanager.SecretManagerServiceClient() instead of secretmanager.SecretManagerServiceClient().from_service_account_info(info=sa_json) which is what I wrote. I can see this because the error that the GCP Client SDK comes back with is:

Stderr:
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Function execution error: Your default credentials were not found. To set up │
│ Application Default Credentials, see                                         │
│ https://cloud.google.com/docs/authentication/external/set-up-adc for more    │
│ information.                                                                 │
╰──────────────────────────────────────────────────────────────────────────────╯

This error indicates that it is looking for credentials outside of what I have told it to do. Does anybody know why dagger is not loading the serviceaccount.json as environment variable into the function correctly?

[wise turret]

Hi @north jasper!! I'm not very familiar with how GCP's SDK resolves credentials. I imagine that is similar to AWS where you have multiple credentials provider that look for different config files or env variables. Could it be that in your local environment (meaning a regular terminal session) you have more env variables and/or config files that would lead GCP to resolve credentials in a way that is different to only specifying the serviceaccount json? Without testing it myself, that is the only difference that I can think of. You already confirmed that the secret and the env variable is loaded and configured correctly, so maybe we are missing some other env variables that would force the GCP sdk to load things in the same way?