CA certs are not utilized / go.work version mismatch | Dagger | Page 1

primal agate Dec 24, 2024, 3:15 PM

#

I found our previous conversation. We didn't create a thread, hence it went missing. Also, side-note, searching in Discord isn't that fun.

#

#daggernauts message

#

I did some more testing. There are 2 parts with two related issues.

Since the tooling is bundled in the engine, the only way for me to get 0.15.x working is to remove my local go.work. If I have any version of go.work in my local, I get TLS cert errors (which is the second issue). Dagger module development in this state is very painful and not sustainable. I'll have to go back to 0.14.0 and even then, make sure the toolchain version in go.work matches the engine go version. I also have to ask all my consumers to match versions (yikes!).
My CA certificates inside the Dagger engine are not recognized by the engine. If the engine has to do any kind of Go toolchain pull (based on my local go.work), it fails. If my toolchain version doesn't match, it tries to download the respective toolchain and fails. If the toolchain version matches, it still tries to pull some grpc and otlp modules and that also fails.

No. 2 seems like a bug and an easy one to fix. However, No. 1 seems like a symptom of how Go modules and workspaces work, IDK how that can be remedied. If No. 2 is fixed, at least 1 is tolerable. Might even render it a non-issue.

I may be way off on my findings 😄 I'm a bit rusty still after getting back from a long vacation. Happy to help test any of this of course!

primal agate Dec 24, 2024, 5:56 PM

#

CA certs are not utilized / go.work version mismatch

jolly agate Dec 24, 2024, 7:03 PM

#

Hey @primal agate ! I've started doing some troubleshooting a while back but was never able to pin point the issues as I got sidetracked.

I'm still a bit unsure how go.work places in all this problem. Seems like pulling the go runtime (which should happen within the Go SDK runtime in the engine) is not being able to use the certificates correctly

#

I'm off this week but I can take a look with a high priority on Monday 👐

primal agate Dec 24, 2024, 7:57 PM

#

Thanks for getting back to me during holidays @jolly agate ! I wanted to document what I'm seeing before I forgot but this can wait till you are back 🙂

There is definitely a correlation (intended or not) between the go version in the go.work and the version inside the engine. Edits to my go.work affect how the engine pulls deps.

Re: certs. It kinda looks like it's the engine itself that doesn't have access to the CA certs. But containers spawned within the engine, do. At least that's what I'm seeing with my testing.

Merry Christmas! 🙂

jolly agate Dec 24, 2024, 8:04 PM

#

primal agate Thanks for getting back to me during holidays <@336241811179962368> ! I wanted t...

Merry Christmas to you too. I'll give this the highest priority on Monday

jolly agate Dec 30, 2024, 4:30 PM

#

👋 @primal agate! I'll be working on this today/tomorrow 🙏

primal agate Dec 30, 2024, 5:23 PM

#

awesome! Thank you for the update 🙂

jolly agate Dec 31, 2024, 8:25 PM

#

@primal agate once you're back can you help me verify something really quick?
can you check your enegine logs if before dagger calls the codegen step, the update-ca-certificates is efectively called?

This is what I'm expecting to see in the engine logs:

time="2024-12-31T20:16:38Z" level=debug msg="starting container" logrus_error="can not add field \"call_id\"" args="[update-ca-certificates]" caller_client_id=9nzt9wpwt7i9psdgnn9j3t2nc id=plx
l6b0ct4yrw9g6k49eyx5xb span="codegen --output /src --module-context-path /src --module-name lala --introspection-json-path /schema.json" spanID=e282d4da9328cac1 traceID=1d9a268a816d93421d3ff6
59a4eab7fd
time="2024-12-31T20:16:39Z" level=debug msg="container done" logrus_error="can not add field \"call_id\"" args="[update-ca-certificates]" caller_client_id=9nzt9wpwt7i9psdgnn9j3t2nc error="<ni
l>" id=plxl6b0ct4yrw9g6k49eyx5xb span="codegen --output /src --module-context-path /src --module-name lala --introspection-json-path /schema.json" spanID=e282d4da9328cac1 traceID=1d9a268a816d
93421d3ff659a4eab7fd
time="2024-12-31T20:16:39Z" level=debug msg="starting container" logrus_error="can not add field \"call_id\"" args="[/.init codegen --output /src --module-context-path /src --module-name lala
 --introspection-json-path /schema.json]" caller_client_id=9nzt9wpwt7i9psdgnn9j3t2nc id=9zsjw9ihrjebbux667dfoc27v span="codegen --output /src --module-context-path /src --module-name lala --i
ntrospection-json-path /schema.json" spanID=e282d4da9328cac1 traceID=1d9a268a816d93421d3ff659a4eab7fd

This basically shows that before the last starting container step, an update-ca-certificates pre-step ran which is the one in charge of setting up the certs before each container execution

primal agate Dec 31, 2024, 8:32 PM

#

Sure! I am about to head out but I will check when I get back. When you say engine logs, do you mean TUI output? Or the docker logs?

jolly agate Dec 31, 2024, 8:52 PM

#

primal agate Sure! I am about to head out but I will check when I get back. When you say engi...

@primal agate a follow-up test I'd love you to do is to create this very simple function:

// Returns a container that echoes whatever string argument is provided
func (m *Lala) Test() *dagger.Container {
    return dag.CurrentModule().Source().AsModule().Runtime().Terminal()
}

and once you get in that container terminal, to verify if your certificates are set correctly there. You should be able to to check this by going to /etc/ssl/certs and you should see that if your custom certs are there

jolly agate Dec 31, 2024, 8:58 PM

#

primal agate Sure! I am about to head out but I will check when I get back. When you say engi...

I'm refering to the docker logs in this case

primal agate Jan 1, 2025, 1:01 AM

#

Finally got some time to test.

can you check your enegine logs if before dagger calls the codegen step, the update-ca-certificates is efectively called?

I don't see update-ca-certificates anywhere in docker logs

a follow-up test I'd love you to do is to create this very simple function:

Looks like my ca-certificates.crt is in /etc/ssl/certs directory.

#

but... I think i see what the problem is. I am directly copying the cert bundle (in crt) to /etc/ssl/certs instead of copying the root chain to /usr/local/share/ca-certificates/ and letting dagger bundle it. That seemed to work well for me when I created a docker image via dagger and copied it to another container but dagger doesn't know the existence of it to copy it deeper into the stack. Hmm.. let me try copying my root cert chain and see how it works. I remember this is the way I did at first and it didn't work for some reason. Hence I reverted to copying the bundled crt.

jolly agate Jan 1, 2025, 2:34 AM

#

primal agate but... I think i see what the problem is. I am directly copying the cert bundle ...

Ohh I see. Yeah.. this is very likely to be the case

#

Because the engine will pass the certificates to the underlying containers by automatically bundling what's in /usr/local/share/certificates

primal agate Jan 1, 2025, 7:07 PM

#

Hey @jolly agate , Happy New Year! I got further. But still running into the same issue. After adding the the ca chain to /usr/local/share/ca-certificates the engine installs the certs. I can run the test function you gave above and in the terminal I can curl to one of our internal sites. I couldn't do this when the certs weren't set up properly. What's strange is, I can also modify the /src go.mod and update the toolchain from go1.23.2 to go1.23.4 and do a go mod tidy and it pulls the new toolchain just fine.

The issue I'm running into is when my go.work go version (1.23.4) doesn't match the engine runtime go version (1.23.2). Here's the command the engine is trying to run (module name obfuscated)

Container.withExec(args: ["codegen", "--output", "/src", "--module-context-path", "/src/mydaggermodule", "--module-name", "MyDaggerModule", "--introspection-json-path", "/schema.json"])

Then the error

running post-command: go mod tidy
go: downloading go1.23.4 (linux/arm64)
go: download go1.23.4: golang.org/toolchain@v0.0.1-go1.23.4.linux-arm64: Get "https://storage.googleapis.com/proxy-
rod/9967e59c28d82157-golang.org:toolchain-v0.0.1-go1.23.4.linux-arm64.zip?Expires=1735843846&GoogleAccessId=gcs-url
40golang-modproxy.iam.gserviceaccount.com&Signature=enW%2B2TAkBRs74TSipXbvF6YLSR85KL9lafnR9s8Lkcq83NZ6yE32i7CPAQ7xi
MY%2BGI6r8UHfIPDBBH1ROsGkfsY4grpwvhwzNQKJJbiDU51bS9Fc7SArb8gc6CJdIQp5Y66iVtuw801EZlnLuWyzP1PU5EqEJzAgotBuy6D2hfNjoV
B%2FVPTti5DWeUIRWeMxxavIbG3FjzA1WSo4LhILhpus20KSVALXWNgmEOK7pY7tKL0ZoPGFt9Nu03PH5aTuwLUeK8BO7DrdJMQjjJWrYcRjSNMCxWe
BkU%2F037z1ZBH4KMP2rP4Q%3D%3D": tls: failed to verify certificate: x509: certificate signed by unknown authority
post-command failed: exit status 1

The question is, where is that codegen command running if not in the runtime container and why does it not have access to the certs?

jolly agate Jan 1, 2025, 8:57 PM

#

primal agate Hey <@336241811179962368> , Happy New Year! I got further. But still running int...

@primal agate do you now see the update-ca-certificates log in the engine when that codegen step runs?

#

You should see a similar log output to the one I've posted above

primal agate Jan 1, 2025, 11:51 PM

#

hmm, i am not seeing update-ca-certificates in the docker logs

jolly agate Jan 2, 2025, 12:44 AM

#

primal agate hmm, i am not seeing `update-ca-certificates` in the `docker logs`

did you disable the debug logs in the engine config by any chance? They're enabled by default and the logs I've posted above get printed in debug level.

Also, is your certificate chain in .crt extension? I have the impression that for some reason .pem extensions not always worked.

primal agate Jan 2, 2025, 1:06 AM

#

I did not disable debug. And yes, my cert is .crt.

jolly agate Jan 2, 2025, 1:23 AM

#

primal agate I did not disable debug. And yes, my cert is .crt.

hmm.. that's super strange. let's get together tomorrow to see if I spot anything

#

It should be running the cert provisioning before the codegen step

jolly agate Jan 2, 2025, 1:55 PM

#

@primal agate LMK if you're around to check this out today 🙏

primal agate Jan 2, 2025, 3:01 PM

#

hey @jolly agate , I just logged in at work

jolly agate Jan 2, 2025, 3:15 PM

#

ok, we can try async. No worries

#

so last thing you tried is using the crt in the /usr/local/share/certiifcates but you don't see any update-ca-certificates log in the engine log, correct?

#CA certs are not utilized / go.work version mismatch