Go SDK fails to pull image from private registry due to certificate signed by unknown authority | Dagger | Page 1

spring badge Nov 30, 2022, 9:18 AM

#

Hi,
I'm trying to port one of my dockerfiles to Go project using Dagger Go SDK but I'm having some issues with pulling the image from private registry.

Setup:

go version go1.19.3 linux/amd64
Rancher Desktop Version: 1.6.1
- using RD with Moby Docker Version: 20.10.17-rd
private registry has self-signed CA that is trusted in the OS

What does work:

docker login to the private registry and docker pull images from the private registry
DOCKER_BUILDKIT=1 docker build and docker push to the private registry

What does not work:

using dagger client in Go SDK to pull image from the private registry with client.Container().From() where I specify the private image path

I've started the discussion here: #1043235579241963621 message
Tried suggestion to log in the code to the private registry, it logs in but doesn't affect anything: #1031785076029128784 message

#

Source code

package main

import (
    "bytes"
    "context"
    "fmt"
    "os"
    "os/exec"

    "dagger.io/dagger"
    "github.com/pkg/errors"
)

func main() {
    fmt.Println("Building with Dagger")

    if err := dockerLogin(); err != nil {
        panic(err)
    }

    ctx := context.Background()
    if err := build(ctx); err != nil {
        panic(err)
    }

}

func dockerLogin() error {
    var out bytes.Buffer

    cmd := exec.Command("docker", "login", "--password", "pw", "--username", "me", "oci.git.company.com")
    // cmd := exec.Command("docker", "login", "-h")
    cmd.Stdout = &out

    if err := cmd.Run(); err != nil {
        return errors.Wrap(err, "Could not get the docker login")
    }

    fmt.Println(out.String())

    return nil
}

func build(ctx context.Context) error {
    // initialize Dagger client

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        return err
    }
    defer client.Close()

    // get reference to the local project
    project := client.Host().Directory("/home/company.net/me/Projects/company/grpit/company.connector2")

    // builder := client.Container().
    client.Container().
        From("oci.git.company.com/cz/it/devops-templates/dotnet/sdk:5.0").
        WithMountedDirectory("/src", project).
        WithWorkdir("/src").
        WithExec([]string{
            "dotnet",
            "publish",
            "Company.Connector2/Company.Connector2.csproj",
            "-c", "release",
            "-o", "/output"})


    return nil
}

#

Output

Building with Dagger
Login Succeeded

#1 resolve image config for oci.git.company.com/cz/it/devops-templates/dotnet/sdk:5.0
#1 ERROR: failed to do request: Head "https://oci.git.company.com/v2/cz/it/devops-templates/dotnet/sdk/manifests/5.0": x509: certificate signed by unknown authority
------
 > resolve image config for oci.git.company.com/cz/it/devops-templates/dotnet/sdk:5.0:
------
panic: input:1: container.from failed to do request: Head "https://oci.git.company.com/v2/cz/it/devops-templates/dotnet/sdk/manifests/5.0": x509: certificate signed by unknown authority

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
    /home/company.net/me/Projects/kube/dagger/main.go:23 +0x9a
exit status 2

peak plinth Nov 30, 2022, 12:18 PM

#

Hi, do you have a special config inside your docker daemon to accept self-signed certificates ?

spring badge Nov 30, 2022, 12:29 PM

#

Hi, no I don't. I've been searching info if this could be somehow caused on the Rancher Desktop side and they say that Lima VM uses same certs that host does. Which makes sense because the docker pull command works as intended

peak plinth Nov 30, 2022, 12:32 PM

#

Ok, we used to have this docs: https://docs.dagger.io/1224/self-signed-certificates/. Basically, when using the docker cli and the BUILDKIT env var. It was for previous versions of Dagger

#

Now, we embed Builkdit (to avoid complexity such as this one in the future). I'll need to dig a bit

spring badge Nov 30, 2022, 12:38 PM

#

Thank you very much. This PR is related I think https://github.com/dagger/dagger/pull/3943

peak plinth Nov 30, 2022, 12:41 PM

#

spring badge Thank you very much. This PR is related I think https://github.com/dagger/dagger...

Yes, perfect. Your best bet, short term, is to rely on that PR until we agree on a proper implementation (the one suggested by Erik). It's a good short-term fix. We will come back with a maintainable/scalable solution asap. Thanks for reminding the priority of this PR 🙏

spring badge Nov 30, 2022, 12:45 PM

#

Thank You for caring!

peak plinth Nov 30, 2022, 12:45 PM

#

Just checked the source code, relying on the env var, as shown in the example, should work

#

So, you need to :

clone our repo
Checkout to this PR
Set env var from PR, with proper sockets -> (example on PR) DAGGER_ENGINE_BIND_MOUNTS=/foo/bar.txt:/foo/bar.txt,/fiz/buzz.txt
Create local dev engine, including changes

$ ./hack/dev

Set env vars to redirect all SDK calls to dev engine

$ export DAGGER_RUNNER_HOST=docker-container://test-dagger-engine # To check, if same with Rancher
$ export DAGGER_HOST=docker-container://test-dagger-engine

#

Keep us in touch 😇

spring badge Nov 30, 2022, 12:55 PM

#

Will do! I'm gonna try that today and let you know. Thank you for the detailed explanation

peak plinth Nov 30, 2022, 12:57 PM

#

I hope I'm correct, if not I'll try to repro locally to unlock you 😇

misty tulip Nov 30, 2022, 1:34 PM

#

👋 I think doing what Erik suggests might be better instead of using that PR code since we're not sure yet if it's going to make it.

https://github.com/dagger/dagger/pull/3943#issuecomment-1322859224 > this seems like a more long term solution

GitHub

Private registry support by RonanQuigley · Pull Request #3943 · dag...

Problem
Dagger used to support private registries via a custom buildkit instance. This made use of docker bind mounts. Here's an example:
docker run --net=host -d --restart always -v /private/...

spring badge Nov 30, 2022, 4:56 PM

#

peak plinth So, you need to : 1. clone our repo 2. Checkout to this PR 3. Set env var from...

I am now trying to get this working and the PR has different ./hack/ directory contents than main branch. There is no dev script, so I simply recreated it and reading the https://github.com/dagger/dagger/tree/main/hack#hackdev it looks like I have to give it an argument of what to execute?

#

after executing it without arguments or with bash argument I get this output

+++ dirname ./hack/dev
++ cd ./hack/..
++ pwd
+ DAGGER_SRC_ROOT=/home/company.net/me/Projects/go/dagger
+ MAGEDIR=/home/company.net/me/Projects/go/dagger/internal/mage
+ pushd /home/company.net/me/Projects/go/dagger/internal/mage
~/Projects/go/dagger/internal/mage ~/Projects/go/dagger
+ GOWORK=off
+ go run main.go -w /home/company.net/me/Projects/go/dagger engine:dev
No .go files marked with the mage build tag in this directory.
exit status 1

peak plinth Dec 1, 2022, 4:37 PM

#

Hi @spring badge,
Won't have the time today, but I'll try to make a small guide (following Erik's recommandations), tomorrow [and directly add it to the docs]

misty tulip Dec 1, 2022, 5:19 PM

#

@spring badge I'm around today if you need a hand with this

spring badge Dec 1, 2022, 6:20 PM

#

It's not critical guys, no need to hurry. I'm occupied right now so maybe tomorrow. That Erik's solution would be nice, I'm just not that proficient in Go to make it myself

spring badge Feb 16, 2023, 1:44 PM

#

Hey guys, It's been quite a while since we've discussed this issue. GitHub issue (https://github.com/dagger/dagger/issues/4217) that would solve this has no new activity and I can understand that this is nothing critical so it can take time.

I just wanted to discuss if there is any possible workaround I could do that would be easy to transition from once the official way is implemented? If you have any suggestions how I could convince my buildkit to trust my self-signed CA of private registry for usage in dagger? I have this CA trusted on my local machine but the buildkit dagger uses doesn't pick it up. Using docker pull, push, login and other commands against this registry works fine.

peak plinth Feb 16, 2023, 1:53 PM

#

Hi @spring badge,
Thanks for the ping 🙏
@rugged temple is currently implementing the service API (https://github.com/dagger/dagger/pull/4505). You might be able to spawn a registry directly from your Dagger pipeline (and maybe without having to self-sign ?)

#

Otherwise, it might become a follow-up implementation. We first wanted to implement a nice way to interact between services prior solving that issue

spring badge Feb 16, 2023, 1:54 PM

#

This registry I'm talking about is internal of company I work for and I have no control over it. But the images I need to use are located there

peak plinth Feb 16, 2023, 1:56 PM

#

spring badge This registry I'm talking about is internal of company I work for and I have no ...

Does that help ? https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#configuration

#

Ok, so we seem to handle it:

Custom CA Certs - If you need any extra CA certs to be included in order to, e.g. push images to a private registry, they can be included under /etc/ssl/certs in the runner image.
This can be accomplished by building a custom engine image using ours as a base or by mounting them into a container created from our image at runtime.

#

Do you want to sync over a zoom ? if that can help you get unlocked ? I'll try to repro on my side too

spring badge Feb 16, 2023, 2:05 PM

#

If it's about running custom runner from modified image which contains the needed CA I should be able to do it

#

I didn't know this was possible or I missed it before

peak plinth Feb 16, 2023, 2:05 PM

#

spring badge If it's about running custom runner from modified image which contains the neede...

Yep exactly. Then rely on _EXPERIMENTAL_DAGGER_RUNNER_HOST to point to that container

#

Yes sorry, it got implemented and I forgot about it, should have pinged you 🙏

spring badge Feb 16, 2023, 2:08 PM

#

No problem, I will try that out. Thanks!

peak plinth Feb 16, 2023, 2:14 PM

#

spring badge If it's about running custom runner from modified image which contains the neede...

I think the command we use to run the image is: https://github.com/dagger/dagger/blob/b26a6d20c8dc922c7b002a02e081a9b981f34082/internal/mage/engine.go#L285-L295. You shouldn't be too far to do something like that and add the mounted volumes to the engine.

spring badge Feb 16, 2023, 2:25 PM

#

I figured it out from the documentation you sent me.
Command

docker run \
  --rm \
  --privileged \
  --name dagger-runner \
  -v dagger-engine:/var/lib/dagger \
  -v /etc/ssl/certs:/etc/ssl/certs:ro \
  registry.dagger.io/engine:v0.3.12

And running _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-runner go run main.go runs my Go code and successfully pulls the image from that one private registry (I also execute docker login before connecting to dagger)

#

It's so good to see that I have errors in my dagger pipeline and not in the image pull 😄

#

Thank you very much, this will do it in the meantime

peak plinth Feb 16, 2023, 2:26 PM

#

So happy that it could unlock you 💥

plain sinew Dec 22, 2023, 2:07 AM

#

spring badge I figured it out from the documentation you sent me. Command ```bash docker run...

with v0.9.5 that doesn't seem to work anymore

spring badge Dec 22, 2023, 5:06 PM

#

I haven't gone with upgrades that far so I cannot help you, but good to know, thanks

misty tulip Dec 22, 2023, 6:33 PM

#

plain sinew with v0.9.5 that doesn't seem to work anymore

that's strange @plain sinew . Was this issue reported?

#Go SDK fails to pull image from private registry due to certificate signed by unknown authority