#DockerBuild from private registry 0.19.10

1 messages · Page 1 of 1 (latest)

sick herald
#

We have a problem in our CI when we run jobs that use the same base image.
We are authenticating like this:

_, err = dag.Container().
  WithEnvVariable("CACHE_BUSTER", time.Now().Format("20060102150405")).
  WithRegistryAuth(harborRegistry, g.HarborUsername, g.HarborPassword).
  Sync(ctx)

And then running our build like this:

func (m *MyModule) BuildBaseFpm() *dagger.Container {
  return m.Source.
    DockerBuild(dagger.DirectoryDockerBuildOpts{
      Dockerfile: "fpm.Dockerfile",
      Platform:   "linux/amd64",
  })
}

I we run a single job it works. As soon as we are running concurrent jobs they start failing with:

harbor.example.com/image/fpm:8.3-slim: failed to resolve source metadata for harbor.example.com/image/fpm:8.3-
  slim: unexpected status from HEAD request to https://harbor.example.com/v2/image/fpm/manifests/8.3-slim: 401
  Unauthorized

Does anyone has run into this issue also?

Dagger version: 0.19.10
The jobs share the same dagger engine

open monolith
#

@sick herald is the username and the password always the same in multiple jobs?

sick herald
#

Its the same

#

yes

#

@open monolith

#

Is there a way I could debug this some how? It's quite a big project 😅

open monolith
#

will provide an update in a bit 🙏

open monolith
#

@sick herald I'm not able to repro. The only thing that comes to my mind is that the credentials are effectively different in both jobs and somehow they're being overwritten. I'm testing this locally by pulling and pushing to a private dockerhub repo concurrently with multiple dagger functions calls and it always succeeds

#

or maybe harbor is doing something funky and unauthorizing requests somehow?

are you able to access your harbor registry from your local machine? you could do a quick test to rule this out. Here's a sample Dagger function I've used to test this

func (m *BuildAuth) Test(ctx context.Context,
    // +defaultPath="."
    dir *dagger.Directory,
) (string, error) {
    s := dag.SetSecret("hub", "$TOKEN")
    _, err := dag.Container().WithEnvVariable("CACHE", time.Now().String()).
        WithRegistryAuth("$ADDRESS", "$USERNAME", s).Sync(ctx)
    if err != nil {
        return "", err
    }

    return dir.DockerBuild(dagger.DirectoryDockerBuildOpts{
        BuildArgs: []dagger.BuildArg{{
            Name:  "foo",
            Value: time.Now().String(),
        }},
    }).
        WithEnvVariable("CACHE", time.Now().String()).
        Publish(ctx, "repo/name:"+strconv.Itoa(int(time.Now().UnixNano())))
}

^ I've tried running dagger call test from multiple terminals concurrently in my machine and all the functions were able to push and pull from the registry successfuly

sick herald
#

I will try to build a minimal test case and see if it keeps happening. Maybe it is our harbor instance.

agile geyser
sick herald
#

Well it does not seem to be our harbor instance. Will investigate further untill i have it nailed down.

sick herald
#

@open monolith Could you try it with this script?

package main

import (
        "bytes"
        "fmt"
        "os"
        "os/exec"
        "strconv"
        "sync"
)

func main() {
        if len(os.Args) < 3 {
                fmt.Println("Usage: repeat <count> <command...>")
                os.Exit(1)
        }

        count, _ := strconv.Atoi(os.Args[1])
        cmd := os.Args[2:]

        ok := 0

        var wg sync.WaitGroup
        for i := range count {
                wg.Add(1)
                go func(n int) {
                        defer wg.Done()

                        c := exec.Command(cmd[0], cmd[1:]...)
                        c.Env = append(os.Environ(), "NO_COLOR=1", "CI=true")
                        out, err := c.CombinedOutput()
                        code := c.ProcessState.ExitCode()
                        if code == 0 {
                                ok++
                                return
                        }

                        // Find the first occurence of error and slim down out from there.
                        idx := bytes.Index(out, []byte("ERROR"))
                        if idx == -1 {
                                fmt.Printf("=== %d ===\ncode: %d\nerr: %v\noutput:\n%s\n", n+1, code, err, out)
                                return
                        }

                        fmt.Printf("=== %d ===\ncode: %d\nerr: %v\noutput:\n%s\n", n+1, code, err, out[idx:])
                }(i)
        }
        wg.Wait()

        if ok > 0 {
                fmt.Printf("\n\nOK: %d / %d\n", ok, count)
        }
}
#

And then call it with:

#

go build -o ./repeat main.go

#

./repeat 50 dagger call --progress=plain -q -m ./module --harbor-username env://HARBOR_USERNAME --harbor-password env://HARBOR_PASSWORD --directory ./ test

#

I do not even do the publish and get the error in the resolving metadata.

#

I sometimes need 3 or 4 tries before it starts failing. and sometimes the first try.

#

This is my module code

// Returns a container that echoes whatever string argument is provided
func (m *DaggerTestcaseHarborFailure) Test(ctx context.Context, harborUsername *dagger.Secret, harborPassword *dagger.Secret) (*dagger.Container, error) {
    username, err := harborUsername.Plaintext(ctx)
    if err != nil {
        return nil, err
    }

    if username != "" && harborPassword != nil {
        _, err := dag.Container().
            WithRegistryAuth("harbor.example.com", username, harborPassword).
            Sync(ctx)

        if err != nil {
            return nil, err
        }
    }

    ctr := m.Source.
        DockerBuild(dagger.DirectoryDockerBuildOpts{
            Dockerfile: "Dockerfile",
            Platform:   "linux/amd64",
        })

    ctr = ctr.WithEnvVariable("CACHE", time.Now().String())

    return ctr.Sync(ctx)
}
open monolith
#

ok, thx for sending this repo. Let me check really quick

sick herald
#

I am trying lot's of things to see if i can fix it but its quite a large repo

#

🙂

sick herald
#

I fixed it for me doing the following. The issue was that the auth step was not executed always.
When I "use" the result it will execute correctly and do the auth.

ctr, err := dag.Container().
    WithEnvVariable("CACHE_BUSTER", time.Now().String()).
    WithRegistryAuth(harborRegistry, g.HarborUsername, g.HarborPassword).
Sync(ctx)

if err != nil {
    return nil, err
}

// "Use" the result.
m.Source = m.Source.WithDirectory("/tmp/auth", ctr.Rootfs())

// Now do the DockerBuild on m.Source.
#

Thank you for your time :).

open monolith
agile geyser