golang: AsTarball fails due to "go mod download" in Dockerfile

[ebon sundial]

I have a repo that has been working great with Dagger until I added scanning with Trivy: https://gist.github.com/gmlewis/c044049bcb09a5ef97a2ef5c655bec96

The AsTarball call is causing the following error:

Trivy ScanContainer error: input: container.build.asTarball resolve: container image to tarball file conversion failed: failed to solve for container publish: process "/bin/sh -c go mod download" did not complete successfully: exit code: 1

The Dockerfile is pretty standard:

FROM golang:1.22 as builder
RUN addgroup --system --gid 1001 appuser && adduser --system --uid 1001 appuser --ingroup appuser
WORKDIR /build
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
...

Any ideas on how to prevent AsTarball from failing?

[lime yew]

👋 is this Dagger v0.11.0?

[ebon sundial]

Yes, sorry.

[lime yew]

checking

@ebon sundial this worked for me :

type Test1 struct{}

// Return a Container from the official trivy image.
func (t *Test1) base() *dagger.Container {
    return dag.Container().
        From(trivyImage).
        WithMountedCache("/root/.cache/trivy", dag.CacheVolume("trivy-db-cache"))
}

// Scan a Dagger Container.
// [severity] can be one or more of: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL".
// [exitCode] can be "0" to always pass or "1" to fail pipeline on failure.
// [format] can be "table" or "json".
func (t *Test1) ScanContainer(ctx context.Context, ctr *dagger.Container) (string, error) {
    tarballName := fmt.Sprintf("/scan/tmp-%v.tar", time.Now().UnixNano())
    return t.base().
        WithMountedFile(tarballName, ctr.AsTarball()).
        WithExec([]string{
            "image",
            "--input", tarballName,
        }).Stdout(ctx)
}

^ simplified version

and then I called it with dagger call scan-container --ctr alpine

not sure if the reason is the dag vs t.Client change

[ebon sundial]

I should have stated that this is in a GitHub Action workflow (and I've honestly never used the dagger CLI command before).
The Publish step works fine without the Trivy step, but the AsTarball step fails.

Here's how I'm calling it if that helps: https://gist.github.com/gmlewis/40b1e34e5131dfd27c7c2a0d7b8b6976

[lime yew]

ok, so you're not using Dagger functions I assume?

[ebon sundial]

I'm using the Dagger Go SDK for all operations.

[lime yew]

👍 . Let me put together a non-modules example

@ebon sundial also works.

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    c, err := dagger.Connect(context.Background(), dagger.WithLogOutput(os.Stderr))
    if err != nil {
        panic(err)
    }
    t := &Trivy{Client: c}
    fmt.Println(t.ScanContainer(context.Background(), c.Container().From("alpine"), "LOW", "1", "table"))
}

the trivy.go I left it untouched of the one you shared

if I run go run main.go on that file it works

[ebon sundial]

(Not sure if this is relevant - but the "go mod download" is accessing private repos... I'm guessing that the Publish step takes that into account but the AsTarball step doesn't.)

[lime yew]

oh, you're sending a Container from a dockerbuild command?

In any case if Publish works, AsTarball should also

[ebon sundial]

Yes, exactly.

[lime yew]

are you able to make it fail locally in your machine?

[lime yew]

have you checked that the Publish step is not using an old cached image?

[ebon sundial]

I don't know how to run dagger locally. I don't think it is using a cached image, but I'm not sure how to verify that.

[lime yew]

are you able to access those private repositories from your local machine?

if you have a few minutes we can check it togerther in a quick #911305510882513037 session

[ebon sundial]

Thank you, @lime yew , but I'll have to check in tomorrow, as I'm now unfortunately away from my computer. I appreciate your help!

[lime yew]

sure, np. Ping me tomorrow and we can 👀 together 🙏

[ebon sundial]

Hi @lime yew - before I trouble you with my ignorance, I've downloaded the dagger CLI on my Mac M2 Max laptop, and am working through the "Quickstart" page (https://docs.dagger.io/quickstart/562821/hello) but get the following error:

$ dagger -m github.com/shykes/daggerverse/hello@v0.1.2 call hello
✘ initialize 2.2s
...
Error: input: moduleSource.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "/usr/local/bin/codegen --module-context /src --module-name hello --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Any ideas what I'm doing wrong? Nothing exists at /usr/local/bin/codegen but I'm guessing that might be a container that it is talking about.

[lime yew]

@ebon sundial mind running with -vvv and sharing the full output please?

[lime yew]

interesting.. not entirely sure why codegen is failing here. cc @ancient mauve @fallen viper any clues?

a wild guess could be something related to your Docker engine config (rosetta maybe?).. but probably Helder or Justin might have a more accurate idea

[ebon sundial]

I'm updating my Docker Desktop from 4.25.2 to 4.28.0 now to see if that fixes the problem...

[lime yew]

please make sure that you have rosetta disabled as well

[ebon sundial]

I'm not sure what "rosetta" is, actually.

[lime yew]

https://docs.docker.com/desktop/settings/mac/

[ebon sundial]

It was on... I turned it off and am trying again...

With rosetta off, docker-eng just remains in the "Restarting" state:

It timed out after 5 minutes.

[lime yew]

hmm that's strange.. by any chance do you have a DOCKER_DEFAULT_PLATORM variable set in your shell config?

[ebon sundial]

Ah yes, I do! Wow, that was a long time ago: DOCKER_DEFAULT_PLATFORM=linux/amd64
I'll try without it.

It still seems to hang, but this time with a warning:

  ✔ exec docker run --name dagger-engine-9cf4d4ecdf5b646b -d --restart always -v /var/lib/dagger --privileged registry.dagger
  ┃ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no s
  ┃ fic platform was requested

[lime yew]

can you remove the engiem image and try again?

seems like it's still trying to use the amd64 image when it should use the arm64 instead

did you unset the variable in your current shell session as well?

[ebon sundial]

yes

Success! hello, world! - thank you, @lime yew ! I'll work through the rest of the Quickstart and make sure my system is happy.

[lime yew]

🚀

[lime yew]

awesome @ebon sundial ! happy to resume our chat about the AsTarball thing once you can run the pipeline locally 🙌

[ebon sundial]

Thank you! On this page: https://docs.dagger.io/quickstart/853930/directories
It says I should be able to cd ./my-dagger-build followed by ./dagger version but instead, I get this:

glenn@glenn-MacM2Pro ~/src/github.com/orsinium-labs/wypes/my-dagger-build ((main)) $ ./dagger version
-bash: ./dagger: cannot execute binary file
glenn@glenn-MacM2Pro ~/src/github.com/orsinium-labs/wypes/my-dagger-build ((main)) $ file ./dagger
./dagger: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, Go BuildID=4u6btZ72_FSoq8T7o0NK/Lpb0E1HnlBHZa_Ux5A9y/_GS0APWLcFtVPBNZikYj/cBF6Fri25kZMO8p0qojW, with debug_info, not stripped

Any ideas why that might be?

Oh, maybe the dynamic linking?!?

Yep, that must be it, as that file does not exist.

Weird that it didn't statically link.

[lime yew]

that's because that module is using the default go build without any flags :S

having said that it's strange that it didn't work in your machine since the binary seemsto be effectively built for aarch64

[ebon sundial]

I thought the default "go build" without any flags always created a statically-linked binary executable?!? (I suppose that's if there is no CGO.)

[lime yew]

not always by default if you're using some packages like net or os that dynamically link it by default

[ebon sundial]

@lime yew - after more investigation, it turns out that AsTarball was never the problem after all (and it performs the same as the Publish with regard to private repos).
The problem started when I upgraded from the Go dagger SDK v0.8.7 to v0.11.0.
Still investigating...

[lime yew]

thx for letting me know. That makes a bit more sense now. LMK how can we help 🙌

[ebon sundial]

@lime yew - after reading this thread: https://github.com/dagger/dagger/issues/7087 I updated all my "go.mod" files to explicitly say "go 1.22.2" and relaunched the GitHub Actions on the repos, and now it is working just fine. Wheh.
Thanks for all your help!
Some day I would love to learn how to debug all this locally... but I think I just need to work through all the tutorials. 😂

[ebon sundial]

@lime yew - I have one repo that I must be doing something stupid in, so I'm trying to debug it, and even my debugging is messed up. I'm still having problems with RUN go mod download -x in one Dockerfile, so now I have this:

FROM golang:1.22 as builder
RUN addgroup --system --gid 1001 appuser && adduser --system --uid 1001 appuser --ingroup appuser
WORKDIR /build
ARG TOKEN=${TOKEN}
ENV CGO_ENABLED=0
ENV GOPRIVATE=github.com/gmlewis/*
COPY go.mod .
COPY go.sum .
RUN go version
RUN go mod download -x

But I can't seem to see the output from either the go version or the go mod download -x commands.
Here's my code: https://gist.github.com/gmlewis/40b1e34e5131dfd27c7c2a0d7b8b6976 and https://gist.github.com/gmlewis/c044049bcb09a5ef97a2ef5c655bec96
and the output is:

2024/04/17 16:52:35 BuildAndPublishToAzure: Building container from "Dockerfile"
2024/04/17 16:52:36 ScanContainer FAILED: input: container.build.asTarball resolve: container image to tarball file conversion failed: failed to solve for container publish: process "/bin/sh -c go mod download -x" did not complete successfully: exit code: 1
stdout:
input: container.build.stdout resolve: process "/bin/sh -c go mod download -x" did not complete successfully: exit code: 1
2024/04/17 16:52:36 Trivy ScanContainer error: input: container.build.asTarball resolve: container image to tarball file conversion failed: failed to solve for container publish: process "/bin/sh -c go mod download -x" did not complete successfully: exit code: 1
exit status 1
Error: Process completed with exit code 1.

Any ideas what I'm doing wrong?

[lime yew]

@ebon sundial wanna jump really quick to #911305510882513037 to check it out?

[ebon sundial]

Sorry, just saw this... are you still there?

[lime yew]

yep

still here

[ebon sundial]

OK, trying to connect my microphone...

yeah, I'm speaking - trying to enable microphone in Chrome...

[lime yew]

just jumped into a quick call with @ebon sundial and was able to verify it's indeed an issue with what it seems the new telemetry not exposing Dockerfile build outputs. Just checking with @rugged solstice if this has been filed/identified already 🙏

[rugged solstice]

nope sounds new to me!

[lime yew]

sweet! 🍬 filing an issue