#AWS CodePipeline - Dagger SHIM

1 messages · Page 1 of 1 (latest)

subtle eagle Feb 23, 2023, 2:07 PM

There are some examples to use dagger (go sdk) within AWS CodePipeline ?

quiet gazelle Feb 23, 2023, 3:17 PM

I don't think I've seen anything so far. But happy to help if that's some use case you want to explore. cc @dark bloom @marble quarry

particularly since you have experience with CodePipeline

marble quarry Feb 25, 2023, 1:11 AM

tldr is, use Dagger with AWS CodeBuild via a buildspec.yml in your repo just like any other CI YAML for GitHub Actions, CircleCI, or GitLabCI. CodeBuild fits perfectly into AWS CodePipeline as a Build Provider. It all works really nicely 🙂 I'll look to turn this into a blog post or docs guide. cc @naive parcel
https://github.com/jpadams/greetings-api/blob/main/buildspec.yml

version: 0.2

phases:
  pre_build:
    commands:
      - echo Setting up QEMU
      - docker run --rm --privileged tonistiigi/binfmt:latest --install all

  build:
    commands:
      - echo Running Dagger Pipeline
      - go run ci/main.go test 

  post_build:
    commands:
      - echo Build completed on `date`

artifacts:
  files:
    - 'output/**/*'
  name: builds/$CODEBUILD_BUILD_NUMBER

With CodePipeline, once you connect your repo, you setup a Builld Provider, it looks like. AWS CodeBuild or Jenkins.

GitHub

greetings-api/buildspec.yml at main · jpadams/greetings-api

Contribute to jpadams/greetings-api development by creating an account on GitHub.

If I choose CodeBuild, I notice that we can either use their Images or your can choose a custom one.

I'm going to try one of their Ubuntu images which has a Standard set of software.

https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html

Docker images provided by CodeBuild - AWS CodeBuild

AWS CodeBuild manages the following Docker images that are available in the CodeBuild and AWS CodePipeline consoles.

Looks like the Ubuntu 22.04 has Golang 1.18, which is perfect for the Go SDK that needs Golang 1.15 or higher.
https://github.com/aws/aws-codebuild-docker-images/blob/master/ubuntu/standard/6.0/Dockerfile#L230
https://docs.dagger.io/sdk/go/371491/install

I'll check Privileged

I'll opt to use a buildspec.yml file where I can hopefully just run my Dagger pipeline via go run main.go

I'll skip the Deploy phase for now

Now I seem to have a pipeline! 🎉

I'll try to add a buildspec.yml to my GitHub repo now and trigger it...

since it doesn't exist yet 😄

added one:

https://github.com/jpadams/greetings-api/commit/6a551247db6aec46925449e1a9dd7c2cee41eeb0#diff-69387ac97f1b775f19989fc28150f89e785406cbff04643a969f5e396573dd5cR10

GitHub

Add buildspec for CodeBuild · jpadams/greetings-api@6a55124

Signed-off-by: Jeremy Adams

Same way I ran my Dagger pipeline locally on my laptop

go run ci/main.go test

marble quarry Feb 25, 2023, 5:12 AM

Got a failure, so started sending my build logs to S3 (CloudWatch another option):

[Container] 2023/02/25 01:45:30 Waiting for agent ping
[Container] 2023/02/25 01:45:31 Waiting for DOWNLOAD_SOURCE
[Container] 2023/02/25 01:45:39 Phase is DOWNLOAD_SOURCE
[Container] 2023/02/25 01:45:39 CODEBUILD_SRC_DIR=/codebuild/output/src101057294/src
[Container] 2023/02/25 01:45:39 YAML location is /codebuild/output/src101057294/src/buildspec.yml
[Container] 2023/02/25 01:45:39 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2023/02/25 01:45:39 Processing environment variables
[Container] 2023/02/25 01:45:39 No runtime version selected in buildspec.
[Container] 2023/02/25 01:45:41 Moving to directory /codebuild/output/src101057294/src
[Container] 2023/02/25 01:45:41 Configuring ssm agent with target id: codebuild:373219b0-33d2-4ddc-ab3e-6d92d35066e5
[Container] 2023/02/25 01:45:41 Successfully updated ssm agent configuration
[Container] 2023/02/25 01:45:41 Registering with agent
[Container] 2023/02/25 01:45:41 Phases found in YAML: 3
[Container] 2023/02/25 01:45:41  PRE_BUILD: 1 commands
[Container] 2023/02/25 01:45:41  BUILD: 2 commands
[Container] 2023/02/25 01:45:41  POST_BUILD: 1 commands
[Container] 2023/02/25 01:45:41 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2023/02/25 01:45:41 Phase context status code:  Message: 
[Container] 2023/02/25 01:45:41 Entering phase INSTALL
[Container] 2023/02/25 01:45:41 Phase complete: INSTALL State: SUCCEEDED
[Container] 2023/02/25 01:45:41 Phase context status code:  Message: 
[Container] 2023/02/25 01:45:41 Entering phase PRE_BUILD
[Container] 2023/02/25 01:45:41 Running command echo Nothing to do in the pre_build phase...
Nothing to do in the pre_build phase...

[Container] 2023/02/25 01:45:41 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2023/02/25 01:45:41 Phase context status code:  Message: 
[Container] 2023/02/25 01:45:41 Entering phase BUILD
[Container] 2023/02/25 01:45:41 Running command echo Running Dagger Pipeline
Running Dagger Pipeline

[Container] 2023/02/25 01:45:41 Running command go run ci/main.go test
go: downloading dagger.io/dagger v0.4.2
go: downloading github.com/aws/aws-sdk-go v1.44.119
go: downloading github.com/Khan/genqlient v0.5.0
go: downloading github.com/iancoleman/strcase v0.2.0
go: downloading github.com/vektah/gqlparser/v2 v2.5.1
go: downloading github.com/adrg/xdg v0.4.0
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading golang.org/x/sys v0.1.0
go: downloading golang.org/x/sync v0.1.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/hashicorp/errwrap v1.1.0
Unable to find image 'ghcr.io/dagger/engine:v0.3.7@sha256:9ad18aca71b1b1bae137d3ac5b8d1c62bd548d4f3b01c42b48120fdbc1490f22' locally
ghcr.io/dagger/engine@sha256:9ad18aca71b1b1bae137d3ac5b8d1c62bd548d4f3b01c42b48120fdbc1490f22: Pulling from dagger/engine

etc

#15 sh -c go test > /src/test.out
#15 DONE 21.2s

#16 sh -c go test > /src/test.out
#16 DONE 22.1s

#14 sh -c go test > /src/test.out
#14 ...

*#13 sh -c go test > /src/test.out
#13 42.17 # runtime/cgo
#13 42.17 Error while loading /go/bin/as: No such file or directory
#13 ...

#14 sh -c go test > /src/test.out
#14 41.76 # runtime/cgo
#14 41.76 Error while loading /go/bin/as: No such file or directory
failed to run task test: input:1: container.from.withMountedDirectory.withWorkdir.withMountedCache.withEnvVariable.withExec.file process "/dev/.buildkit_qemu_emulator sh -c go test > /src/test.out" did not complete successfully: exit code: 2

Please visit https://dagger.io/help#go for troubleshooting guidance.
exit status 1

[Container] 2023/02/25 01:49:00 Command did not exit successfully go run ci/main.go test exit status 1
[Container] 2023/02/25 01:49:00 Phase complete: BUILD State: FAILED
[Container] 2023/02/25 01:49:00 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: go run ci/main.go test. Reason: exit status 1*

[Container] 2023/02/25 01:49:00 Entering phase POST_BUILD
[Container] 2023/02/25 01:49:00 Running command echo Build completed on `date`
Build completed on Sat Feb 25 01:49:00 UTC 2023

[Container] 2023/02/25 01:49:00 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2023/02/25 01:49:00 Phase context status code:  Message:

Aha, didn't setup QEMU stuff like this: https://github.com/jpadams/greetings-api/blob/main/.github/workflows/test.yml#LL14C11-L14C78
Which I need since this does a multi-arch build 🙂

Success!

Now I'll try to add the artifacts line that I see in most buildspec.yml examples to perhaps save out my built artifacts like I see on my Mac M1 laptop:

greetings-api ➤ tree output                                            git:main
output
└── linux
    ├── amd64
    │   ├── 1.18.out
    │   └── 1.19.out
    └── arm64
        ├── 1.18.out
        └── 1.19.out

I'll try this approach https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax:~:text=is generated from.-,artifacts/files,-Required sequence. Represents

artifacts:
  files:
    - 'output/**/*'
  name: builds/$CODEBUILD_BUILD_NUMBER

marble quarry Feb 25, 2023, 5:57 AM

Sure enough...build artifacts got saved to S3!

I downloaded the Zip to check ✅

So here's the final buildspec.yml I used 🙂

https://github.com/jpadams/greetings-api/blob/main/buildspec.yml

version: 0.2

phases:
  pre_build:
    commands:
      - echo Setting up QEMU
      - docker run --rm --privileged tonistiigi/binfmt:latest --install all

  build:
    commands:
      - echo Running Dagger Pipeline
      - go run ci/main.go test 

  post_build:
    commands:
      - echo Build completed on `date`

artifacts:
  files:
    - 'output/**/*'
  name: builds/$CODEBUILD_BUILD_NUMBER