#Troubleshooting "yarn install" using "node:16" image and Go SDK

1 messages · Page 1 of 1 (latest)

solemn plover
#

I'm brand new to Dagger and would like to write all my pipelines in Go, even for my Vue3 web apps.
So I wrote the following in my Dagger "main.go" file: (WHUPS! I hit the message file size limit... trimming down...)

...    node = node.WithExec([]string{"yarn", "--version"})
    yarnVersion, err := node.Stdout(ctx)
...
    // mount cloned repository into `node` image and run the pipeline
    node = node.WithMountedDirectory(".", src).
        WithExec([]string{"yarn", "install"}).
        WithExec([]string{"yarn", "test:unit"}).
        WithExec([]string{"yarn", "lint"})
    results, err := node.Stdout(ctx)
...

and it works great up until the "yarn install" command:

$ go run main.go
Running on-push-or-pull-request CI/CD pipeline using Dagger
#1 resolve image config for docker.io/library/node:16
#1 DONE 0.3s
2023/01/24 14:08:29 Running node version: v16.19.0
...
#1 resolve image config for docker.io/library/node:16
2023/01/24 14:08:29 Running yarn version: 1.22.19
#1 ...
...
#8 docker-entrypoint.sh yarn install
#8 ERROR: process "docker-entrypoint.sh yarn install" did not complete successfully: exit code: 1
2023/01/24 14:08:30 input:1: container.from.withExec.withExec.withMountedDirectory.withExec.withExec.withExec.stdout process "docker-entrypoint.sh yarn install" did not complete successfully: exit code: 1
Stdout:

Stderr:


Please visit https://dagger.io/help#go for troubleshooting guidance.
exit status 1

Any ideas what I'm doing wrong?
I'm having troubles figuring out how to triage the terse and cryptic error from "docker-entrypoint.sh" (which I did not write).

half hamlet
#

That looks like maybe the docker image you’re using as a base has a built-in entrypoint?

#

Note that the entrypoint would explain seeing “docker-entrypoint.sh”. But that may not necessarily be the cause of your error

#

by the way feel free to share a Github gist, or open a github issue if it helps share the full code

solemn plover
#

Oh! Thank you, @half hamlet - I was not using a container previously for the build (that I'm aware of) - I was just using a Bitbucket Pipeline or a GitHub Action - however it is a requirement for me to choose a container in order to use Dagger, correct? Should I just start with "alpine" and build up what I need?

half hamlet
#

Dagger runs everything in containers, including those WithExec you pasted above. It takes care of the container orchestration for you. I’m curious to see the rest of your code , to see if anything is missing

solemn plover
#

OK, thanks! Sure... one moment and I'll make a gist...

half hamlet
#

I just assumed your code earlier started with something like client.Container().From(…)

solemn plover
#

Yes, it did... just like the Go SDK "getting started" example, and I chose "node:16"... one moment... gist upcoming...

mortal ingot
#

@solemn plover think I found the issue. The default CWD of the node:16 image is /. So when you're doing your WithMountedDirectory you're effectively replacing the entire / mountpath with your project files. What you need to do is mount your code in a different path and then use WithWorkingDir to switch there

solemn plover
#

Oh! WOW! I'm so glad you found that... I would have never guessed. Thank you, @half hamlet and @mortal ingot ! Keep up the awesome work!
I'm looking forward to replacing all my pipelines! Woohoo!!! 😄

mortal ingot
#

that's why all the initial yarn --version works but then it breaks afterwards

mortal ingot
solemn plover
#

I've almost got this thing working... but the "yarn install" step is consistently taking a very long time (31sec) when iterating over and over again...
Are there any ways to manually or explicitly cache the results of running this step?

solemn plover
half hamlet
#

I believe yarn has a cache directory other than node_modules

#

not sure what the default is

mortal ingot
# solemn plover I tried "WithMountedCache", but couldn't seem to get it to work: https://gist.gi...

seems like the issue is that you're changing the cache-key for each execution here: https://gist.github.com/gmlewis/bd446031759a4465f488ea205ed5b22e#file-main-go-L81-L82 since you're appending the current time to the cache-key

that will basically cause each run to generate a brand new cache which will in consequence reinstall dependencies each time.

Gist

Dagger newbie troubleshooting first Go SDK pipeline - main.go

half hamlet
#

yeah that too 😁 i didn’t see that

#

It seems that the default location is ./.yarn/cache

mortal ingot
# half hamlet make sure to mount a 2nd cache volume for this: https://yarnpkg.com/configuratio...

AFAIK ~/.yarn/cache is a OS user level cache so that when you remove node_modules from a project and you do a yarn install it doesn't download everything from the web again. From the experience that I personally have seeing at node / front Dockerfiles, I don't recall many people actually using ~/.yarn/cache.

I guess it comes very useful when you can re-use that cache across different projects withing your same dagger engine worker

half hamlet
#

I’m just looking at what we did in the yarn package in the CUE sdk (we cached both). No idea if that was actually important to do

mortal ingot
solemn plover
#

Removing the timestamp worked! Thank you!
I have 2 questions:

  1. How do I ensure that the cache is ignored if my "package.json" or "yarn.lock" files changed since the last run?
  2. Is there a way to force the run of any particular step (e.g. "yarn --version") just so its output can always be seen in the logs instead of "#10 CACHED"?
solemn plover
#

In addition to the 2 questions above (#1067527046307266681 message), I have a few more questions, if you don't mind.
I think I understand the CI part of CI/CD using Dagger now, thanks!

Now I'm trying to wrap my head around a few things:

  1. checking out the code, bumping a VERSION constant in a *.go file, making a commit back to the repo, building the image, then pushing that image to a container registry (Azure in my current case),
  2. checking out, bumping the version, then committing to a separate repo containing Helm charts, then calling Helm to deploy the image to the dev cluster (again, Azure, although it shouldn't matter),
  3. creating branches for production and tagging them as release candidates within the same GitHub action.

I'm guessing these are do-able, but just finished reading through all the Go SDK Guides for Dagger, and have run through the official docs: https://pkg.go.dev/dagger.io/dagger and it looks like the majority of operations will have to be performed by using a bunch of WithExec calls calling out to tools like "sed", "git", "helm", etc., which makes sense now that I say it... and I'll pursue that but please correct me if I'm way off-base here.
Thanks again!

Discord

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

mortal ingot
half hamlet
#

For 2. the solution is correct, BUT I would suggest thinking about why you want it printed in the logs every time

#

If your goal is simply to make sure the output of that command is printed to the user, then it's not really a caching issue: you should capture the output of that command, and print it yourself. The logs are not guaranteed to be printed to the end user

mortal ingot
solemn plover
# mortal ingot 1. Your cache will be automatically invalidated if any your source file changes....

When you say "1. Your cache will be automatically invalidated if any your source file changes. So you don't have to worry about this. " - correct me if I'm wrong, but I would like to keep the SAME cache if any files EXCEPT for "package.json" or "yarn.lock" change. In other words, "yarn install" should never change until I modify either "package.json" or "yarn.lock", right? How can I control the cache such that this is the case?

mortal ingot
#

it's the same as calling yarn install multiple times in your local project?

solemn plover
#

I think the only difference is that "yarn install" checks to see if "package.json" or "yarn.lock" have CHANGED since the last run, and if so, it will basically ignore the node_modules, or at least attempt to bring it up-to-date.

#

Ah! Maybe I could add those two files to the cache?!?

mortal ingot
solemn plover
#

Oh! I've been a dork. Of course you are right. Sorry for wasting your time!!!

mortal ingot
solemn plover
#

Quick Question - do you recommend reading/parsing/modifying/writing files in a dagger.Host object or a dagger.Query object?
I've been working on the former, but it appears that if I then want to perform a "git add", "git commit", "git push", that I need the latter... is that correct?

mortal ingot
solemn plover
#

Oh, sure... one moment... gist coming up...

solemn plover
#

Gotta step out now, but with the updated gist, I get this error:

#6 go version
#6 0.219 go version go1.19.5 linux/amd64
2023/01/25 16:42:35 Running Go version: go version go1.19.5 linux/amd64
#6 DONE 1.1s

#3 resolve image config for docker.io/library/golang:1.19
#3 DONE 0.9s

#7 mkdir /src
#7 DONE 0.0s

#8 mkfile /src/version.js
#8 DONE 0.0s

#9 git config user.name Automated Minor Version Bump
#9 0.201 fatal: not in a git directory
#9 ERROR: process "git config user.name Automated Minor Version Bump" did not complete successfully: exit code: 128
2023/01/25 16:42:36 input:1: container.from.withExec.withMountedDirectory.withWorkdir.withExec.withExec.withExec.withExec.stdout process "git config user.name Automated Minor Version Bump" did not complete successfully: exit code: 128
Stdout:

Stderr:
fatal: not in a git directory
mortal ingot
mortal ingot
# solemn plover https://gist.github.com/gmlewis/680621bc9ed2477e6cfa5832fcb7194e

going back to this, I think doing that either in the host or Query objects its pretty much the same. What I'd raise as an observation here is the fact that sometimes fetching the file contents vs to your pipeline code and writing them back again with the WithNewFile operation might be more complex than just running a WithExec in the pipeline and do the modifications there.

Having said that, is there a reason you need to perform the version bump after calling the HostRepo function here? https://gist.github.com/gmlewis/680621bc9ed2477e6cfa5832fcb7194e#file-example-go-L37. You could just do all this https://gist.github.com/gmlewis/680621bc9ed2477e6cfa5832fcb7194e#file-example-go-L46-L58 before and call the HostRepo function later? That won't require a Contents and WithNewFile call unless I'm missing something.

Gist

Work in progress to bump the version in a repo and push back to repo - example.go

solemn plover
#

Fantastic! I'll take a look at that tonight. Thank you again, @mortal ingot !

solemn plover
#

One thing that is puzzling me... the only thing in the Go SDK docs that has KeepGitDir: true in it is this:

    gitRepo := client.Git(repoURL, dagger.GitOpts{KeepGitDir: true}).
        Commit(gitRef).
        Tree()

which forces me to provide a repoURL, which is not what I want.

I need a way to access the "current" repo within the context of a GitHub Action "on: push:" so that I can perform "git add", "git commit", "git push" actions within my Dagger workflow.
Is there any way to do this without providing a repoURL ?

half hamlet
#

I think what you want is include in host.Directory. @mortal ingot are you sure .git is excluded by default?

solemn plover
#

Thank you, @half hamlet ! Progress!

#7 ls -la
#7 0.118 total 12
#7 0.118 drwxr-xr-x 1 root root 4096 Jan 26 02:29 .
#7 0.118 drwxr-xr-x 1 root root 4096 Jan 26 02:29 ..
#7 0.118 drwxr-xr-x 9 root root 4096 Jan 26 02:25 .git
#7 DONE 0.3s

#8 git config user.name Glenn Lewis
#8 DONE 0.3s

#9 git config user.email 6598971+gmlewis@users.noreply.github.com
#9 DONE 0.2s

#10 git fetch
#10 0.376 Host key verification failed.
#10 0.378 fatal: Could not read from remote repository.
#10 0.378 
#10 0.378 Please make sure you have the correct access rights
#10 0.378 and the repository exists.
#10 ERROR: process "git fetch" did not complete successfully: exit code: 128
2023/01/25 21:29:17 input:1: container.from.withExec.withMountedDirectory.withWorkdir.withExec.withExec.withExec.withExec.withExec.stdout process "git fetch" did not complete successfully: exit code: 128
#

Don't worry about the errors... now I just need to get my git fu up and running. 😁

half hamlet
#

depending on what state github actions leaves your repository in, you may or may not need to configure a remote to push to

mortal ingot
#

IIRC yes. That's the reason KeepGitDir was added for

solemn plover
#

In case it wasn't clear, I couldn't use KeepGitDir. I did this instead:

    gitRepo := common.HostRepo(client, dagger.HostDirectoryOpts{
        Exclude: []string{"node_modules"},
        Include: []string{".git"},
    })

I think the Exclude is completely unnecessary, actually. It's the Include that's key.

#

Sorry, common.HostRepo is this:

// HostRepo returns the host repo from Dagger.
func HostRepo(client *dagger.Client, opts ...dagger.HostDirectoryOpts) *dagger.Directory {
    _, filename, _, _ := runtime.Caller(0)
    scriptDir := filepath.Dir(filename)
    // Since this script lives in cicd/go-push-or-pull-request/main.go,
    // we need to get the base dir of the repo and start there.
    repoBaseDir := filepath.Dir(filepath.Dir(scriptDir))
    log.Printf("script dir: %v, repo base dir: %v", scriptDir, repoBaseDir)

    return client.Host().Directory(repoBaseDir, opts...)
}
half hamlet
#

To my knowledge Host.Directory doesn’t have a KeepGitDirectory argument @mortal ingot

mortal ingot
#

Oh! You're right, I was getting confused with the Git operation. My bad 😁

raven dove
#

Regarding yarn and cache dir, worth knowing about node linker setting. Node modules doesn’t get used unless the mode is set to it, often for making timing easier to use that doesn’t support yarn Berry version. https://yarnpkg.com/configuration/yarnrc#nodeLinker. I know y’all moved on since then but for future reference. 👍

solemn plover
raven dove
#

If you want a node modules directory yes. If you don’t care about that and your tooling doesn’t require then it’s not needed. I know I had to use it for easier onboarding and some compatibility with angular project in the past. YMMV. It’s not clearly documented so wanted to point it out since it changes the behavior of node modules even showing up or not. 👍

half hamlet
#

So if nodeLinker is not set to node-modules, should we cache a different directory?

solemn plover
#

I'm having a heck of a time pushing an image to the Azure Container Registry.
Here is the code: https://gist.github.com/gmlewis/536345ad27c6986e41ae8ff7f5c0f7ff
and here is the error:

2023/01/31 21:14:57 PublishToAzure: ***/odafe/builder:0.8.0

#35 docker login *** -u *** -p ***
#35 ERROR: process "docker login *** -u *** -p ***" did not complete successfully: exit code: 1
2023/01/31 21:14:57 input:1: container.withExec.withExec.stdout process "docker login *** -u *** -p ***" did not complete successfully: exit code: 1
Stdout:

Stderr:


Please visit https://dagger.io/help#go for troubleshooting guidance.
exit status 1
Error: Process completed with exit code 1.

Any ideas what I'm doing wrong or why I'm seeing no output on Stderr?

mortal ingot
#

@solemn plover seems like you're running docker login withing your Dagger pipeline?

#

that needs to happen outside of it, in the client machine that's invoking the pipeline

#

credentials get passed from the docker client to dagger automatically

solemn plover
#

Oh! So the GitHub Action itself must perform that step? Again, I never would have guessed. Thank you, again, @mortal ingot !

raven dove
#

I just built a go based pipeline today to publish to acr. What are the odds! And yes dockerlogin azure pipeline task with a service connection based setup and you should be golden. Make sure you fully qualify the image address (registryname.acr.imo/reponame/image:tag) for example. Ping me if you get stuck and I can also provide a second look. Definitely painful the first time I did it.

mortal ingot
half hamlet
#

It would make more sense from within the pipeline @solemn plover - that’s how we did it in Dagger 0.2, but lost that feature with 0.3 (new graphql API) and haven’t gotten it back yet

solemn plover
#

@raven dove - when you have time, I could use your help... I'm still stuck on pushing to Azure Container Registry. 😢
First, here are some important files:

I have two problems still:

  1. GitHub cache is never preserved. The error message is:
Run actions/cache@v3
  with:
    path: node_modules
    key: Linux-build-cache-node-modules-8e60f01ab852b1daa1a2def054e48a1eaaebab031b4615d2e15971d5aa525bcd
    enableCrossOsArchive: false
    fail-on-cache-miss: false
  env:
    DOCKER_CONFIG: /home/runner/work/_temp/docker_login_1675385229408
    cache-name: cache-node-modules
  
Cache not found for input keys: Linux-build-cache-node-modules-8e60f01ab852b1daa1a2def054e48a1eaaebab031b4615d2e15971d5aa525bcd

The second error is failure to push to ACR:

2023/02/03 01:00:57 input:1: container.build.publish failed to solve: failed to push ***/odafe/builder:0.12.0: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized

Any ideas what I'm doing wrong?

raven dove
#

Where is this defined? PublishToAzure

#

For example i calculate the image name and this part is really important. Just want to make sure it's correctly setup. Maybe replace your registryname with "tacos" and leave the rest and post here so I can see what you are providing?

solemn plover
raven dove
#

maybe try running the step manually like this issue shows and see if it works? Maybe your running into the fun of a bug in a prebuilt task and just using the cli directly will work better? https://github.com/Azure/docker-login/issues/52#issue-1468342301

GitHub

I am trying to use this code to push a .NET 7 container image to my ACR: - name: Azure Container Registry Login uses: Azure/docker-login@v1 with: username: ${{ secrets.ACR_USERNAME }} password: ${{...

#

Sounds like maybe that's related. If your path is fully qualified then I did this today but the docker login was an azure pipelines task. Maybe it's just an issue with the task itself for you. 🤞 s

solemn plover
#

Pushing to Azure Container Registry solved using "azure/docker-login" in the GitHub Action. Thanks for all your help, @raven dove , @half hamlet , and @mortal ingot !
Next up: pulling helm charts from bitbucket, tweaking version strings, then pushing back to bitbucket. Fun,fun,fun. 😁

solemn plover
#

Pull from bitbucket, bump version, push back to bitbucket from GitHub Action and Dagger using Go is now done and works. 🎉

@raven dove - I'm having a heck of a time trying to "helm upgrade" a chart to my Azure dev cluster from Dagger using Go... you wouldn't happen to have experience with this, would you?
I spent a full day trying to get "kubectl" to behave, and completely gave up and removed it... however I'm pretty sure that just installing and using "helm" will not work properly if I don't have "kubectl" set up properly.
Do you have any tips or pointers for me? (I see that Dagger has a Cue/Azure implementation, but I'm not sure if I can take advantage of that.)

half hamlet
#

@raven dove what kind of problems are you seeing? Errors? Or silent failures? Something else?

solemn plover
#

My current problem is understanding how to "helm upgrade"... do I only need helm? Do I also need "kubectl"?
If I only use helm alone, I get this error message: "Error: UPGRADE FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused".

#

If helm requires the use of "kubectl", it looks like the "kubelogin" plugin is also required for Azure, and getting all this set up correctly looks like a royal pain, so I was hoping that @raven dove could give me some pointers.

mortal ingot
#

@solemn plover happy to provide assistance with the errors you're getting to authenticate to your cluster

solemn plover
#

So is the "kubelogin" plugin also required for Azure?

mortal ingot
solemn plover
#

Yes, it's an AKS cluster. OK, thanks. I'll try without then.

solemn plover
#

"az login" now succeeds with "Login successful." from the GitHub Action: https://gist.github.com/gmlewis/0c4ef08f51bb74e8869ee20654d5e90a
but the "az aks get-credentials" now fails from within the Go Dagger workflow: https://gist.github.com/gmlewis/a88ba51d2589d88f1a55960fd260404b
with the error message:

#73 sh -c az aks get-credentials -n *** -g Development
#73 1.780 ERROR: Please run 'az login' to setup account.
#73 ERROR: process "sh -c az aks get-credentials -n *** -g Development" did not complete successfully: exit code: 1
2023/02/11 01:08:27 input:1: container.from.withExec.withMountedDirectory.withWorkdir.withEnvVariable.withEnvVariable.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withWorkdir.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withEnvVariable.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.stdout process "sh -c az aks get-credentials -n *** -g Development" did not complete successfully: exit code: 1
Stdout:

Stderr:
ERROR: Please run 'az login' to setup account.

Any ideas why? Are there more env vars that I need to pass in to the Dagger workflow that I'm missing?

mortal ingot
#

👋 from mobile. Since you're doing az login in the GHA action, you then need to mount the $HOME/.azure folder in your dagger pipeline so get-credentials work inside Dagger

solemn plover
#

After 5 days of trial-and-error, I have come to the conclusion that having more than one .WithMountedDirectory on a Container causes my GitHub Actions workflow to fail without any useful error messages after running for 3-6 hours (!).

So here is my first conceptual question I'm having difficulty understanding:
In my GitHub Actions, I'm running my Dagger workflow itself within a uses: actions/setup-go@v3 which makes me think... isn't my dagger Host also a Container ?
If so, why don't I have all the nice Container methods on my Host object like WithExec ? (In other words, why must I create a Go Container in Dagger if my Host is already a Go container?)

Secondly, I'm now struggling with trying to figure out how I can mount the /home/runner/.azure directory (set up by the GitHub Actions workflow) into my Dagger "Go" Container (under its ${HOME} which is /root) so that my Dagger workflow stops complaining about "please run az login"... all without using a second .WithMountedDirectory. Any ideas? (If you need gists to figure out what I'm talking about, I'll have to build them up, as the code is now spread across multiple files.)

mortal ingot
#

re: isn't my dagger Host also a Container ?

Well.. technically no since your local host is completely outside the minimal container sandboxing properites ie.

  • It doesn't have a base image
  • It's not a OCI layer itself. i.e: if you call .Push(<address>), will you push the entire host FS?
  • Content addressable caching doesn't work

and there are some other things like this. Having said that, it doesn't mean that cannot eventually extend Host to allow some other commands like Run to enable running things locally instead of having to do that outside the Dagger API. IIRC there's been some discussions about this but not enough traction/prioritization yet. cc @half hamlet

#

causes my GitHub Actions workflow to fail without any useful error messages after running for 3-6 hours (!).
definitely seems like an abnormal behavior. happy to help troubleshoot what could be happening here

solemn plover
#

OK, that makes sense, @mortal ingot - thanks for the explanation!
I can't spend any more time trying to figuring out .WithMountedDirectory problems now, so I'm thinking of making a gzip'd tarball of /home/runner/.azure on my Host and then tar xf dot-azure.tgz -C /root it on my Go Container. I'll let you know how it works. ~Copying dot-azure.tgz into my Go Container is going to be weird, I think... but I'll try to hack on it.~ Edit: WithNewFile might be what I need.

raven dove
#

I always set timeouts to avoid never ending jobs. Good practice imo to avoid using up a lot of runner minutes. My default is 15 mins because anything past that is typically a problem.

mortal ingot
#
    client.Host().Directory("/", dagger.HostDirectoryOpts{
        Include: []string{"/root", "/home/runner/.azure"},
    })
solemn plover
#

Hmm... I'm not sure I understand that, but thank you. I'll try it later.
Nevertheless, I now got Zip+Base64+WithNewFile+Unzip working.
My next snag is getting "az aks get-credentials" working.

I'm doing this:

        WithExec([]string{"sh", "-c", fmt.Sprintf("az aks get-credentials -n %v -g Development", azureClusterName)}).
        WithExec([]string{"sh", "-c", "kubelogin convert-kubeconfig -l azurecli"}).
        WithExec([]string{"sh", "-c", "kubectl config get-contexts"}).
        WithExec([]string{"sh", "-c", fmt.Sprintf("kubectl config use-context %v", azureClusterName)})
#

but am getting this error:

#75 sh -c az aks get-credentials -n *** -g Development
#75 1.794 ERROR: (AuthorizationFailed) The client 'c2effca7-39b6-4bf1-9c6b-5dd28805d21b' with object id 'c2effca7-39b6-4bf1-9c6b-5dd28805d21b' does not have authorization to perform action 'Microsoft.ContainerService/managedClusters/listClusterUserCredential/action' over scope '/subscriptions/***/resourceGroups/Development/providers/Microsoft.ContainerService/managedClusters/***' or the scope is invalid. If access was recently granted, please refresh your credentials.
#75 1.794 Code: AuthorizationFailed
mortal ingot
#

I'm not an azure expoert but seems like the creds you're getting don't have scope to get the aks cluster credentials?

#

does the az aks get-credentials actually work outside Dagger in the github action?

#

seems like it's picking up the .azure creds... and trying to use them

solemn plover
#

OK, news flash - my teammates said to stop pursuing this, and let "argo" update our dev cluster when a change is made to the helm charts repo... so I'm calling it a wrap and am no longer messing with "az login" and the rest of that.
Thanks for all your help again!

mortal ingot
#

😢 sorry to hear that this will go back to the YAML landscape. For future references, happy to jump into a quick audio channel to sort this out faster if needed @solemn plover

half hamlet
#

@solemn plover just to confirm, you're dropping this thread entirely, or just one aspect of it? No more dagger for you in the immediate future?

If I may ask, what would you say the root cause is the accumulation of issues you encountered here, or something more fundamental with Dagger's fit in your project? In other words, if everything had worked on the first try, would the outcome have been different?

Sorry that you ran into these issues by the way..

solemn plover
#

Sorry @half hamlet , just the final aspect of it... the final "helm upgrade" step was just a royal pain with the "az login"/kubectl/kubelogin stuff... so I'm only dropping that part of it.
I'm quite happy with Dagger... especially that I can use Go.
I will say that I think the Dagger autogenerated-godocs could possibly be improved by providing simple examples for each method... but otherwise think Dagger is amazing.
If this last "az login" step had been simpler and I could get it to work faster, then I would have kept and used it... but fortunately there is an easier option using "argo" so I can keep using Dagger for all the other parts.
Thank you for all your help! Now to switch all our repos over to using Dagger. 😁

mortal ingot
#

if this last "az login" step had been simpler and I could get it to work faster, then I would have kept and used it... but fortunately there is an easier option using "argo" so I can keep using Dagger for all the other parts.

if you still think this is something that you'd like to show your colleagues about how it works in Dagger, happy to put an example with GHA and an aks cluster to showcase how the az login would work in Dagger

solemn plover
#

@mortal ingot - I would love to see an example, and would also love to show them this in Dagger, but can't guarantee they'll use it, so please don't spend a lot of time on it for my sake. Thank you!

mortal ingot
#

@solemn plover do your devs also have access to some aks environment locally? Just trying to understand if it's useful for them or your team to be able to run this locally instead of through argo each time

solemn plover
#

So, yes, we all have local access to the aks environment... I've been so focused on getting the GitHub Actions pipeline working that I've rather neglected the "local" aspect of it all, unfortunately.
But yes, that would be awesome if it worked in both environments... I've just found that the GitHub Actions environment has been incredibly painful. 😞

mortal ingot
#

. I've just found that the GitHub Actions environment has been incredibly painful

What's the "biggest" main pain RN?

solemn plover
#

In my flow, I needed two "top-level" GitHub Actions before I start my Go Dagger container:

    steps:
      - name: Login to Azure Container Registry
        uses: azure/docker-login@v1
        with:
          login-server: ${{ secrets.DOCKER_HUB_REGISTRY }}
          username: ${{ secrets.DOCKER_HUB_USER }}
          password: ${{ secrets.DOCKER_HUB_PASSWORD }}
      - name: Login to Azure  # https://github.com/marketplace/actions/azure-login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS_DEVOPS_RG }}
      - uses: actions/setup-go@v3
        with:
          go-version: 1.19
      - uses: actions/checkout@v3
      - name: Cache node modules
        id: cache-yarn
        uses: actions/cache@v3
        env:
          cache-name: cache-node-modules
        with:
          path: node_modules
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('yarn.lock') }}
      - name: Run Dagger pipeline
        env:
          AZURE_CLUSTER_NAME: ${{ secrets.AZURE_CLUSTER_NAME }}
          BITBUCKET_SSH_PRIVATE_KEY: ${{ secrets.BITBUCKET_SSH_PRIVATE_KEY }}
          BITBUCKET_SSH_PUBLIC_KEY: ${{ secrets.BITBUCKET_SSH_PUBLIC_KEY }}
          DOCKER_HUB_REGISTRY: ${{ secrets.DOCKER_HUB_REGISTRY }}
          DOCKER_HUB_USER: ${{ secrets.DOCKER_HUB_USER }}
          DOCKER_HUB_PASSWORD: ${{ secrets.DOCKER_HUB_PASSWORD }}
          KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
        run: go run cicd/on-push-to-develop/main.go
#

And then, within my Dagger Go pipeline, I had to start a new Go Container with:

  • the contents of the current (GitHub) repo,
  • a "git clone" of the helm charts (cloned from Bitbucket),
  • a copy of the host's /home/runner/.azure directory placed within the Go Container's /root/.azure ${HOME} directory (from the top-level action)
    and getting all these permissions and copies all set up properly... I finally got all the pieces together, but got stuck internally within my Dagger workflow with the "az aks get-credentials" step having the wrong permissions/scopes.
solemn plover
#

I was just told by my devops team that they would like me to include "trivy" (https://github.com/aquasecurity/trivy-action) into my GitHub Actions pipeline.
Obviously, it would either run within my Dagger pipeline (after the image(s)) are built, or as the final step of the GitHub Actions pipeline.
The downside of the latter is that the Dagger pipeline itself is the one that figures out the exact name of the container image, so to make it work, I think I'll have to call it from within the Dagger workflow.
Just FYI. 😄 Thanks again for Dagger!

GitHub

Runs Trivy as GitHub action to scan your Docker container image for vulnerabilities - GitHub - aquasecurity/trivy-action: Runs Trivy as GitHub action to scan your Docker container image for vulnera...

#

(I'm OOO all next week in case you don't hear from me in a while.)

mortal ingot
#

for what I see in the readme seems like "scanning a tarball" could be the proper way to use it since Export in dagger should give you what you need for such purpose.

raven dove
#

I have everything setup for grype but not sure the value since I already run Mend(Renovate) for dependencies scanning. Be sure to post another thread with details if you end up seeing some use out of it.

mortal ingot