How to debug why my dagger function isn't working | Dagger | Page 1

fierce crow Apr 26, 2024, 12:32 PM

#

I'm using a CLI tool called Okteto which deploys preview environments. I want to create a dagger function that outputs the URLs for the deployed preview environments. To do that I have to run the CLI commands for deploying a preview environment, then wait for that command to get finished, then run another command which retrieves the endpoints of the deployed preview environment.

Here's the code I have currently:

func (m *OktetodoDagger) PreviewEnv(ctx context.Context,
    // Repo to deploy
    repo string,
    // Branch to deploy
    branch string,
    // URL of the pull request to attach in the Okteto Dashboard
    pr string,
    // Okteto context to be used for deployment
    context string,
    // Token to be used to authenticate with the Okteto context
    token string) (string, error) {
    c := m.SetContext(context, token)
    c.WithExec([]string{
        "okteto", "preview", "deploy", "--branch", branch, "--sourceUrl", pr, "--repository", repo, "--wait", branch,
    })
    c.WithExec([]string{
        "okteto", "preview", "endpoints", branch,
    })
    return c.Stdout(ctx)
}

But when I run the function with dagger call preview-env --repo=https://github.com/okteto/todolist-pulumi-s3 --branch=name-change --pr=https://github.com/okteto/todolist-pulumi-s3/pull/2 --context=arsh.okteto.me --token=$OKTETO_TOKEN all I see as the output is:

!  Initializing context with the value of OKTETO_TOKEN environment variable
 ✓  Context 'arsh.okteto.me' created
 ✓  Using rinkiyakedad @ arsh.okteto.me

Which is the output of the SetContext dagger function which the PreviewEnv function calls. I'm not sure how to debug this, can someone please help me figure out how to get this working? 🙏

daring ruin Apr 26, 2024, 3:07 PM

#

Hey @fierce crow 👋 There's a couple of things here. First is that since the SDK uses a builder pattern to construct the dag, so a WithExec does not change c in place, it returns a modified version of c. To use the modified version of c, reassign c like so:

c = c.WithExec([]string{
        "okteto", "preview", "deploy", "--branch", branch, "--sourceUrl", pr, "--repository", repo, "--wait", branch,
    })
c = c.WithExec([]string{
        "okteto", "preview", "endpoints", branch,
    })

This is a good example for when 'chaining' works well. The following is equivalent:

c = c.WithExec([]string{
        "okteto", "preview", "deploy", "--branch", branch, "--sourceUrl", pr, "--repository", repo, "--wait", branch,
    }).
    WithExec([]string{
        "okteto", "preview", "endpoints", branch,
    })

The next problem/solution in a follow up comment because of discord limits

#

The next problem is that c.Stdout() returns the output of the last command, not all commands, which is probably not ideal for this use case. I think there's 2 good ways to achieve what you want here!

Option 1, concat the stdout from both commands:

func (m *OktetodoDagger) PreviewEnv(ctx context.Context,
    // Repo to deploy
    repo string,
    // Branch to deploy
    branch string,
    // URL of the pull request to attach in the Okteto Dashboard
    pr string,
    // Okteto context to be used for deployment
    context string,
    // Token to be used to authenticate with the Okteto context
    token string) (string, error) {
    c := m.SetContext(context, token).
      WithExec([]string{
        "okteto", "preview", "deploy", "--branch", branch, "--sourceUrl", pr, "--repository", repo, "--wait", branch,
    })
    deployOut, err := c.Stdout(ctx)
    if err != nil {
      return "", err
    }
    c = c.WithExec([]string{
        "okteto", "preview", "endpoints", branch,
    })
    endpointsOut, err := c.Stdout(ctx)
    if err != nil {
      return "", err
    }
    return deployOut + endpointsOut, nil
}

The second option in a follow up comment because discord limits 😅

#

The second option is to redirect the output of the important commands to a file, and return the contents of that file. This can be done with an extra option to WithExec: https://pkg.go.dev/dagger.io/dagger#ContainerWithExecOpts

Here's how that would look:

func (m *OktetodoDagger) PreviewEnv(ctx context.Context,
    // Repo to deploy
    repo string,
    // Branch to deploy
    branch string,
    // URL of the pull request to attach in the Okteto Dashboard
    pr string,
    // Okteto context to be used for deployment
    context string,
    // Token to be used to authenticate with the Okteto context
    token string) (string, error) {
    c := m.SetContext(context, token).
      WithExec([]string{
        "okteto", "preview", "deploy", "--branch", branch, "--sourceUrl", pr, "--repository", repo, "--wait", branch,
    }, ContainerWithExecOpts{ RedirectStdout: "/stdout" }).
      WithExec([]string{
        "okteto", "preview", "endpoints", branch,
    }, ContainerWithExecOpts{ RedirectStdout: "/stdout" })
    
    return c.File("/stdout").Contents(ctx)
}

fierce crow May 3, 2024, 8:15 AM

#

Thanks for the detailed reply Kyle, super helpful! Some follow up questions:

Is there a place in the docs where this information lives so I can read up more?
Does one WithExec wait for the previous command to finish running when they are chained? Is this behaviour different when they are not chained?

#

@daring ruin I have the basic module now working! I need some help with other things, could we get on a call to work out this integration? What I primarily need help with is:

publishing the dagger module
getting it to run in CI (I saw the docs: https://docs.dagger.io/integrations/642920/github-actions, do I need a Dagger Cloud account to run in GitHub actions)
How do I get the output from dagger and have a bot comment on the pull request with the endpoints I got from the output. Is this something that can be done within Dagger only?

#

I found this: https://daggerverse.dev/mod/github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d but looks like it only allows commenting on GitHub issues and not PRs

github-comment :: Daggerverse

Use github-comment as a Dagger module.

daring ruin May 3, 2024, 3:14 PM

#

Is there a place in the docs where this information lives so I can read up more?
That specific part is only in the SDK docs right now, but it totally makes sense for us to add it to the cookbook section

Does one WithExec wait for the previous command to finish running when they are chained? Is this behaviour different when they are not chained?
Each operation on a container is executed sequetially, since you're building a DAG after all. This is true no matter how the object is constructed (chained or not)

publishing the dagger module
Hopefully this helps https://docs.dagger.io/guides/821742/publish-modules
The gist of it is you push the code to a public GitHub repo, give it a semver tag, and then enter that repo + tag here: https://daggerverse.dev/publish

It will also get published automatically when it gets executed by anyone. So if you dagger call the module from github, like dagger -m github.com/foo/bar@v1.0.0, that will cause it to publish

getting it to run in CI
Dagger Cloud is completely optional. It provides advanced visualization that GitHub can't. If you're not using Cloud, just omit that cloud-token line from the config. The action itself is just a convenience composite action, so it's optional as well. It really just manages downloading the dagger cli, executing the command, and cleaning up the engine afterwards

get the output from dagger and have a bot comment on the pull request
Totally possible, the action you linked might be able to do it. From fuzzy memory of doing this kind of thing in the past, I think issue comments and PR comments are the same on GitHub's side. If it doesn't work, maybe we can make a PR on that module to add the functionality

fierce crow May 6, 2024, 1:26 PM

#

Okay I guess before proceeding with anything else I should try that module out and see if it's commenting on PRs. I tried running:

dagger -m=github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call create –-github-token=ghp_asdjij1312j3b1k23jk -–owner=rinkiyakedad –-repo=okteto-dagger-sample -–issue=1 comment -–body "hello world"

But I keep getting this error that I haven't set the right flags:

✘ initialize 11.8s
  ✔ ModuleSource.asModule: Module! 9.7s
  ✔ Module.initialize: Module! 0.6s
    ✔ exec go build -o /runtime . 0.4s
    ✔ exec /runtime 0.1s
Error: required flag(s) "github-token", "owner", "repo" not set

Which I dont understnad because I have actually specified them.

Even when I run the --help for create I get the same error message:

$ dagger -m=github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call create --help
✘ initialize 0.2s
  ✔ ModuleSource.asModule: Module! 0.1s
Error: required flag(s) "github-token", "owner", "repo" not set

arctic maple May 6, 2024, 6:28 PM

#

I think I know what is wrong, but I am curious @rose pollen does daggerverse.dev examples support the constructor pattern - I think the example (https://daggerverse.dev/mod/github.com/samalba/aluzzardi-daggerverse/github-comment@3e86a5c24de5bb066487d3f8930e826a4f03cad1) is actually wrong because it seems these should be passed before the function is called.

@fierce crow I think you may have leaked your github token - id suggest you change it as soon as you can 🙂

Can you try changing the order of your arguments to look like this:

dagger -m github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call –github-token env:GITHUB_TOKEN –owner aluzzardi –repo daggerverse –issue 1 comment –body “hello world”

github-comment :: Daggerverse

Use github-comment as a Dagger module.

rose pollen May 6, 2024, 7:23 PM

#

arctic maple I think I know what is wrong, but I am curious <@274903880343748619> does dagger...

Hey Lev, the examples do support the constructor (or at least should ahah). It seems to be correct though, the --github-token and the -- owner and --repo are all before the function call in those examples

arctic maple May 6, 2024, 7:24 PM

#

rose pollen Hey Lev, the examples do support the constructor (or at least should ahah). It s...

Oh oops so sorry - I misread it!

rose pollen May 6, 2024, 7:25 PM

#

~~However, something is weird: it seems that there is a comment function available, not exposed on the page~~.

But the confusion comes from the comment left from Andrea: https://daggerverse.dev/mod/github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d#GithubComment.delete where the order is mismatched

daring ruin May 6, 2024, 7:47 PM

#

rose pollen Hey Lev, the examples do support the constructor (or at least should ahah). It s...

I'm not sure what you mean @rose pollen , the flags are after call

Screen_Shot_2024-05-06_at_3.47.07_PM.png

rose pollen May 6, 2024, 7:50 PM

#

daring ruin I'm not sure what you mean <@274903880343748619> , the flags are after call

Mmmh, the constructor arguments for the github-token need to be called prior the first function call, in your example it is the create function, doesn't it ? 🤔

daring ruin May 6, 2024, 7:51 PM

#

d'oh, sorry, yes they need to be between call and create 👍

rose pollen May 6, 2024, 7:53 PM

#

But I agree that this is a bit confusing, should we improve the readability by using CLI usage syntax ? [] for optionality <> for mandatory arguments requiring changes from user ? Though on a copy-paste it is a bit more work (or maybe making it a new line ?)

daring ruin May 6, 2024, 7:59 PM

#

rose pollen But I agree that this is a bit confusing, should we improve the readability by u...

I'm on board with trying a separate line for constructor args vs function args

fierce crow May 7, 2024, 1:53 PM

#

@arsh I think you may have leaked your github token - id suggest you change it as soon as you can 🙂

That's not the actual token 🙂

#

I'm lost 😕

First of all should these be -repo or --repo (similarly for all) I tried both and none worked.
Secondly what is this way of specifying the GitHub token: –github-token env:GITHUB_TOKEN I don't understand it. The token I have is in the form ghp_asdjij1312j3b1k23jk so should I be doing –github-token env:ghp_asdjij1312j3b1k23jk or –github-token ghp_asdjij1312j3b1k23jk

I ran these two commands and both failed with the same error:

$ dagger -m github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call –github-token env:ghp_blah123123asdasd –owner rinkiyakedad –repo okteto-dagger-sample –issue 1 comment –body “hello world”
✘ initialize 9.2s
  ✔ ModuleSource.asModule: Module! 7.2s
  ✔ Module.initialize: Module! 0.5s
    ✔ exec go build -o /runtime . 0.4s
    ✔ exec /runtime 0.1s
Error: required flag(s) "github-token", "owner", "repo" not set
$ dagger -m github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call –-github-token env:ghp_askdnas123123KJSKDJ -–owner rinkiyakedad –-repo okteto-dagger-sample –-issue 1 comment –-body “hello world”
✘ initialize 0.2s
Error: required flag(s) "github-token", "owner", "repo" not set

daring ruin May 7, 2024, 3:47 PM

#

Sorry @fierce crow we may have added to the confusion there!

It should be --repo (two dashes) for all flags
Since the github-token input is a type Secret, it can't be passed by value directly. By default, a Secret will refer to an environment variable containing the sensitive value, such as GITHUB_TOKEN. So you'd want to export GITHUB_TOKEN=ghp_abcdefg and then use --github-token env:GITHUB_TOKEN. You can also use a file or a command as a source for a secret. More here: https://docs.dagger.io/manuals/developer/typescript/527581/secrets/

Secrets | Dagger

Dagger allows you to utilize confidential information, such as passwords, API keys, SSH keys and so on, in your Dagger Modules and Dagger Functions, without exposing those secrets in plaintext logs, writing them into the filesystem of containers you're building, or inserting them into the cache.

fierce crow May 7, 2024, 4:06 PM

#

Thanks a lot Kyle, that clears a lot of things 🙂

Still running into the same error though:

$ dagger -m github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call –-
github-token env:GITHUB_TOKEN -–owner rinkiyakedad –-repo okteto-dagger-sample –-issue 1 comment –-body “hello world”
✘ initialize 19.0s
  ✔ ModuleSource.asModule: Module! 16.6s
  ✔ Module.initialize: Module! 0.7s
    ✔ exec go build -o /runtime . 0.5s
    ✔ exec /runtime 0.2s
Error: required flag(s) "github-token", "owner", "repo" not set

daring ruin May 7, 2024, 4:08 PM

#

fierce crow Thanks a lot Kyle, that clears a lot of things 🙂 Still running into the same ...

Can you confirm which version of dagger you have? dagger version. The command looks correct to me

fierce crow May 7, 2024, 4:08 PM

#

dagger v0.10.2 (registry.dagger.io/engine) darwin/arm64

daring ruin May 7, 2024, 4:09 PM

#

fierce crow dagger v0.10.2 (registry.dagger.io/engine) darwin/arm64

Just to rule it out, can you upgrade to latest? (0.11.2). brew upgrade dagger if you installed via brew

fierce crow May 7, 2024, 4:14 PM

#

Update to: dagger v0.11.2 (registry.dagger.io/engine) darwin/arm64

Same error with a slightly different format of the output 😛

$ dagger -m github.com/aluzzardi/daggerverse/github-comment@f17b8131ea25ff620606041134a5f1b2b5a7799d call –-github-token env:GITHUB_TOKEN -–owner rinkiyakedad –-repo okteto-dagger-sample –-issue 1 comment –-body "hello world"
✘ initialize 35.9s
! required flag(s) "github-token", "owner", "repo" not set
  ✔ Module.initialize: Module! 4.1s
    ✔ exec go build -o /runtime . 3.8s

Error: required flag(s) "github-token", "owner", "repo" not set

Is there a way to know if the module is working or not? Like if we're doing something wrong or maybe the module isn't working.

daring ruin May 7, 2024, 4:22 PM

#

Ok tested on my end copy/pasting your command and found 2 things:

the function should be create not comment even though the help text says comment. This is a bug in the module and I'm guessing the function was renamed at some point
There's weird dashes in there. At first I thought it was discord's copy/paste but I think it's legit. Each flag has 1 normal dash and 1 weird dash for some reason

Also confirmed the module works on PRs and issues 🙂

#

Screen_Shot_2024-05-07_at_11.23.26_AM.png

fierce crow May 7, 2024, 4:29 PM

#

There's weird dashes in there. At first I thought it was discord's copy/paste but I think it's legit. Each flag has 1 normal dash and 1 weird dash for some reason

WDYM? Should I be using - instead of -- for all flags? Can you please send me the command you ran

daring ruin May 7, 2024, 4:31 PM

#

It should be --, however if you look at the screenshot it seems like the command I copied from your message has some character that is not the normal - character

fierce crow May 7, 2024, 4:34 PM

#

Ah it's Discord messing with us I think. (or my terminal) But I got it working: https://github.com/RinkiyaKeDad/okteto-dagger-sample/pull/1#issuecomment-2098862721

Thanks! Now to chaining these to pass the output of the preview environment command to this one 🚀

Thanks a lot Kyle!! 🙏

GitHub

changing name of the todolist by RinkiyaKeDad · Pull Request #1 · R...

daring ruin May 7, 2024, 4:34 PM

#

Amazing! 🚀

fierce crow May 7, 2024, 5:40 PM

#

@daring ruin is there a way to force stop caching? Right now it's caching the output of the command and when I run it it doesn't actually deploy the environment but instead just gives me the endpoints it gave last time

daring ruin May 7, 2024, 5:44 PM

#

fierce crow <@135620352201064448> is there a way to force stop caching? Right now it's cachi...

Here's the best way to do that today https://archive.docs.dagger.io/0.9/cookbook#invalidate-cache
We're in the process of moving the doc I linked to the current docs. The key part is
WithEnvVariable("CACHEBUSTER", time.Now().String()).. Your module code will get executed every time, however if all inputs to the pipeline are the same as before, it will be cached. Adding an environment variable with the current timestamp is a way to guarantee the steps after that line will always get executed

Cookbook | Dagger

Filesystem

fierce crow May 7, 2024, 6:28 PM

#

Thanks! Also I published the Dagger module and got it working in the GitHub action 🎉

jobs:
  build:
    name: build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      -
        name: Call Dagger Function
        uses: dagger/dagger-for-github@v5
        with:
          version: "latest"
          verb: call
          module: github.com/okteto/dagger-module@v0.0.2
          args: preview-deploy --repo=https://github.com/${GITHUB_REPOSITORY} --branch=$GITHUB_HEAD_REF --pr=github.com/${GITHUB_REPOSITORY}/pull/${PR_NUMBER} --context=https://arsh.okteto.me --token=${{ secrets.OKTETO_TOKEN }}

What I want to ask is what's the best way to get the output of this command and pass it to the github comment dagger module? Can this be done with just one step in the GitHub action by chaining dagger commands or do I need to store the output first of this one step and then run another step which uses dagger and pass that stored output?

daring ruin May 7, 2024, 6:45 PM

#

good question! Personally, I would have one function call that does both. So in okteto/dagger-module i'd install the github-comment module and either
a) call that function at the end of your preview-deploy function if certain optional args are passed in, like --github-token or even a bool flag like --comment
b) write a new function that first calls preview-deploy and then calls github-comment create and call that new function in the github action

covert shard May 8, 2024, 5:42 AM

#

This whole thread is so great!
Awesome work @fierce crow 🎉
It would be great to get you on a Dagger Community Call 🙂
@storm pumice

I'll also DM you about Dagger <> Okteto opptys

storm pumice May 8, 2024, 4:17 PM

#

hi @fierce crow! Great to see you here 🙂 I'll DM you.

fierce crow May 8, 2024, 6:35 PM

#

Thanks! Replied to you both on DMs 🙂

fierce crow May 8, 2024, 7:13 PM

#

Personally, I would have one function call that does both
The problem I'm struggling with in that approach is that the command becomes just TOO long. Like there would be 9 flags to pass - isn't that alot? Plus wouldn't introducing one module as a dependency for my modeul mean I'll have to continusouly monitor if the other module is working or not? Or is getting updates, etc.

Is there a sample on the other approach (calling two dagger functions in different steps)? Do you know how I can save the output from one dagger call in a GitHub action step and pass it to the other?

daring ruin May 8, 2024, 7:25 PM

#

Plus wouldn't introducing one module as a dependency for my modeul mean I'll have to continusouly monitor if the other module is working or not?
Whenever you install a module as a dependency, it's automatically pinned at a git sha so that it will not change unless you want it to

command becomes just TOO long
Yeah, that is an issue for sure. Here's a github issue discussing ways we might solve that within dagger: https://github.com/dagger/dagger/issues/6723

The workaround for that is basically what it sounds like you're leaning towards already, which is scripting something that calls 2 different functions. That could look something like:

dagger call ... preview-deploy ... > deploy_output.txt
dagger call ... github-comment ... create --body $(cat deploy_output.txt)

fierce crow Jun 5, 2024, 3:23 PM

#

Hey @daring ruin! Sometimes I'm seeing errors when running Dagger in GitHub actions, any idea what could be causing this? It's not too often but has happened only twice in the 20+ runs I did maybe

daring ruin Jun 5, 2024, 3:37 PM

#

Interesting I think that has to do with the ghcr, cc @quasi bear

quasi bear Jun 5, 2024, 3:46 PM

#

Hey @fierce crow! Unfortunately we've been seeing people encounter this issue somewhat frequently for the past few days (cc @shrewd tartan). I'll ran some tests and contact our provider to see if we can find what is causing them

fierce crow Jun 5, 2024, 3:47 PM

#

Thanks @quasi bear! It occurs rarely only so not a huge bother for me 🙂

#

Another issue I'm facing is that the dagger function to destroy the preview environment keeps running even after the preview environment has been destroyed.

#

This is the output of running the okteto preview destroy command which is what the dagger function calls:

 ✓  Repository 'okteto-dagger-sample' successfully destroyed
Kind=Action 60dadb6a-6f72-4235-8554-fd5b8485316f destroyed
Kind=Action fc3a3b1f-cd1e-43c4-a4e0-8f337a2ef5d2 destroyed
Kind=Pod server-5ccfbbb495-fjws2 destroyed
WARNING: Failed to process metrics.k8s.io/v1beta1 PodMetrics rinkiyakedad-patch-8 server-5ccfbbb495-fjws2: the server does not allow this method on the requested resource
Namespace successfully deleted
 ✓  Preview environment 'rinkiyakedad-patch-8' successfully destroyed

daring ruin Jun 5, 2024, 4:10 PM

#

fierce crow This is the output of running the `okteto preview destroy` command which is what...

Are you able to share the code for that function?

fierce crow Jun 5, 2024, 6:13 PM

#

Here you go: https://github.com/okteto/dagger-module/blob/main/dagger/main.go#L96-L110

I added the --wait=false flag on line 104 bec of which it's now working but it should still work without that flag too I feel

GitHub

dagger-module/dagger/main.go at main · okteto/dagger-module

A Dagger Module for Okteto. Contribute to okteto/dagger-module development by creating an account on GitHub.

daring ruin Jun 5, 2024, 6:30 PM

#

Got it! Sorry not familiar enough with the okteto cli (yet 😉 ), but it sounds like the okteto destroy process isn't closing/returning rather than the dagger function itself, does that sound right?

fierce crow Jun 6, 2024, 9:21 AM

#

Yes that's what I was thinking but I'm not sure what "closing" actually means. When I just run okteto preview destroy I get the output I pasted above - that does prove that the process ended right?

daring ruin Jun 6, 2024, 2:53 PM

#

Yeah as long as the process exits then dagger should continue

fierce crow Jun 6, 2024, 3:18 PM

#

Then if the process is exiting on running the command locally then why not when inside a Dagger container? 🤔 Do you know how I can investigate more to learn where the issue might actually be?

#How to debug why my dagger function isn't working