#Trying out a module, need some help

1 messages · Page 1 of 1 (latest)

delicate cargo
#

this is what I have so far (untested)

# 
package main

import (
    "context"
)

type Scan struct{}

func (m *Scan) Snyk(ctx context.Context, ctr *Container, snykToken *Secret) (*Container, error) {

    c := ctr.
        WithWorkdir("/tmp").
        WithExec([]string{"curl", "https://static.snyk.io/cli/latest/snyk-linux", "-o", "snyk"}).
        WithExec([]string{"chmod", "+x", "snyk"}).
        WithExec([]string{"mv", "./snyk", "/usr/local/bin"}).
        WithWorkdir("/src").
        WithSecretVariable("SNYK_TOKEN", snykToken).
        WithExec([]string{"snyk", "monitor", "--all-projects"
})
    return c, nil
}
#

How do I invoke this from a pipeline?

#

I tried

...
    _, err = runner.
        Scan().
        Snyk(ctx, snykSecret)

which returns

runner.Scan undefined (type *dagger.Container has no field or method Scan)
#

Trying out a module, need some help

pliant crypt
#

@delicate cargo are you trying to invoke Scan() from the same module that defines it? I think for technical reasons, the codegen doesn't support it (chicken and egg problem)

#

More generally, end-user invocation is still rough and not finished. Note that in the zenith README, it only mentions raw graphql queries with dagger query as a means of testing your module

#

So in your case:

dagger query <<EOF
{
 Scan {
  Snyk("mysecret"} {
   id
  }
 }
}
EOF
delicate cargo
hard gull
#

@delicate cargo assuming the Scan module is in ./scan and the client module is in ./, I think you'd want to run dagger mod extend ./scan first. That should add it as a dependency and re-generate your client code so you can call it

#

(disclaimer: haven't actually tried that yet! and we'll probably rename that command)

delicate cargo
#

I have the Scan module in one location (zenith branch clone as per README) and the invoking pipeline is in a different location (app directory). Would dagger mod extend work if the custom module is in a different (non contiguous) directory?

hard gull
#

It should but I'm not 100% sure. A relative or absolute path should work fine

pliant crypt
#

Re: changing the name. What is the equivalent command name for npm, yarn, go? Is there a clear winner?

delicate cargo
# hard gull It _should_ but I'm not 100% sure. A relative or absolute path should work fine

Tried this, running dagger mod extend from my app dir with the additional -m parameter as it errored out otherwise

dagger mod extend /home/vikram/public/examples/zenith-branch/zenith/vikram-scan/ -m /home/vikram/public/examples/zenith-branch/zenith/vikram-scan/

It's running but doesn't seem to be getting anywhere, now 8m and counting. It seems to be stuck at asModule(). Is it expected that it would take so long or is there an error?

hard gull
#

hmm that looks like it would extend the module with itself. the -m should be the client module that you're extending

#

it should default to your working directory, if you run it from the client module

#

if not that's a 🐛

delicate cargo
#

OK I see

#

When i run it from the project I'm trying to build, I see an error

vikram@dagger:/tmp/snyk-demo-todo$ dagger mod extend /home/vikram/public/examples/zenith-branch/zenith/vikram-scan/
Error: failed to get module config: failed to read local config file: open /tmp/snyk-demo-todo/dagger.json: no such file or directory
#

Do I need a dagger.json with my pipeline?

hard gull
#

yeah - the client caller is also a module, so you need to dagger mod init it too

delicate cargo
#

Tried that in the project, result:

vikram@dagger:/tmp/snyk-demo-todo$ dagger mod init
Error: unsupported module SDK:
hard gull
#

need to pass --name=foo and --sdk=go like the other module

#

For some context: there is no plan for "naked" use of modules - it's always one module calling another module. The start of your Dagger journey in the future would be to create a module. Given enough refinement it should feel a lot more natural, and it'll become a lot more accessible once we get back to defining high-level entrypoints and such (like dagger checks, dagger artifacts).

delicate cargo
#

OK I see

# 
vikram@dagger:/tmp/snyk-demo-todo$ dagger mod extend /home/vikram/public/examples/zenith-branch/zenith/vikram-scan/
WARNING: Using development engine; skipping version compatibility check.                                           
WARNING: Using development engine; skipping version compatibility check.                                           
WARNING: already bootstrapped                                                                                      
• Engine: b8053c12eeb8 (version devel ())
⧗ 25.44s ✔ 27 ∅ 1 ✘ 1
#

Not quite sure what happened here, it seems to have hit an error but didn't display any message

#

This is the error

go mod tidy
20: [1.43s] main/node_modules/snyk-go-plugin/gosrc imports
20: [1.43s]     ./resolver: "./resolver" is relative, but relative import paths are not supported in module mode
20: [1.43s] main/node_modules/snyk-go-plugin/gosrc/resolver imports
20: [1.43s]     ./dirwalk: "./dirwalk" is relative, but relative import paths are not supported in module mode
20: [1.43s] main/node_modules/snyk-go-plugin/gosrc/resolver imports
20: [1.43s]     ./graph: "./graph" is relative, but relative import paths are not supported in module mode
20: go mod tidy ERROR: exit status 1
#

This is weird, I'm not sure why it's picking up stull from the app node_modules dir

#

temp fix by removing those node_modules

# 
vikram@dagger:/tmp/snyk-demo-todo$ dagger mod extend /home/vikram/public/examples/zenith-branch/zenith/vikram-scan/
WARNING: Using development engine; skipping version compatibility check.                                                                 
WARNING: already bootstrapped                                                                                                            
• Engine: b8053c12eeb8 (version devel ())
⧗ 7.51s ✔ 16 ∅ 4

now succeeds

#

but running the pipeline after still displays

# 
 runner.Scan undefined (type *dagger.Container has no field or method Scan)
red vapor
#

since your function is func (m *Scan) Snyk(ctx context.Context, ctr *Container, snykToken *Secret) (*Container, error), I think instead of

_, err = runner.
        Scan().
        Snyk(ctx, snykSecret)

you want

_, err = dag.Scan().Snyk(ctx, runner, snykSecret)
#

Last week Erik talked about being able to extend core types in modules (like Container to be able to chain as you originially had), but I don't think that's in yet

pliant crypt
red vapor
#

oh wait maybe it is in, I haven't tried it yet. Will report back

pliant crypt
#

Erik seemed confident when he mentioned it

red vapor
pliant crypt
#

General vibe: “oh you wanted room temperature fusion? Why didn’t you say so, here you go buddy enjoy”

#

“diamonds? hold on I think I have one in my shoe actually “

red vapor
#

The lesson is we need to ask for the impossible more often 🤣

#

so in that case @delicate cargo , what you should do is instead update your Snyk function to look like func (ctr *Container) Snyk(ctx context.Context, snykToken *Secret) (*Container, error) and the pipeline should work as-is

pliant crypt
#

incredible

#

I see possible naming conflicts though

red vapor
#

Yeah definitely, I think Erik brought that up as well. Can't remember what the plan was

pliant crypt
#

My naive thought:

  • Module snyk registers Container.Scan
  • Module docker-scout registers Container.Scan
  • My module imports both
  • My module sees Container.SnykScan and Container.DockerScoutScan
#

More elaborate, possibly even more naive option:

  • Container.Snyk().Scan and Container.DockerScout().Scan
red vapor
#

As long as it's done in a predictable way, whatever works I guess 🤷 The SnykScan feels better, although still plenty of room to break things if you're malicious 🤠 My module will be called With

pliant crypt
#

lol

#

a whole new creative medium awaits

#

Are you having second thoughts on extending core types @red vapor ?

#

Snyk.Scan(mycontainer) ?

red vapor
#

Not at all, I think being able to chain modules with core types makes it worth solving this

pliant crypt
#

cold fusion it is

red vapor
#

got it, working on the Adobe ColdFusion SDK now

pliant crypt
#

objectively a solid idea for a module 😁

red vapor
#

is that still a thing? my first dev job was writing CF. Really weird

delicate cargo
red vapor
delicate cargo
#

Module code looks like this

# 
package main

import (
    "context"
)

type Scan struct{}

func (ctr *Container) Snyk(ctx context.Context, snykToken *Secret) (*Container, error) {

    c := ctr.
        WithWorkdir("/tmp").
        WithExec([]string{"curl", "https://static.snyk.io/cli/latest/snyk-linux", "-o", "snyk"}).
        WithExec([]string{"chmod", "+x", "snyk"}).
        WithExec([]string{"mv", "./snyk", "/usr/local/bin"}).
        WithWorkdir("/src").
        WithSecretVariable("SNYK_TOKEN", snykToken).
        WithExec([]string{"snyk", "monitor", "--all-projects", "--org=5e86b410-1a77-462a-a352-901a216fc3a6"})

    return c, nil
}
delicate cargo
red vapor
#

and I think you need dagger mod sync on both projects if you haven't

delicate cargo
#

ah, maybe that's it. I did it on the scan module but not on the other.

#

Nope. dagger mod sync works on the Snyk module, but fails on the main app

dagger mod sync --progress=plain ERROR: failed to load dependencies: failed to serve dependency module input:1: host.directory.asModule.serve failed to install module schema: schema validation failed: input:1726: Undefined type Scan.
red vapor
#

that's weird since it shouldn't be looking for Scan anymore 🤔

delicate cargo
#

Did a quick search and I do see references to Scan in dagger.gen.go but I don't know if these are expected or an error

# 
type Scan struct {
    q *querybuilder.Selection
    c graphql.Client
}

func (r *Scan) Snyk(ctr *Container, snykToken *Secret) *Container {
    q := r.q.Select("snyk")
    q = q.Arg("ctr", ctr)
    q = q.Arg("snykToken", snykToken)

    return &Container{
        q: q,
        c: r.c,
    }
}```
#

(however my pipeline doesn't use Scan() anywhere.)

#

wrapping up, will come back to this tomorrow - thanks for all your help Kyle, Alex, Solomon!

pliant crypt
#

Very exciting to see a real module being built 🙂

#

This motivated me to try building my own first module, too

delicate cargo
#

Nice!

#

Hopeful to get this working soon, I want to turn it into a "build your first dagger module" guide if I can

pliant crypt
#

Can't wait for a "build your first dagger module" module 😛

delicate cargo
#

any hints why dagger mod sync might fail with this error on the caller mod @hard gull?

ERROR: failed to load dependencies: failed to serve dependency module input:1: host.directory.asModule.serve failed to install module schema: schema validation failed: input:1726: Undefined type Scan.
hard gull
#

hmm nothing off the top of my head, but happy to take a look if you want to push all this code somewhere temporary

delicate cargo
#

ok, sure - thanks!

hard gull
#

@delicate cargo btw, you can try passing --focus=false when you run into a cryptic error, it'll give you full output instead. still need to do fine-tuning there

#

what are the steps to repro? dagger mod sync from snyk-demo-todo repo?

delicate cargo
#

Yes

#

You might need to adjust some paths in dagger.json

hard gull
#

i swapped it out for a git:// ref now that that's fixed

delicate cargo
#

I wonder if the issue is that it needs at least one method for the Scan struct

pliant crypt
#

Had the same problem

delicate cargo
#

Ah ok

#

I'll try that tomorrow unless Alex finds something else

hard gull
#

lol, i found an even more heinous bug, if I run dagger mod sync in dagger-scan it clobbers main.go and goes into an infinite go mod tidy loop chef_kiss

#

not sure why you haven't run into that

delicate cargo
#

Nope, I didn't see that. It worked super smoothly in dagger-scan

delicate cargo
#

Pulled the latest code and dagger mod sync in scan module, dagger mod use in pipeline module are now both working

#

However, when I try to invoke the module from the pipeline, I still see [1.16s] ERROR go run ci/main.go ┃ # command-line-arguments ┃ ci/main.go:41:3: runner.Snyk undefined (type *dagger.Container has no field or method Snyk) ┻

delicate cargo
#

I have it working using graphql though

dagger q -m "git://github.com/vikram-dagger/dagger-scan?ref=main" << EOF
query test {
        container {
                from(address: "alpine") {
                        withExec(args: ["apk", "add", "curl"]) {
                                withExec(args: ["apk", "add", "git"]) {
                                        withExec(args: ["git", "clone", "https://github.com/snyk/snyk-demo-todo.git", "/src"]) {
                                                snyk(token: "TOKEN", path: "/src") {
                                                        stdout
                                                }
                                        }
                                }
                        }
                }
        }
}
EOF
hard gull
delicate cargo
#

would it be helpful to add this scan/snyk example to your README @hard gull?

#

if yes, I can write it up quickly before the demo today...lmk

hard gull
delicate cargo
#

I'm not sure what you mean by dependency example?

#

I wasn't able to get it working from my main Go pipeline after all

hard gull
red vapor
delicate cargo
#

trying out dependency stuff, will add more as I get it working

hard gull
#

nice!

red vapor
delicate cargo