#Dagger local dependencies for test
real marsh
Seems like you have to reinstall the dependency every time with dagger install ..?
Think I'm also hitting some caching or something? It's really not clear why changes to the module are not reflected when using that module as a local dependency at all
real marsh
Hitting a wall and putting this down for now, if someone has any experience with it give me a shout!
dense ore
Seems like you have to reinstall the dependency every time with `dagger install ...
guessing this is because the dependency is pinned to a specific version
did you try dagger update?
real marsh
"dependencies": [
{
"name": "docker-builder",
"source": ".."
}
]
No pin, does Dagger pin behind the scenes?
fast hound
Weird, I haven't seen this behavior. I have many, same root dependencies and they update when the parent project is updated. Something else is going on here.
lethal grotto
https://docs.dagger.io/api/module-tests/?sdk=python#test-module
@winter ember any ideas?
@real marsh wondering where your git boundary for the context directory is too. What your overall monorepo structure looks like.
winter ember
I haven't seen anything like this either
real marsh
I'm basically copying this structure
.
├── internal
│ ├── dagger
│ ├── querybuilder
│ └── telemetry
└── tests
├── fixtures
└── internal
├── dagger
├── querybuilder
└── telemetry
11 directories
When I first followed the docs instructions above, all went just fine, but I made a mistake and redid things, then I started to get this error when I ran dagger install .. from the tests subdir
Error: failed to update dependencies: input: moduleSource.withDependencies local module dependency context directory "/Users/jeremyadams/src/greeter" is not in parent context directory "/Users/jeremyadams/src/greeter/tests"
Parent has to be a a git repository, I hit that earlier
lethal grotto
So I knew that the context dir was being defined by the dagger.json of tests and I was escaping, I think. Going to try to go up to greeter and git init.
Yep. Just wondering if you'd hit that already. 🙂
I'm guessing that maybe a change in caching of function calls is at play?
Looking. Going to try a cachebuster on the test function
Actually, I can't repro your issue. I changed one char in the greeter module and it was detected by my test. Didn't need to bust cache or anything.
from dagger import function, object_type
@object_type
class Greeter:
greeting: str = "Hellow"
@function
def hello(self, name: str) -> str:
"""Greets the provided name"""
return f"{self.greeting}, {name}!"
greeter ➤ dagger -m tests call hello
✔ connect 0.2s
✔ load module 0.5s
✔ tests: Tests! 0.7s
✘ .hello: Void 1.8s
! unexpected greeting
│ ✔ greeter: Greeter! 0.6s
│ ✔ .hello(name: "World"): String! 0.6s
I followed those docs above plus did the git init in the parent dir.
real marsh
I wonder what I'm seeing then. Here's the params for my function:
(
// Dockerfile path
// +defaultPath="./tests/fixtures"
Dir *dagger.Directory,
// Dockerfile name
// +default="Dockerfile"
File string,
// Platforms
// +default="linux/arm64"
Platform string,
// Build arguments
// +optional
Args []string,
)
And the usage of that function in test:
func (m *Tests) DefaultDockerfileBuild(ctx context.Context) error {
dir := dag.CurrentModule().Source().Directory("./fixtures")
_, err := dag.DockerBuilder().BuildImage(dir, "Dockerfile", "linux/amd64", nil)
return err
}
IDE error on dag.DockerBuilder().BuildImage(): ├╴ cannot use dir (variable of type *dagger.Directory) as dagger.DockerBuilderBuildImageOpts value in argument to dag.DockerBuilder().BuildImage compiler (IncompatibleAssign) [29, 43], and the definition for that function func (r *DockerBuilder) BuildImage(opts ...DockerBuilderBuildImageOpts) *Container { // docker-builder (../../../main.go:94:1) seems to explain where that's coming from
That same function is running just fine from the CLI:
build-image --dir ./tests/fixtures --file Dockerfile.other --platform linux/arm64 --args PLATFORM=linux/arm64 |
with-exec "sh,-c,env | grep PLATFORM" |
stdout
In your test can you try
dir, _ := dag.CurrentModule().Source().Directory("./fixtures").Sync(ctx)
I'm guessing that dir is being lazy and you might need to force it.
real marsh
Adding .Sync(ctx) works fine, IDE still complains about the parameters passed to .BuildImage
The test doesn't actually run to test the lazy evaluation:
./main.go:30:12: assignment mismatch: 2 variables but dag.DockerBuilder().BuildImage returns 1 value
./main.go:30:43: cannot use dir (variable of type *dagger.Directory) as dagger.DockerBuilderBuildImageOpts value in argument to d
ag.DockerBuilder().BuildImage
./main.go:30:48: cannot use "Dockerfile" (untyped string constant) as dagger.DockerBuilderBuildImageOpts value in argument to dag
.DockerBuilder().BuildImage
./main.go:30:62: cannot use "linux/amd64" (untyped string constant) as dagger.DockerBuilderBuildImageOpts value in argument to da
g.DockerBuilder().BuildImage
./main.go:30:77: cannot use nil as dagger.DockerBuilderBuildImageOpts value in argument to dag.DockerBuilder().BuildImage
Sounds like you need some golang opts magic, perhaps
real marsh
There's another (I think related) issue: the function returns (*dagger.Container, error) but my IDE is convinced it returns *dagger.Container
The dagger.gen.go signature does return *dagger.Container... But why?
On that note, the dagger.gen.go file still contains ContainerEcho and GrepDir but I deleted both ages ago
Is there some interaction with Git here that I'm forgetting, I need to commit it all?
lethal grotto
I don't think so, switching to golang to try to repro
real marsh
I can just send you my code to save time?
main.go:
// A generated module for DockerBuilder functions
//
// This module has been generated via dagger init and serves as a reference to
// basic module structure as you get started with Dagger.
//
// The first line in this comment block is a short description line and the
// rest is a long description with more detail on the module's purpose or usage,
// if appropriate. All modules should have a short description.
package main
import (
"context"
"dagger/docker-builder/internal/dagger"
"fmt"
"slices"
"strings"
)
var (
ValidPlatforms = []string{"linux/arm64", "linux/amd64"}
)
type DockerBuilder struct {
// AWS_ACCESS_KEY_ID value
AccessKey *dagger.Secret
// AWS_SECRET_ACCESS_KEY value
SecretKey *dagger.Secret
// Session token if available
// +optional
SessionToken *dagger.Secret
// AWS Region code
Region string
}
func (m *DockerBuilder) WithCredentials(
// AWS_ACCESS_KEY_ID
AccessKey *dagger.Secret,
// AWS_SECRET_ACCESS_KEY
SecretKey *dagger.Secret,
// AWS_SESSION_TOKEN
SessionToken *dagger.Secret,
// AWS Region code
// +default="eu-central-1"
Region string,
) *DockerBuilder {
m.AccessKey = AccessKey
m.SecretKey = SecretKey
m.SessionToken = SessionToken
m.Region = Region
return m
}
func (m *DockerBuilder) Aws(ctx context.Context) (*dagger.Container, error) {
if m.AccessKey == nil || m.SecretKey == nil {
return nil, fmt.Errorf(
"AccessKey and SecretKey must not be nil or empty. Use with WithCredentials method to set them.",
)
}
ctr := dag.Container().
From("public.ecr.aws/aws-cli/aws-cli:latest").
WithEnvVariable("AWS_REGION", m.Region).
WithSecretVariable("AWS_ACCESS_KEY_ID", m.AccessKey).
WithSecretVariable("AWS_SECRET_ACCESS_KEY", m.SecretKey)
if m.SessionToken != nil {
ctr = ctr.WithSecretVariable("AWS_SESSION_TOKEN", m.SessionToken)
}
return ctr, nil
}
func (m *DockerBuilder) BuildImage(
// Dockerfile path
// +defaultPath="./tests/fixtures"
Dir *dagger.Directory,
// Dockerfile name
// +default="Dockerfile"
File string,
// Platforms
// +default="linux/arm64"
Platform string,
// Build arguments
// +optional
Args []string,
) (*dagger.Container, error) {
if !slices.Contains(ValidPlatforms, Platform) {
return nil, fmt.Errorf("Invalid platform: %s. Only linux/arm64 and linux/amd64 are allowed", Platform)
}
var buildArgs []dagger.BuildArg
for _, arg := range Args {
parts := strings.SplitN(arg, "=", 2)
if len(parts) != 2 {
return nil, fmt.Errorf("Invalid build arg: %s, must be in name=value format", arg)
}
buildArgs = append(
buildArgs,
dagger.BuildArg{Name: parts[0], Value: parts[1]},
)
}
ctr := Dir.DockerBuild(dagger.DirectoryDockerBuildOpts{
Dockerfile: File,
BuildArgs: buildArgs,
Platform: dagger.Platform(Platform),
})
return ctr, nil
}
func (m *DockerBuilder) PushImage(
Ctr *dagger.Container,
AccountId string,
Tepository string,
// +optional
Tags []string,
) {
// password, err := m.Aws(ctx).WithExec([]string{"aws", "--region", m.Region, "ecr", "get-login-password"}).Stdout()
}
tests/main.go:
package main
import (
"context"
"dagger/tests/internal/dagger"
"github.com/sourcegraph/conc/pool"
)
type Tests struct{}
// Executes all tests.
// Run with `dagger call test`
func (m *Tests) Test(ctx context.Context) error {
p := pool.New().WithErrors().WithContext(ctx)
p.Go(func(ctx context.Context) error {
return m.DefaultDockerfileBuild(ctx)
})
p.Go(func(ctx context.Context) error {
path := dag.CurrentModule().Source().Directory("./fixtures")
return m.OtherDockerfileBuild(ctx, path)
})
return p.Wait()
}
func (m *Tests) DefaultDockerfileBuild(ctx context.Context) error {
_, err := dag.DockerBuilder().BuildImage()
return err
}
func (m *Tests) OtherDockerfileBuild(
ctx context.Context,
// +defaultPath="./fixtures"
Path *dagger.Directory,
) error {
_, err := dag.DockerBuilder().BuildImage(Path, "Dockerfile.other", "linux/amd64", nil)
return err
}
tests/fixtures/Dockerfile:
FROM alpine:latest
ENV DOCKERFILE=Dockerfile
Working my way down.
dagger -c 'build-image --dir ./tests/fixtures --file Dockerfile --platform linux/arm64 --args PLATFORM=linux/arm64'
Does the generated code update when you run dagger develop?
real marsh
The generated code for the module is correct, but the test module generated code doesn't accept the parameters
Neither dagger develop nor dagger update in the test submodule seem to help - my IDE still has issues with dag.DockerBuilder().BuildImage(): 2 variables but dag.DockerBuilder().BuildImage returns 1 value and cannot use nil as dagger.DockerBuilderImageOpts
lethal grotto
Neither `dagger develop` nor `dagger update` in the test submodule seem to help ...
similar issue
real marsh
Is this a bug in codegen or am I misconfiguring somewhere?
real marsh
Same two errors
lethal grotto
I wasn't seeing PLATFORM getting set, maybe, so maybe the condition in the BuildImage function is met and it returns (nil, error) which comes through as just error? Hypothesis
err
Maybe because we expect (*somedaggertype, error) to come from a call to dag that has ctx and BuildImage creates it's own error with fmt.Errorf... 
real marsh
How can I test that? Replace/remove fmt.Errorf?
shy canyon
@lethal grotto need any help here?
fast hound
This may be related - https://github.com/dagger/dagger/issues/8421
shy canyon
Just saw Vikram's ping
lethal grotto
<@933501536624054272> need any help here?
yes please!
shy canyon
k, let me catch up
lethal grotto
Module code with tests code above 🙂
shy canyon
Module code with tests code above 🙂
@lethal grotto @real marsh I don't see any issues here. This works as designed when complex types through module and the Go SDK
lethal grotto
After removing all of the *dagger.Container return types and just returning error in BuildImage I move the IDE errors to
shy canyon
we're talking about this error, correct?
lethal grotto
yes
shy canyon
^ just to re-iterate, this works as designed due to the laziness model of Dagger. More info here: https://github.com/dagger/dagger/issues/3617
also here's a very similar thread another user opened some time ago with the same confusion: #1324360448778178633 message
Alex's reply here pretty much summarizes it: #1324360448778178633 message
real marsh
Think I follow, seems like there's two issues here then? Returning (*dagger.Container, error) and that second issue?
shy canyon
Think I follow, seems like there's two issues here then? Returning `(*dagger.Con...
the return types is one thing. What's the other one?
this one?
real marsh
The submodule doesn't understand the parameters for the module under test. It won't accept actual parameters, insisting it should be a DirectoryDockerBuilderOptions
shy canyon
The submodule doesn't understand the parameters for the module under test. It wo...
yes that's also correct because all the arguments are optional
and the only way to express that in Go is via typed parameters
lethal grotto
So there's never an error to return from a client-side API that returns an object, even if the server-side implementation says it returns an error.
So if module function under test says it returns <- this seems to not be an issue(*dagger.Container, error) then we'll have this issue. We'll only receive the lazy container object if successful, not the error slot with nil in it, even if we force a Sync(ctx), right?
real marsh
So my next step is removing the (*dagger.Container, error) and just using *dagger.Container
shy canyon
So `there's never an error to return from a client-side API that returns an obje...
if you call Sync in the container, you'll get the actual error if the API returns it
since that will force the server side function execution
lethal grotto
the point of the test is to see if there is an error, so we kinda need that 🙂
welll one point is to see if error, then to see if correct
shy canyon
the important thing to note here is that any function that returns an object is lazy by default
real marsh
Call sync in the function under test?
shy canyon
Call sync in the function under test?
yes, exactly
Sync or any method that returns a scalar type so the DAG gets evaluated
lethal grotto
I was trying that a bit ago. Will try again after resetting all thrash 😄
func (m *DockerBuilder) BuildImage(
ctx context.Context,
// Dockerfile path
// +defaultPath="./tests/fixtures"
Dir *dagger.Directory,
// Dockerfile name
// +default="Dockerfile"
File string,
// Platforms
// +default="linux/arm64"
Platform string,
// Build arguments
// +optional
Args []string,
) (*dagger.Container, error) {
if !slices.Contains(ValidPlatforms, Platform) {
return nil, fmt.Errorf("Invalid platform: %s. Only linux/arm64 and linux/amd64 are allowed", Platform)
}
var buildArgs []dagger.BuildArg
for _, arg := range Args {
parts := strings.SplitN(arg, "=", 2)
if len(parts) != 2 {
return nil, fmt.Errorf("Invalid build arg: %s, must be in name=value format", arg)
}
buildArgs = append(
buildArgs,
dagger.BuildArg{Name: parts[0], Value: parts[1]},
)
}
ctr, err := Dir.DockerBuild(dagger.DirectoryDockerBuildOpts{
Dockerfile: File,
BuildArgs: buildArgs,
Platform: dagger.Platform(Platform),
}).Sync(ctx)
return ctr, err
}
@lethal grotto you shouldn't have to call Sync there since again, you're still returning an object from the server side function
Sync needs to be called in the test function
lethal grotto
ah, so that 👆 Sync() won't matter to the client, won't force the last err to be filled? I thought that was part of the 1 return val vs 2 issue.
Will try in test func.
package main
import (
"context"
"dagger/tests/internal/dagger"
"github.com/sourcegraph/conc/pool"
)
type Tests struct{}
// Executes all tests.
// Run with `dagger call test`
func (m *Tests) Test(ctx context.Context) error {
p := pool.New().WithErrors().WithContext(ctx)
p.Go(func(ctx context.Context) error {
return m.DefaultDockerfileBuild(ctx)
})
p.Go(func(ctx context.Context) error {
path := dag.CurrentModule().Source().Directory("./fixtures")
return m.OtherDockerfileBuild(ctx, path)
})
return p.Wait()
}
func (m *Tests) DefaultDockerfileBuild(ctx context.Context) error {
_, err := dag.DockerBuilder().BuildImage().Sync(ctx)
return err
}
func (m *Tests) OtherDockerfileBuild(
ctx context.Context,
// +defaultPath="./fixtures"
Path *dagger.Directory,
) error {
_, err := dag.DockerBuilder().BuildImage(Path, "Dockerfile.other", "linux/amd64", nil).Sync(ctx)
return err
}
more like that?
shy canyon
ah, so that 👆 Sync() won't matter to the client, won't force the last `err` to ...
yes, exactly. That Sync doesn't change things since it's the client the one that needs to unlazy the dag
shy canyon
more like that?
yep, yep. BuildiImage args still need fixing but yes, that's the general idea
lethal grotto
opts, right
real marsh
Can you link an example of the change needed to the args?
lethal grotto
func (m *Tests) OtherDockerfileBuild(
ctx context.Context,
// +defaultPath="./fixtures"
Path *dagger.Directory,
) error {
_, err := dag.DockerBuilder().
BuildImage(
dagger.DockerBuilderBuildImageOpts{
Dir: Path,
File: "Dockerfile.other",
Platform: "linux/amd64",
Args: nil,
},
).Sync(ctx)
return err
}
prob that? trying
yep @real marsh seems to work
dagger -m tests -c 'test'
package main
import (
"context"
"dagger/tests/internal/dagger"
"github.com/sourcegraph/conc/pool"
)
type Tests struct{}
// Executes all tests.
// Run with `dagger call test`
func (m *Tests) Test(ctx context.Context) error {
p := pool.New().WithErrors().WithContext(ctx)
p.Go(func(ctx context.Context) error {
return m.DefaultDockerfileBuild(ctx)
})
p.Go(func(ctx context.Context) error {
path := dag.CurrentModule().Source().Directory("./fixtures")
return m.OtherDockerfileBuild(ctx, path)
})
return p.Wait()
}
func (m *Tests) DefaultDockerfileBuild(ctx context.Context) error {
_, err := dag.DockerBuilder().BuildImage().Sync(ctx)
return err
}
func (m *Tests) OtherDockerfileBuild(
ctx context.Context,
// +defaultPath="./fixtures"
Path *dagger.Directory,
) error {
_, err := dag.DockerBuilder().
BuildImage(
dagger.DockerBuilderBuildImageOpts{
Dir: Path,
File: "Dockerfile.other",
Platform: "linux/amd64",
Args: nil,
},
).Sync(ctx)
return err
}
That one has Sync(ctx) on both tests
real marsh
That's good news. Makes the test a bit opaque from a user perspective though. Thanks for debugging this one!
lethal grotto
Thanks Profesor Marcos! 🙏
real marsh
Yep, that's working for me 🙂