How to Structure my E2E Test-Pipeline

[bleak wolf]

I am very new to dagger and still confused on how I should structure my pipeline. I have three different repos (frontend, backend, e2e-tests). My rough goal would be:

• When the frontend or backend repo get an update on «main», I would like to trigger the e2e pipeline
• The pipeline needs 4 containers: frontend, backend, database, playwright
• The frontend and backend containers should be built as docker files from the respective repos
• The actual test then basically composes the four containers using withService and runs the tests

Here's the current version of my TypeScript module: https://gist.github.com/stefnnn/6dafdfec0ad035a0cc01820c4255a055

Here are some of my current questions:

1. The e2e-repo has the frontend and backend repo integrated as git submodules. Traditionally I would simply checkout the e2e-repo include submodules and then have all the code available in my pipeline. Is this the way to go with dagger, or should I better directly reference the other repos?

2. When I want to refer to a directory on the host, what's the currently recommended way. Is my current approach of defining a module parameter "dir" ok? To me it doesn't much sense to introduce this parameter, as this is not a reusable module that I want to share with the world, but it seems that's the only way to make a host directory accessible in the module code. However, my @func list_dir doesn't seem to work.

3. How do I build a docker image from a specific Dockerfile in the host directory? What I'm doing in the gist:54 above is obviously not working.

If you can give me some pointers for some of those questions I'd be glad to continue my quest!

[tiny hill]

Hay @bleak wolf!!! Great questions. I'm checking out the module you shared and will reply back in just a bit!

[tiny hill]

1. I think this depends on whether you want your e2e-tests repository to have pinned versions of the frontend and backend on each commit that happens to the repo itself and be used as a central place for debugging. Meaning that when an e2e-test fails, the process to debug is to download the e2e-test repo, checkout the commit and debug the frontend/backend component within the context of the e2e repo. An alternative I can think of is to call your dagger module with a specific ref for each component and have the dagger module clone the repository at the specified ref. The benefits of doing this is that you essentially remove any custom scripts (i.e updating git subdmoules before calling dagger, etc). In this scenario, the only things that you have within your e2e-repo is your actual tests + the dagger module that orchestrates the whole thing. It's a bit cleaner since there are less moving pieces, however, I believe the most important thing to consider is what I mentioned before: how/if do you want developers to interact with the e2e tests repository?

2. The approach you are currently taking were you explicitly specify the directory is exactly what you need to do . Specifying function paramters is unrelated to whether you plan on sharing the module or not. Dagger function calls are executed in an isolated environment within the context of the Dagger engine, they do not have any host-specific information of where the actual dagger call function happened. There is some info that the dagger CLI automatically sends, but it is limited and it is only about the context of the module itself. Here you can see a more in depth explanation of the architecture: https://docs.dagger.io/manuals/developer/overview/482011/architecture. Lets illustrate with an example: you have an "SERVER_PORT" env variable in your host that you want your dagger function to use. If you call your function and then within the function print the contents of getenv("SERVER_PORT") you will see that it is empty, since the function call is happening inside a container started by the container of the dagger engine. Every parameter that your function requires needs to be explicitly specified

3. Using dir.dockerBuild is the right way of buliding a container from a dockerfile. Could you share what it is that is not working in this case?

[bleak wolf]

Thank you so much @tiny hill for the answer. I'm currently on an Easter break but I'll definitely continue on my quest with your suggestions and report back here on this channel 👍

[bleak wolf]

Ok, I managed to progress on this @tiny hill . Here are a few more questions I stumbled upon:

1. How can I authenticate requests to github (i.e. see https://gist.github.com/stefnnn/6dafdfec0ad035a0cc01820c4255a055#file-index-ts-L38), I know how I can pass a secret (GitHub PAT), but I'm unsure how to use this with the dag.git() command. I think it would be helpful to update the example in the docs as well: https://docs.dagger.io/manuals/developer/typescript/527581/secrets/

2. What's a good way to log something. When I do console.log within a @func, I can see this somewhere burried deep within the dagger cloud logs, but not really in my terminal

3. Is there anything I need to configure for caching to work (locally)? It seems that every run takes a long time, even if I do zero changes.

4. Is there a way to expose a service to more than one containers? It looks like I need to expose my frontend to the playwright container, which in turn is exposed to the backend container, ideally both should have access to the frontend service.

Would be great, if someone could give me some further pointers 🙂

[still jasper]

Hey @bleak wolf 👋

1. How can I authenticate requests to github
   Maybe the following module could help you.
   It is used for git with ssh authentication
   https://daggerverse.dev/mod/github.com/puzzle/dagger-module-git-actions/git-actions@ba354569d342d9ce311cadad88391048c610d584

2. What's a good way to log something. When I do console.log within a @func, I can see this somewhere burried deep within the dagger cloud logs, but not really in my terminal

Did you try to append the verbose flag -v when running your command?
you can also increase verbosity with -vv, -vvv, and you can also use --debug to show debug logs and full verbosity.
https://docs.dagger.io/manuals/developer/go/924957/debugging

[grand kernel]

👋 re 1: you can do something like this (go code):

// Returns lines that match a pattern in the files of the provided Directory
func (m *Lala) Test(ctx context.Context, username string, password *Secret) (*Directory, error) {
    ppwd, err := password.Plaintext(ctx)
    if err != nil {
        return nil, err
    }

    return dag.Git(fmt.Sprintf("https://%s:%s@%s", username, ppwd, "github.com/marcosnils/rooster")).
        Branch("master").Tree(), nil
}

at least until https://github.com/dagger/dagger/issues/6747 is implemented so you can use the sshAuthSocket in the ClientGitOpts object here: https://docs.dagger.io/reference/typescript/modules/api_client_gen#clientgitopts

[grand kernel]

3. Is there anything I need to configure for caching to work (locally)? It seems that every run takes a long time, even if I do zero changes.

no, there's not. Caching should work out of the box. Is there any way you can share what you're trying to do so we can check it out?

4. Is there a way to expose a service to more than one containers? It looks like I need to expose my frontend to the playwright container, which in turn is exposed to the backend container, ideally both should have access to the frontend service.

yes, you can just bind that service to as many containers as you need. So you can basically call withServiceBinding in yourbackend and playwright containers and use that frontend service in both. Does that make sense?

[bleak wolf]

Thanks so much @still jasper and @grand kernel for your suggestions!!

• "-debug" does help. However, I think it would be helpful, if this would disable the feature of automatically cleaning up the terminal output. I have opened a ticket for that (https://github.com/dagger/dagger/issues/7056)

• I tried authenticating with user:[e-mail]. Strangely I get an error "remote: Write access to repository not granted.", even tough I just want do dag.git().branch().tree()

• I will try exposing my service to multiple services. But I assume I must take care to always use the same container instance in the withServiceBinding call in order not to start multiple instances.

• As for caching, even my very simple @func listDir() from the above gist will take almost a minute to execute each time. The directory that I pass as an argument has about 80MB of content and doesn't change between calls.

[grand kernel]

"-debug" does help. However, I think it would be helpful, if this would disable the feature of automatically cleaning up the terminal output. I have opened a ticket for that (https://github.com/dagger/dagger/issues/7056)

using -v disables that

[still jasper]

using -debug + -v, right?
e.g: dagger call --dir $HOME/forks/dagger-zenith/simple-react-app list-dir -debug -v

[grand kernel]

-v only works for me as well

[still jasper]

actually -debug display everything for me

[grand kernel]

I tried authenticating with user:[e-mail]. Strangely I get an error "remote: Write access to repository not granted.", even tough I just want do dag.git().branch().tree()

have you checked if the PAT has the right permissions? I've also tried using a PAT and works in my case. You could also try with git clone https://user:[e-mail]... to check if that works

[grand kernel]

yes, I've also noticed that debug doesn't clear the screen for me. Which Dagger version are you on @bleak wolf ? The latest v0.11.x release has some major changes in the CLI output

[bleak wolf]

Yes, after the update to v0.11 all lines will be shown at the end of the run 👍

[bleak wolf]

Ah darn, I was confused by the error message. It was indeed an issue with the PAT permissions and it now works.

[pale salmon]

That'll put the secret plaintext in cache though, right?

[grand kernel]

yes, that's correct

[pale salmon]

@bleak wolf, remember to update to using WithAuthToken in next release (https://github.com/dagger/dagger/pull/6992) or pass the URL via the CLI using the ssh protocol in the meantime.

[grand kernel]

thx Helder ❤️

[half quartz]

Hi, I was playing with the hello-dagger-template trying to run the Cypress E2E simple test included, using the Doc.

I just need to be guided with a simple usage on using Dagger to achieve this properly 👀

PS:

• I've red through the gist of this topic, but a bit way complex using Docker build and Playwright...

• Also had an unsuccessful try with Daggerverse Cypress module :
  https://daggerverse.dev/mod/github.com/quartz-technology/daggerverse/cypress@627fc4df7de8ce3bd8710fa08ea2db6cf16712b3#module-about

/**
 * Return the result of running Cypress e2e tests
 */
@func()
async cypress(@argument({ defaultPath: './' }) source: Directory): Promise<string> {
  return await dag.cypress(source).run()
}

[half quartz]

Using the daggerverse Cypress module seems to struggle with the source argument (which is project current directory...) 🤔

[half quartz]

Hacking a simple e2e test, stuck with the ⁨⁨The cypress npm package is installed, but the Cypress binary is missing.⁩⁩ 👁️

⁨⁨```typescript
/**

• Return the result of running Cypress e2e tests
  */
  @func()
  async e2e(@argument({ defaultPath: '/' }) source: Directory): Promise<string> {
  const service = await this.build(source).asService().start()

return await dag
  .container()
  .from('cypress/included:13.6.2')
  .withServiceBinding('nginx', service)
  .withDirectory('/e2e', source)
  .withWorkdir('/e2e')
  .withExec(['npm', 'run', 'test:e2e:ci', 'run'])
  .stdout()

}