Local e2e test with parallel long running service containers

[earnest axle]

Hello everyone,

I'm working for the dyrector.io company which is an open-source self-hosted CD platform with version management.
This is our site: https://dyrectorio.com
And our github repository: https://github.com/dyrector-io/dyrectorio

Currently we are evaluating Dagger for our use cases (mainly with its Go SDK) and would prepare a POC for local e2e testing and self-hosted Github actions workflow too.

We are keen on achieving the same setup with Dagger, allowing us to run everything locally within containers using the Dagger engine and leveraging its extremely fast caching mechanism.

While exploring Dagger, we tried to review all relevant information in the Dagger docs, especially the Cookbook and Go SDK sections, but also read all the posts on the Dagger Blog and help here on Discord.
(expose ports, cache dependencies, service containers, persist service state between runs, understand the service lifecycle, connecting to docker engine on the host, etc)

There are very interesting and relevant existing discord posts and github discussions and issues,
perhaps this latest PR is the most interesting (+it's a really nice piece of work ^^):

• Services v2: new API, c2h and h2c networking, start/stop support, run via gateway: https://github.com/dagger/dagger/pull/5557

We haven't been able to fully implement our local e2e POC yet, but based on the information provided, we are considering waiting for the release of Dagger 0.9.x for the services v2 new API PR.
We have two questions related to this:

• 1. Do you have a planned release date for Dagger 0.9?
     We are eager to know as we believe it will be beneficial for our project.
• 2. As an alternative, is it possible to achieve our task using Dagger 0.8.4?
     We are wondering if we can creatively combine the existing building blocks to fulfill our requirements without necessarily relying on Dagger 0.9.
     (if it is not absolutely necessary and/or if it still takes a long time to release it)

.

Additional information for context:

Our e2e test environment involves several containers, including Traefik, Kratos, a NestJS-based backend, a Next.js-based frontend, and dedicated Postgres databases for the backend and Kratos. We also have migration containers as part of our setup.
When these are launched, our e2e test with Playwright will start.

Currently when we run the e2e tests locally, the backend and frontend run in separate terminals, while Playwright runs in a 3rd terminal from the frontend directory, so these 3 are not containers locally, just everything else.
In the case of Github workflow, everything runs within containers.

These are very interesting and relevant existing discord posts:
(there are more, but this post is already way too long ^^)

• Run multiple containers at same time: https://discord.com/channels/707636530424053791/1035090714951942236
• Running containers locally / re-inventing the wheel: https://discord.com/channels/707636530424053791/1143138103125618718

And following the above posts these github discussions and issues:

• Running tasks parallel
  • https://github.com/dagger/dagger/discussions/5644
  • https://github.com/dagger/dagger/issues/4766
  • https://github.com/jpadams/dagger-examples/blob/parallel/go/npm-build/ci.go
• Services v2: new API, c2h and h2c networking, start/stop support, run via gateway: https://github.com/dagger/dagger/pull/5557

.

[cyan mist]

Welcome Imre! Glad to hear that you're evaluating Dagger to help with your e2e test suite. Let us know whatever we can do to make that happen!

1. Not a hard date but we generally do releases often whenever things become ready. @runic kindle's PR is feature complete and currently under review, so I'd say it'll go out the door anytime.

2. For what I understand, you should be able to get started with v0.8.4 and then leverage on the upcoming release for the missing bits and pieces. The overall Dagger API won't change that much in the next release, it's mostly adding some new capabilities and some internal refactoring