Cyclical Dependency between container and services

[wanton ivy]

Hi,

So at my current employer there is an established pattern where the integration tests expose mock APIs that the API container will query as needed. This allow the integration tests to mock services dynamically based on the tests being ran. This means that the Integration Test would need to be exposed as a Service, but the integration test itself depends on the API.

We currently use docker-compose and since a running container can act as a service/expose ports it works fairly well.

flowchart TD
    Tests[Integration Tests] --API Test call--> API
    API --Mocked API A--> Tests
    API --Mocked API B--> Tests

I am pretty sure it is impossible to model with the current Dagger API (a service that depends on a service), but even if it was possible we would also run into and issue where integration test container cannot be a service if we want to extract files like tests reports at the end of a run. If it was possible to extract files from a service that would also work.

So my question, is there anything in buildkit that would prevent us from:

1. Having cyclical dependencies between services expose a running container.
2. Extract files from a service.

Thanks,

JM

[charred prawn]

#1164508519588958348 message

#1180660724452818985 message

@restive osprey any reasons we couldn't extend the Dagger API to allow export of Directory or File from Service containers? see posts just above ☝️

Comment from user:

integration test container cannot be a service if we want to extract files like tests reports at the end of a run. If it was possible to extract files from a service that would also work.

I see the "use volumes" trick mentioned and then "might not need volumes anymore" mentioned. Not sure of current state or what's possible.

[restive osprey]

In the past it was a technical limitation, but something I'm coincidentally working on at this exact moment will lift that limitation, so yeah I think it would make sense at this point.

It would still currently be required that the Service is still running, which makes it a bit tricky for the case of automatically started/stopped services, but we have support for manual start/stop now too so it would make sense there. I think this would have to be in the bucket of "advanced use cases" since the fact that the service must still be running is a requirement on the user, not just us, but that's probably okay to start out with.

cc @scenic igloo I think the stuff I'm writing to enable serving nested sessions from the executor could be re-used here, specifically the filesync part of it

[charred prawn]

🙏

[restive osprey]

I don't think there's a GH issue for this, so would appreciate one that we can attach to the right project

This does come up fairly often on discord so worth not just letting it go stale in the backlog forever

[charred prawn]

I'll create one now

https://github.com/dagger/dagger/issues/7377

[wanton ivy]

Ah, yes, the volume trick should work; I keep forgetting about it. That being said, we would probably also need some kind of method for waiting for the service (in our case, the integration test) to be done.

Something like:

        startedService, err := service.Start(context.Background())
        if err != nil {
            return nil
        }
        // Wait doesn't exist for Service
        startedService.Wait()
        // File doesn't exist for Service
        file := startedService.File("out")

Going back to the second question in the initial post. Would there be any way to have cyclical dependencies between services?

Something like:

    apiSvc := apiCtr.AsService()
    integrationSvc := integrationCtr.WithServiceBinding("api", apiSvc).AsService()
    // WithServiceBinding doesn't exist for Service
    apiSvc.WithServiceBinding("integration", integrationSvc)

I am not convinced the references would even be valid since we mutate (or create a new ref) the apiSvc after using it in the integrationSvc.

[rugged acorn]

Going back to the second question in the initial post. Would there be any way to have cyclical dependencies between services?

current not possible by any means unfortunately. Having said maybe it's not that bad idea to have a method in the Service type like WithHostname which you could set that allows overriding the service hostname so it can be reached from other containers witout explicitly using WithServiceBinding?
cc @scenic igloo

[scenic igloo]

(Catching up!) - hmm yeah it seems like cycles can't be represented at the moment. You can almost do it by manually starting the services instead using .endpoint(), but passing the endpoint to one services changes the endpoint the other needs to use; the cycle is still there. So yeah, it seems like we'd need a way to pin that hostname like you're suggesting.

Alternatively services need to be reworked so that cycles can be represented. The DAG approach is convenient but not a perfect match for how networks really work. I think @elder junco had some thoughts on this, but don't remember a TODO/issue

[wanton ivy]

ou can almost do it by manually starting the services instead using .endpoint(), but passing the endpoint to one services changes the endpoint the other needs to use; the cycle is still there. So yeah, it seems like we'd need a way to pin that hostname like you're suggesting.

For my own understanding, currently, are all the services running in a DAG able to communicate with each other as long the endpoint is known? Even if they are not directly bound with WithServiceBinding? I am assuming all containers/services in a DAG are running in the same network.

In more concrete terms, so if we have:

        svcA := apiCtr.AsService()
        svcB := integrationCtr.WithServiceBinding(svcA)
        svcC := someCtr.WithServiceBinding(svcB)
                svcC.Start(ctx)

In this case, if we pass the endpoint (or hostname) svcA to svcC and knowing that svcA must be up for svcC start, could svcC connect to svcA without explicitly setting WithServiceBinding(svcA) (it's a bit of a pointless example)? If so we would indeed need a way to expose a hostname to the whole containers' network.

[rugged acorn]

For my own understanding, currently, are all the services running in a DAG able to communicate with each other as long the endpoint is known? Even if they are not directly bound with WithServiceBinding? I am assuming all containers/services in a DAG are running in the same network.

yes, this is correct but you still have the same issue because the endpoint is derived from the definition of the service. So in order for svcA and svcB to know about each other, you're still blocked by the cycle dependency thing.

[scenic igloo]

Yeah, you wouldn't be able to pass the endpoints to each other, because the endpoints would "change" in the process. You can almost express it in code but I'm pretty sure you'd end up with a "zombie" service and a broken endpoint. You could maybe do some weird reverse-proxy thing with dynamically registered endpoints, but that's a lot of work 😅

[rugged acorn]

lol.. thought about the same preverse-proxy approach but it felt super convoluted as well.

[elder junco]

my best guess is we would need a new core type that is the equivalent of a docker network. It takes a list of containers and implements a tightly coupled group of services that can all reach other. It can't be built on top of Service, has to be a parallel implementation of the same stuff - inject dns, /etc/hosts, etc.