Service container waiting for requests blocks pipeline from continuing | Dagger | Page 1

unique hornet Apr 12, 2023, 8:27 AM

#

Experimenting with services (specifically the vault agent api proxy: https://developer.hashicorp.com/vault/docs/agent/apiproxy). I've started a service with port 8100 exposed, and configured the agent to listen on port 8100. The service starts up and reaches it's idle state waiting for requests sent to it, but it blocks the pipeline from continuing at that point.

Service:

        vault_agent = (
            client.container()
            .from_("hashicorp/vault:1.13.0")
            .with_file("/home/vault/config.hcl", vault_config_file)
            .with_file("/home/vault/.vault-token", vault_token_file, 777)
            .with_entrypoint(
                [
                    "vault",
                    "agent",
                    "-config=/home/vault/config.hcl",
                ]
            )
            .with_exec([])
            .with_exposed_port(8100)
        )

(Config file contains, among other stanzas, a listener with address 127.0.0.1:8100 and tls_disable=true)

This sits idle when running the pipeline:

#0 0.350 2023-04-12T08:21:13.246Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
#0 0.350 2023-04-12T08:21:13.246Z [INFO]  agent.auth.handler: starting renewal process
#0 0.406 2023-04-12T08:21:13.302Z [INFO]  agent.auth.handler: renewed auth token

That is now going to wait indefinitely for requests.

This part of the pipeline is therefore never reached:

vault_client = (
            client.container()
            .from_("alpine:latest")
            .with_service_binding("agent", vault_agent)
            .with_exec([])
        )

@hazy obsidian Continued from my question in #general . It may well be that this isn't a suitable service, this is all experimental for now.

hazy obsidian Apr 12, 2023, 8:52 AM

#

I think I got it, you need to expose the port before the empty with_exec

#

Otherwise, the exec will run, but the port won't be open, and then, the depending service will try to connect to the service without the port open/service name discoverable, and so, it fails by waiting for it to open, but it won't because it is running in the foreground first, and then would expose the port.

unique hornet Apr 12, 2023, 8:57 AM

#

I'll try that now

#

Still idles in the service container unfortunately:

#5 0.474 2023-04-12T08:58:43.298Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
#5 0.474 2023-04-12T08:58:43.298Z [INFO]  agent.auth.handler: starting renewal process
#5 0.524 2023-04-12T08:58:43.348Z [INFO]  agent.auth.handler: renewed auth token

#

Interestingly, if I leave it for long enough a healthcheck fails and exits the process:

dagger.exceptions.QueryError: start 0IA0J5QQ6CKJK (aliased as agent): health check errored: exit code: 1

#

That error contains

gql.transport.exceptions.TransportQueryError: {'message': 'start 0IA0J5QQ6CKJK (aliased as agent): health check errored: exit code: 1', 'locations': [{'line': 9, 'column': 11}], 'path': ['container', 'from', 'withServiceBinding', 'withExec', 'stdout']}

#

Does that suggest the 'client' part of the code is running but it's not being shown in the output?

hazy obsidian Apr 12, 2023, 9:08 AM

#

Actually, it should also be put before the entrypoint, I believe. Have you done that?

unique hornet Apr 12, 2023, 9:11 AM

#

vault_agent = (
            client.container()
            .from_("hashicorp/vault:1.13.0")
            .with_file("/home/vault/config.hcl", vault_config_file)
            .with_file("/home/vault/.vault-token", vault_token_file, 777)
            .with_exposed_port(8100)
            .with_entrypoint(
                [
                    "vault",
                    "agent",
                    "-config=/home/vault/config.hcl",
                ]
            )
            .with_exec([])
        )

#

This still idles on the service, not sure if anything is happening in the background

hazy obsidian Apr 12, 2023, 1:28 PM

#

I can reproduce.
@late sun do you happen to know if there is something that could impact service discovery since 0.4.2?

late sun Apr 12, 2023, 2:32 PM

#

v0.4.2 and a patch or two earlier were actually affected by a nasty bug that caused services to exit, which was fixed in the latest release(s)

#

you could try setting _DAGGER_DEBUG_HEALTHCHECKS=1 to see what the healthcheck is saying (I'd like to route these logs somewhere nicer but it's not possible yet, so they're silenced for now)

unique hornet Apr 12, 2023, 3:32 PM

#

[check tj32ahmi3a5eeyyhzi4u6rgy3] port not ready: dial tcp 10.87.0.20:8100: connect: connection refused; elapsed: 10.426062879s

#

I get a lot of those, though by that point the service is idling ready to accept requests

late sun Apr 12, 2023, 3:43 PM

#

maybe Vault needs to be configured to listen on 0.0.0.0 instead of 127.0.0.1? 🤔

#

either that or the port is wrong?

hazy obsidian Apr 12, 2023, 3:44 PM

#

both good points

unique hornet Apr 12, 2023, 4:09 PM

#

The port is definitely correct in the config.hcl and the exposed port, I'll try 0.0.0.0 now

#

So at first glance that appears to have worked 🤦‍♂️

late sun Apr 12, 2023, 5:51 PM

#

nice. yeah, this might be worth including somewhere in the services docs; I think it's a common default, especially for security-focused things like Vault. cc @halcyon latch

halcyon latch Apr 12, 2023, 7:08 PM

#

unique hornet So at first glance that appears to have worked 🤦‍♂️

Sorry, I didn't completely understand this. Do you mean include a note that service containers should be configured to listen on 0.0.0.0 instead of 127.0.0.1?

unique hornet Apr 12, 2023, 7:11 PM

#

That may be a good idea. As @late sun has said, 127.0.0.1 is common in documentation (vault example: https://developer.hashicorp.com/vault/docs/agent/apiproxy#example-configuration) and it doesn't currently work with Dagger. Might save people a few hours 😅

late sun Apr 12, 2023, 7:11 PM

#

halcyon latch Sorry, I didn't completely understand this. Do you mean include a note that serv...

yep - the issue with 127.0.0.1 is it'll only be reachable within the container itself, so other services (including our healthcheck) won't be able to connect to it

#

0.0.0.0 on the other hand will allow connections to any destination IP, i.e. the container's 10.87.xx.xx IP in the Dagger network

unique hornet Apr 13, 2023, 10:01 AM

#

Curious, as an extension of this, is it possible to get files from service containers with something like:

with_file("<path_to_place_file>", service_container.file("<location_of_file_wanted>"))

#

I've tried that exact with_file and it causes the service to idle/freeze again

#Service container waiting for requests blocks pipeline from continuing