#Capturing state from service bound containers

1 messages · Page 1 of 1 (latest)

gleaming stone
#

👋 I've been noodling on some problems with capturing state (particularly logs) from containers run as bound services.

Here is what I want to solve:

We have an integration test suite and it exercises our application through its API.
In dagger we stand up our own service using the same *dagger.Container (via the Go SDK) that we ultimately (intend to) publish.
This container is bound to our integration test container which runs a Go test suite (https://github.com/flipt-io/flipt/blob/main/hack/build/internal/test/integration.go#L20-L25) using our own apps Go SDK (clients) against this running service.
(You might also note that I spin up this combination integration suite + our app in various configurations to ensure compatibility in a number of scenarios)

This past 24 hours I've been trying out ways to manage the log output of each of the containers we create.
At the moment, as you know, it is all just muxed together on stdout.

Our existing integration test suite (a bunch of bash) sinks our services logs into a file, while the test suite sits at the front of our Github action logging on stdout.
When the test ends, we use github actions upload artifact feature to push our applications logs into an action artifact:
https://github.com/flipt-io/flipt/blob/main/.github/workflows/integration-test.yml#L93-L99

This allows us to keep the test output at the forefront and remove the noise of the application logs.
However, when the test fails, we have somewhere to go for the application logs and the missing context.

[Breaking this up to get around the Discord word limit]

#

I've tried a couple of things to recreate this experience with Dagger and no luck yet.
Most of which, in retrospect, I think I understand why they didn't work.

  1. I tried re-using the container instance after passing it to WithServiceBinding() in order to access the log file via .File().Export().

Didn't work and I think I understand why. I read somewhere Export() works on the generated layers. I assume I cannot get hold of any resulting layers produced by running the container as a concurrently running service. I assume this ephemeral containers state is never committed to any layers in the first place? Seems like the difference between a step in a Dockerfile and docker run. I don't get a layer out of the latter.

  1. I tried mounting a host directory and writing into that.

This part was problematic for our setup. We define a non-root user and group for our resulting image/container.
Which ultimately has led to this failing for permissions reasons. Maybe there is some trick I can do here.

I suspect if I had a little more control wrt dagger output, e.g. controlling log output at the container level of fidelity, I could solve this problem.
Is there a design or proposal in this area? (is this what a Pipeline might ultimately help with?)
I feel like I've either seen one or seen glimpses of conversations related to this in this Discord, but I am at a loss this morning.

Thanks 🙏

wheat pine
#

have you seen that already?

gleaming stone
#

Hey @wheat pine 👋 I did momentarily play with using a mounted cache instead of a host directory.
However, I had some similar permissions problems to what I saw in (2).

Correct me if I am wrong in how I might leverage this for my problem, but this is what I tried (If I remember correctly):

Create a cache volume (either one shared or one per service) and mount that into my service bound container.
Once I have finished my test suite, run another container and mount in that cache volume(s).
Copy the contents of the volume into a layer? So that I can then export it locally using .File().Export().

If I remember correctly I had the same permissions problems attempting to write into the mounted directory.
Maybe that is a work around here to allow my non-root container user to access a mounted cache?

wheat pine
#

seems like you're very likely writing and reading to/from the cache volumes with different UID's

#

you need to validate that the container (s) that write and read from the cache mount have the correct permissions or do some chmodding somewhere

gleaming stone
#

Ah interesting 🤔 for a mounted cache volume, what permissions does it get / who owns it?
Does it result in getting mounted as root owned into the container?
Or does something else happen?

#

Maybe I just need to chown the cache directory before I use it in this way (I will have a play)

wheat pine
gleaming stone
#

I will report back 😄

wheat pine
#

I recall there's an issue where we still don't allow chmodding it through the withMountedCache call, but we should allow that

#

so you have to do it manually in WithExec for now

gleaming stone
#

That did it 🙂 thank you for the help 🙏