#stdout with await

1 messages · Page 1 of 1 (latest)

urban creek
#

I'm using the python API for doing things like 'dagger call test'. Following the example here:
https://docs.dagger.io/quickstart/test
Works fine. However if i call my 'test' function from another function (e.g. https://docs.dagger.io/quickstart/publish) I no longer get the stdout for the unit tests. Is this expected? Adding -vv etc works fine for understanding the current state of execution, but it does not help me for example when all my tests fail to run because of some syntax error or something.
#general message

I guess my question is: Is there a recommended best practice (or utility) to running traditional CI tasks using call and seeing their stdout without needing to manually handle this, particularly when using async.

Run unit tests | Dagger

Run unit tests

spring ferry
#

Hi. I’m not entirely sure what you mean by no longer getting the stdout and needing to manually handle it. Can you clarify?

urban creek
#

My function looks something like this:

    @function
    async def test_typescript(self, source: dagger.Directory) -> str:
        """Run TypeScript unit tests"""
        return (
            await (
                self.build_typescript_env(source)
                .with_exec(["yarn", "run", "test"])
                .stdout()
            )
        )

When 'called' directly it works find (stdout is returned so displayed from Dagger terminal invocation). When it's called via another @function I cant see the output:

    @function
    async def build(self,
                    source: dagger.Directory) -> None:
        """Build the application container"""
        await self.test_typescript(source)
spring ferry
#

You need to assign the return value of that function to a variable if you want to do something with it. But in this case, for a "build" function, it seems you'd only need the tests to pass, and not necessarily do anything with that output. If the tests fail, test_typescript will raise an exception and if you don't catch it in build it'll bubble up, in which case you'll see the output from the failed tests when calling build.

urban creek
# spring ferry You need to assign the return value of that function to a variable if you want t...

That makes sense @spring ferry . For coverage reports or other artifacts I could generate a file for further processing. I guess I just feel a bit disconnected from my code. For example right now I'm trying to work with this Helm package, to create a file and export it to the host's OS. The file doesn't exist, the execution looks OK but I don't know whats wrong:

    @function
    def build_helm(self, source: dagger.Directory) -> dagger.File:
        return dag.helm().chart(source).package().file()

    @function
    async def export_helm(self, source: dagger.Directory, helm_package: str) -> bool:
        packaged_file = self.build_helm(source)
        return await packaged_file.export(helm_package)

The call to ...package().file() is supposed to return a dagger.File (https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/helm@d814d0d7c421348f51cdda96870a05ca2aa8e96a#Package.file) and I am expecting that file to get copied to my OS via File.export. When executing I get the output:

dagger call export-helm --source=./console-chart  --helm-package=./my_helm_package.pkg -v
✔ connect 2.7s
  ✔ starting engine 2.1s
    ✔ create 2.1s
      ✔ exec docker start dagger-engine-340fc17822076376 0.2s
      ┃ dagger-engine-340fc17822076376
  ✔ connecting to engine 0.5s
  ✔ starting session 0.2s
✔ initialize 9.4s
✔ prepare 0.1s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.resolveDirectoryFromCaller(path: "./console-chart"): Directory! 0.0s
✔ sstConsoleService: SstConsoleService! 1.2s
✔ SstConsoleService.exportHelm(
    helmPackage: "./my_helm_package.pkg"
    source: ✔ ModuleSource.resolveDirectoryFromCaller(path: "./console-chart"): Directory! 0.0s
  ): Boolean! 5.5s

true

I can't find the file anywhere on my OS and other than asking for help in this forum I don't know how I should be addressing this

helm :: Daggerverse

The package manager for Kubernetes.

spring ferry
#

That's because modules run in a specialized container, so when you call export within the function like that, you're actually exporting to that container, and not your host. To get files from and into your machine, you need to use the CLI. So instead of the export_helm function, just export from build_helm like this:

dagger call build-helm --source=./console-chart export --path=./my_helm_package.pkg -v
urban creek
#

OK that helps me, thanks a lot. However I wonder how I would have found that out myself without either logs, or a debugging environment (or perhaps reading the manual). I'm sure there will be further challenges ahead.

I am working in enterprise and would like us to move away from Jenkins and other traditional pipelines. Even Github Actions / Gitlab pipelines etc is vendor specific which we dont like. I have a lot of respect for the Dagger way of doing things and what you are doing, and I would like to build a case for it in my organisation.

However, as I mentioned earlier I feel disconnected from my execution. Can I connect to my running dagger from a debugging session? I feel that if I could just add a new run execution that supports breakpoints (e.g. from IntelliJ) It would free me up to commit to this. Is anything like this supported or in the works?

spring ferry
#

If you go to the Developer Manual and follow along the docs, you'll soon open the "Execution Environment" that explains the concepts here. The cookbook has practical examples if code samples is what you need. If you didn't read the documentation, it would help us if you can give feedback on why. Or even what could be better in the docs that would have made you aware of this.

#

To be clear, that's not debugging the python code like you mentioned, it's debugging Dagger Container artifacts by dropping into a shell inside of them.

#

I don't see an issue in GH to support connecting to a debugger in the language, but you can create one 🙂 I suppose it's technically possible, I just don't know atm if there's any networking limitation but we do support connecting services to and from the host, but it may require a bit of plumbing in the runtime container. We could even have a common CLI command that would work the same way no matter the SDK.

spring ferry
urban creek
#

From my fresh perspective (I am clearly new to this):

Developer Manual name put me off because it feels that is an area for people contributing to the Dagger project and didn't seem relevant to me. I now know I am a Dagger developer, not user.

Another reason I like Dagger for my org is that this approach offers my org a clear way to separate responsibility between devs and cloud engineering (Devs vs Users). If devs are responsible for how to build the application (beyond a dockerfile), it makes life easier for everyone since there is a clear handover artifact. I am a developer by experience and when I'm writing code I always feel a bit crippled if I can't debug it. At least print statements persist on the output (which I only just found out through experimentation)

#

Regarding adoption - I have another question. We have security limitations and will not expose any telemetry to a cloud environment we don't control. We expect to run this on our k8 cluster, and would like telemetry on the execution. I understand you are using OTEL to generate traces. We do not need Dagger cloud, and indeed would be unwilling to use the service as it stands now (we need to keep things in-house). In addition we are using Grafana to build a high level of all parts of delivery. Being able to have our own OTEL Collector collect the traces and ship them to our infrastructure would be ideal for us. Can we point dagger to our own collector?

spring ferry
spring ferry
urban creek
#

Thanks. Just to simplify the question:
We would like traces to go to our own OTEL Collector. Is this possible?

spring ferry
#

Yes, I believe it is. \cc @vestal pebble

vestal pebble
urban creek
#

fantastic, thanks

#

do you ship any other telemetry? Logs / metrics?

vestal pebble
#

No metrics, but OTel logs + traces yes

urban creek
#

Thanks for all the help @spring ferry and @vestal pebble . It's highly appreciated

urban creek
#

my dagger engine image is: 197b27c657552fdba0b9dff313a1ba4bbe86acc5a1fa21feebbb3101ea75fa09

#

my dagger engine image is: 197b27c657552fdba0b9dff313a1ba4bbe86acc5a1fa21feebbb3101ea75fa09

spring ferry
#

The changes in the demo haven't been released yet, but have been merged. They'll be available in v0.12.0. In your version, if you have a function that returns a container, you can add terminal to the end of the CLI command to get into a shell.

urban creek
#

is there an ETA on 0.12? Or somewhere I can monitor for announcements like this?

urban creek
spring ferry