Stuck on "DBG frontend exporting spans"

[gentle walrus]

The Dagger pipelines runs fast when executed locally but takes for ever on our cluster executed via ARGO Workflows.
I always seems stuck at some commands (e.g. a "upload" step). I enabled the debug logs to see what is going on, and everytime it is stuck the last log line is "DBG frontend exporting spans".

What is this? What does it export? Where? Is this necessary or can this be disabled?

[rugged hinge]

Hey there! does it get stuck on random steps? Or it's always at the same one?

Have you followed our docs here : https://docs.dagger.io/ci/integrations/argo-workflows/ on how to setup Dagger in Argo?

[gentle walrus]

Yes, with one difference where I had to also specify the command for the docker engine sidecar: command: ["dagger-entrypoint.sh"]

It feels like it's always the same steps.

[rugged hinge]

do you have kubectl access to the pod where your engine is running in your k8s cluster? If so, you could try setting the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable to the k8s pod name as shown here: https://docs.dagger.io/ci/integrations/kubernetes/#example and check if the dagger call works ok when calling your remote engine with your local machine's pipeline 🙏

[gentle walrus]

I'm not sure what you are suggesting. It pipeline is generally working, it just get's stuck for a while when it hits these DBG ... logs.

Not stuck in the sense of not continuing

but in the sense of taking some minutes before continuing.

The pipeline took 1-2min locally but takes 1h on the cluster

[rugged hinge]

oh, so eventually finishes.. my bad I thought it was just stuck there forever

is there any chance you can connect Dagger Cloud to your pipeline so we can see what's happening from our side?

[gentle walrus]

It's a payed service for companies, right?

What are these "DBG frontend exporting spans"? What is happening in this stage?

[rugged hinge]

that only shows in debug mode. It definitely shouldn't be taking any time there even if the logs are showing that message. Those "exporing spans" messages are a bit noisy in debug mode

[rugged hinge]

it's free for single users

so you're good to create an account and use a token in your argo server to send telemetry there

[gentle walrus]

If this is not configured, there is nothing send outside our network? and it is also not trying in these steps (having connection issues and hence being stuck?)

[rugged hinge]

the engine will not send anything outside your network if you don't have a token set.

On a slightly different topic, how much resources are you allocating to your engine sidecar? From this other message (#1389624856156307516 message) I see that you're allocating 500 millicores and 1Gb of RAM. That's way too low for the engine to work reasonably fast

eventually, it's also recommended if you can put the engine state volume /var/lib/dagger in a physical disk and not network block storage as well. Dagger is mostly disk intensive since each operation you perform in a container, gets it's overlay CoW snapshot.

[gentle walrus]

You mean to mount a PVC to /var/lib/dagger?

[rugged hinge]

yes, PVC is it's backed by a local disk or something is preferred

[gentle walrus]

ah you mean local to the node the build is running on?
We don't have specific build nodes atm.

[rugged hinge]

yes. It's mentioned here : https://docs.dagger.io/ci/integrations/kubernetes/#recommendations

[rugged hinge]

that's fine. You can use network drives for PoC's but take into account that you'll have a considerable performance penalty. Specially when you compare it against your local machine which probably has a lot of cores and fast NVMe drives

[gentle walrus]

I will need to check that with our IT.

[rugged hinge]

cc @humble sequoia @hollow arch

[gentle walrus]

Does it make sense to have a separate deployment of the dagger engine? (ensuring both will run on the same node).

[rugged hinge]

yes, in many cases that's the recommended architecture. More about that here: https://docs.dagger.io/ci/integrations/kubernetes/#persistent-nodes

[gentle walrus]

A single dagger engine can handle multiple requesting build jobs at the same time?

[rugged hinge]

yep. In most of the cases this is advisable given that if by some chance two jobs are doing the same operation like pulling the same image, installing dependencies, etc. These operations will get de-dupped and will be efficiently cached

[gentle walrus]

We are looking into configuring a dedicated build node and I'm planning to make the enginge a standalone deployment and not a sidecar.
Will report back how that goes 🙂

[rugged hinge]

awesome! ping if you need any help

[gentle walrus]

Still with the sidecars: When I launch two pipeline executions at the same time, then the second one fails to connect to the socket. Is that expected? Is that also a limitation when deploying a single engine? Or should it be able to run multiple builds in parallel?

[rugged hinge]

Yo should be able to run pipelines in parallel

Have you checked that both pods land in the same node?

Oh wait

You're using sidecars

[gentle walrus]

yes, two workflows each with their own sidecar

the second one starting and running in parallel can not connect to the socket.

Socket is mounted as empty dir

[rugged hinge]

There's definitely something odd in the k8s/argo config somewhere. I'm AFK now it's 7pm here sorry. @real moss @modern osprey or @sour stump could help if they're around

[gentle walrus]

As we are looking into switching to a separate engine deployment, it might not matter for us, but at least consider it reported.

I will report back if the issue persists with a separate engine deployment.

[gentle walrus]

Can confirm that parallel builds are not a problem for the separate deployment. We now also have the dedicated PVC.

Build time went down significantly

I'm happy now

[gentle walrus]

Thanks for the help

[real moss]

Great to hear! You're most welcome 🙂