Does reusing same container clearing old output between runs?

[onyx oar]

Hello Team

I have weird scenario, We are using dagger engine 0.14.0 and re utilizing same container to execute many commands for faster runs, at the end of each exec .WithExec
we collect output using container.File("/mnt/stdall).Contents(ctx)

Upon checking traces in honeycomb, we are seeing some performance hit, only for container.file span but not for actually exec command.

So my question, could it be possible that re-utilizing same container does not reset previous output(or may be creating some sort of weighted debris inside container) which is slowing down Container.File operation

As I am still investigating this but meanwhile if you have any clue of internal behavior of engine on such usecase, please share your input.

Thanks

[faint siren]

👋 does the container.File get slower on each call? Can you clarify a bit what you mean "we are seeing some performance hit"?

Depending on the size of the file generated container.File might take a bit more time to return

[onyx oar]

That's my assumption as I am still digging in, but for visible fact from traces, container.File which access /mnt/stdall is getting slower.
So I was wondering what if output also accumulated during reruns of the same container could be a possible bottleneck ?

[faint siren]

strange.. haven't seen this before. I can try with a quick repro later today 🙏

[onyx oar]

thanks @faint siren , i will also dig deeper to understand more 🙂

[onyx oar]

Hey @faint siren did you get chance to look into this one?

[onyx oar]

here is trace from honeycomb, basically the actual command execution does not take much time, but Container.File take significant time(in our code we do container.File("/mnt/stdall).Contents(ctx)). So if we sum up all child span time, it does not add up that high compare to what we have on root span. My initial thought that it somehow accumulating logs or debris because we are reusing container for some usecases and also these are little heavy builds overall taking > 20+ min
or issue could be entirely something else

[faint siren]

but in this screenshot File.contents is taking 2ms?

that's fast, isn't it?

[onyx oar]

That's fast but before that there's Container.file a top level span which has high latency

[onyx oar]

@faint siren I am still trying to look for leads on what could have been the root cause here

[split zephyr]

@onyx oar this is because honeycomb is confused. Dagger supports lazy (async) execution, so an operation may be scheduled for execution at one point in the workflow, and actually executed (unlazied) at another point. Here the container.file is probably what triggered the execution that produce that container filesystem on demand. Honeycomb doesn't know how to track that.