#🧵 want to have a discussion about

1 messages · Page 1 of 1 (latest)

scenic vessel
#

Hallo 🙂

#

once again, i come to you with the fact that the logs from the engine are quite verbose 🙂
this means that again, we're in the place of trying to get raw logs from the github interface in our test/testdev workflows is impossible - the logs are now 149MB (if i'm not wrong, we're also needing to store these logs in cloud? that feels like quite a lot of growing storage)

#

obviously, i'm trying to use dagger cloud as much as possible here, but also, since a lot of these logs are actually producing on stdout/stderr for each test, then these are still tricky to sort through there when looking for useful lines

#

So, some potential ideas:

  • Could we rework our tests to just produce less noise in general? Maybe only producing output on failure (though this doesn't affect the engine logging, only the client-side/sdk logging)
  • Could we rework the level of some of our logs? There's a lot of exporting log/recording span/etc, as well as new buildkit client logs (specifically uv pip seems to generate a lot of stuff here).
  • Could we reduce logging to info for our tests? While keeping prod engines the same, so we can keep having insights into debugging specific issues.
  • Could we somehow capture all the engine output at once, and then display it separately, etc, a la https://github.com/dagger/dagger/issues/6553
#

Obviously, there's a balance to strike here, where we're trying to investigate a few ongoing cache/telemetry related issues, which is why we enabled this logging in the first place - we need to keep doing this.
But the problem is that this is making it much harder to understand what's happening in our own CI at a glance (especially for external contributors who aren't a member of the dagger cloud org, like community SDK maintainers).

#

Not trying to stir the pot 😄 I know we've had this conversation a few times, but I'd like to try and spend a bit of time myself trying to clean up a bit in the next couple of weeks - having readable logs feels like it could be a reasonable part of prod-ready engine (from at least the admin side)

#

cc @tight grove @inner lodge @fluid furnace @low cloud who all have had opinions about this (sorry for giant text wall ❤️)

fluid furnace
#

Can you link to an example of super verbose logs, just so we're on the same page? I recently added Buildkit console format logs back to the mix since a customer was hitting an issue that required them to figure it out, so not sure if you mean that or something else. (edit just to connect dots: the customer issue was the same issue as #commander-sagikazar message)

#

Also asking because I know @low cloud recently did a ton of work already to reduce log verbosity, so curious what's still too verbose

scenic vessel
#

there's a few things that are a bit sus - we seem to get schemas printed out a lot as a result of getting buildkit console logs?

#

we also have lines that are just the letter "b" repeated 100s of times?

scenic vessel
#

not quite sure why those might output to stdout at all

fluid furnace
scenic vessel
#

thanks thanks ❤️

fluid furnace
#

@scenic vessel merged