Serverless Logs Inconsistant

[snow sage]

Hello,

We are currently testing various Docker files to ensure the stability and reliability of the systems we’ve built. However, we’ve encountered significant challenges with the logging system.

At this time, logs only appear to function properly about 10% of the time. Additionally, telemetry data tends to reset whenever we open the details for individual workers, and the log output is blank in approximately 90% of cases.

Could you please let us know if this is a known issue, or if there’s a recommended approach for capturing logs and debugging errors more effectively in this environment?

Thank you,

[lavish badger]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[halcyon hull]

Use prometheus or any logging tools if you want to make sure it works everytime with long logs

What are you logging?

[formal furnace]

how are you looking at logs? there are 2 types of logs, serverless logs on the tab, or worker logs when you click a worker

[deep sluice]

I can't say for sure I had the same problem but this post looked similar to my problem. Initially I had simple logging using Python logging to console. I would see most of my log output but many lines would be dropped. I then switched to importing and using the runpod logger, but no improvement.

Here's what worked:

        start_params = {
            'handler': handler,
            'return_aggregate_stream': True
        }
        runpod.serverless.start(start_params)

I added return_aggregate_stream: True. Note, my app does not require the /stream output. I just guessed that runpod's internal logging system required it. All my logs appear as expected now.

[deep sluice]

wow -- I spoke too soon. It's back to the old behavior. Guess I just got lucky.

what is the deal with runpod logging!!!???

why is this hard