using LLM (ChatGPT, ollama, etc) to debug Home Assistant logs

[leaden sky]

I've been using LLM a lot lately, kind of figuring out the use cases where it's really helpful vs where it falls flat. I recently had zigbee2mqtt kind of randomly fall over, maybe after a software update that I don't remember casually accepting, whereby I kept getting a failure loop in my logs. There was a conspicuous info message in the logs to check the guide for possible solutions, but I wanted to see if ChatGPT (4o model) was smart enough yet to figure it out.

The results were a mixed bag. ChatGPT was really clever about debugging what USB devices were seen by the system, using lsof to see if any other process had it opened, etc. For the most part, it read the logs well.

Except it totally ignored the part of the error message that recommended a possible solution. Kudos to the zigbee2mqtt devs for putting the link to the likely solution right there in the log!

I went back and forth with ChatGPT a bit to see if it might eventually come to the correct conclusion and it never really did.

It gets me to thinking... should we be thinking about logging from the perspective of (optionally) labeling things in a more AI-friendly format? So like right now the default looks like a typical syslog with a date & time stamp, loglevel, and then some strings of text. When I think of the big enterprise log management platforms like Splunk and Datadog, there's usually a bit more lifting to format log messages in such a way that the respective platforms can make more sense of the context of each log event.

While the Nabu Casa folks are figuring out where local LLM like ollama can fit into our ecosystem (something I'm playing a lot with right now myself), I think there's likely a lot of potential benefit to having Home Assistant logs labeled for something like ollama to consume, sending webhooks to an assistant to decide how to handle events of special interest.

Here's an example of how zigbee2mqtt was originally failing that started me down this whole rabbit hole:

[13:41:31] INFO: ^[[32mSocat enabled^[[0m
[13:41:31] INFO: ^[[32mStarting socat^[[0m
^[[34mSocat startup parameters:^[[0m
^[[34mOptions:     -d -d^[[0m
^[[34mMaster:      pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777^[[0m
^[[34mSlave:       tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5^[[0m
[13:41:31] INFO: ^[[32mStarting socat process ...^[[0m
2025/01/05 13:41:31 socat[46] N PTY is /dev/pts/0
2025/01/05 13:41:31 socat[46] N listening on AF=2 0.0.0.0:8485
[13:41:32] INFO: ^[[32mStarting Zigbee2MQTT...^[[0m
Starting Zigbee2MQTT without watchdog.
[2025-01-05 13:41:40] ^[[32minfo^[[39m:         z2m: Logging to console, file (filename: log.log)
[2025-01-05 13:41:41] ^[[32minfo^[[39m:         z2m: Starting Zigbee2MQTT version 2.0.0 (commit #unknown)
[2025-01-05 13:41:41] ^[[32minfo^[[39m:         z2m: Starting zigbee-herdsman (3.2.1)
[2025-01-05 13:41:41] ^[[31merror^[[39m:        z2m: Error while starting zigbee-herdsman
[2025-01-05 13:41:41] ^[[31merror^[[39m:        z2m: Failed to start zigbee-herdsman
[2025-01-05 13:41:41] ^[[31merror^[[39m:        z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-01-05 13:41:41] ^[[31merror^[[39m:        z2m: Exiting...```

Pay special attention to this line of the log, the one ChatGPT missed entirely, which ended up having the solution:

Link pasted outside of log context here for your clicky convenience: https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html

And here's my chatlog with ChatGPT for the morbidly curious: https://chatgpt.com/share/677ae2d0-e4cc-800b-a00d-e6626e7ab540

for giggles I'm going to let ollama try it out with a couple of different open source language models.

for my first try on local LLM, I'm going to try qwen2.5-coder:14b model running on ollama on my base model Mac Mini M4. I'm starting with the same prompt, same log snippet.

I am running Home Assistant with the zigbee2mqtt add-on. Suddenly the Zigbee network stopped working. Here are the logs. Can you help me troubleshoot?

I'm running this through an open-webui front end which gives me some RAG features I wouldn't get on the command line. So I toggled on this model's ability to use web search and do crawling and deep analysis.

ChatGPT 4o was actually way better at this. Here's what qwen said in the opening volley:

Sure, I'd be happy to help with troubleshooting your Zigbee network issue in Home Assistant.

1. Check the zigbee2mqtt configuration: Make sure your zigbee2mqtt configuration is correct and that all necessary settings are properly configured.
2. Restart the zigbee2mqtt add-on: Sometimes a simple restart can resolve connectivity issues. You can restart the zigbee2mqtt add-on in Home Assistant from Configuration -> Add-ons -> zigbee2mqtt (click the three dots next to it and select Restart).
3. Check Zigbee device status: Check if your Zigbee devices are responding correctly. If not, try resetting them and re-pairing with zigbee2mqtt.
4. Review logs for any errors or warnings: Look at the zigbee2mqtt logs in Home Assistant (Configuration -> Logs) for any error messages or warnings that could provide more insight into the issue.

If none of these steps work, you may want to seek help from the zigbee2mqtt community or check if there are any known issues with the add-on.

It even cited a number of web sites that it crawled after running a search. But it never occurred to this model to crawl the link that was in the log itself. 🤣

llama3.2:latest was only confused by its search results.

It appears that you have multiple unrelated threads here. The first three threads are related to Siri's Notes feature not working properly on iOS devices and HomePod speakers, while the fourth thread is a new user query about troubleshooting issues with their Zigbee network using Home Assistant.

To address your original question, I'll provide an answer based on the information available in the first three threads. If you have any further questions or concerns, please feel free to ask!

Based on the posts, it seems that the issue with Siri's Notes feature not working properly is likely a known problem caused by the iOS 15 update. Several users have reported this issue, and some have found temporary workarounds, such as clicking the "Me Too" button in a discussion thread or updating their HomePod software.

However, it's unclear whether these workarounds are permanent fixes or if they'll be addressed by Apple in future updates.

Regarding the Zigbee network issue, I don't see any relevant information from the new user query. If you'd like to provide more context or details about your Home Assistant setup and the issues you're experiencing with your Zigbee network, I may be able to offer more specific guidance or recommendations for troubleshooting.

(I get much better ollama results when I don't allow web search, but it still totally misses the obvious answer in the logs)

qwen2.5-coder:14b came dangerously close to actually noticing the link but didn't realize there might be something there. I got a long response from this model that put this line item up at the top of its response but failed to grasp the point of it.

With ollama you can start a response with a # and then the URL you want it to crawl. It'll download that page, scrape it, and use it to inform the response to whatever prompt follows. I tried that with the qwen2.5-coder:14b model... and it still totally missed the point. 🙄

I give up.

[manic zinc]

Hi @leaden sky,
https://community.home-assistant.io/t/want-to-help-others-leave-your-ai-at-the-door/523704
That how much Nabu Casa thinks about what you are asking about...
Someday, maybe. Not yet.

[leaden sky]

the funny thing is... it doesn't have to be wrong.

[manic zinc]

Fine, no problem, use what you want. Use it if you like. Don't bring it here or any of the other Home Assistant Community Socials to help others with. Some day this will change, but that's not today. The models are not taught for this language, so it makes shit up. And I'm not kidding. Sometimes it is even the logical thing that probably should work logically, but they were not taught using the HA Docs, so it comes out weird.
At this point most of the LLM's speak languages other than Home Assistant. So use it to make cool forecasts or to collect thoughts on some web page or something. That's all good. and LLM's are good at that.

Maybe it can search logs and find a pattern, don't know. Currently use for support to others is not welcomed. If you get something going and can teach people to use it for themselves I don't know what that is.

For others to decide I guess.