Streaming LLM Responses in Home Assistant Voice Assistant

[heady solar]

Hi everyone,

I’m developing a custom LLM agent and looking for a device to integrate it. Home Assistant seems like a great option due to its open-source nature and flexibility. However, I have a couple of questions regarding how responses are handled in the Voice Assistant:

1️⃣ Can I stream the text as my LLM generates the answer, or do I need to wait for the final response from the HTTP call before speaking?
2️⃣ Can I make another HTTP request to my LLM while the first one is still being processed, allowing for parallel interactions?

I want to achieve a more fluid interaction where the assistant starts speaking while still generating the response, and at the same time, it should be able to handle new requests dynamically if the user continues talking.

Any guidance or insights would be greatly appreciated! 🚀

Thanks in advance!

[shell karma]

streaming replies from LLM is coming in Core HA, don't think it is there yet but it was announced in their last voice stream. Think TTS is also eventually going to support streaming, but isn't there yet.

[heady solar]

Thanks, Nick, for your answer!
I saw something about that, but I wasn’t sure if they announced any release date or a beta version to test these features.
By the way, I really hope this amazing project continues to evolve, and I can integrate it into my business soon.
Let’s create tools for a better world!

[spice laurel]

In the meantime you can take a look at this, maybe also for inspiration for your custom LLM agent: https://github.com/eslavnov/ttmg_server