HA Assist is this bad???

[solemn turret]

Looking for some insights here as im totally caught off guard. I sparingly use a voice assistant when within my smart home as im acustomed to just using the HA dashboard on my phone. However from time to time ill remember i can just yell, "Hey Google, turn off the kitchen lights". Even, "Hey Google, whats the bedroom thermometer temperature" and ill get a response. I however then tried, "Hey Google, is the garage door opener door obstruction clear?" and it didnt know what to do. I checked and saw that the Obstruction sensor wasnt exposed to the Google Assistant and couldnt be. But it was exposed to the HA Assistant.

This is where i then told my self, great, my mind is already telling me to start building or purchase a Home Assistant Assistant speaker and start using local assistant interactions within my homne instead of relying on Googles ecosystem. However, when trying to interact with the HA Assistant, it was terrible. I was trying to reference a bunch of different sensor names to get statuses of them and it just didnt know what to do. And its randomly saying its doing things, yet it didnt even do them.

Examples...

Me: Whats the status of the Garage Door Opener Door Obstruction?

Assist: Opening

Me: Is the garage door opener obstruction sensor clear?

Assist: Sorry, I am not aware of any area called Is

Me: Are the kitchen lights on?

Assist: Turned on the lights

How it decides to open the garage door opener from my first question is beyond me. And it didnt even open it anyway. Trying to get the value of the Obstruction sensor is just a complete fail. And again, asking if lights are on, and instead of telling me, it say it turns them on, but it doesnt actually even do that. Am i doing something wrong? How is it this bad?

[scarlet warren]

what does your pipeline look like?

[solemn turret]

i havent done anything except whats set ootb.

[scarlet warren]

ok so. you are currently just using the stock agent. which is very rigid on commands. it doesn't have a "brain" behind it. so it cant really understand questions. you could add custom responses with automations or you would need to set up using an LLM as an agent so it has a "brain" and can look at what is exposed to it and work out what you are asking.

[solemn turret]

So i need to setup this piper\whisper integration then?

[scarlet warren]

whisper and piper are not the "brain"

[solemn turret]

ok so i gotta setup my own ai service then like ollama?

[scarlet warren]

the flow is:
speak to device
STT converts speech to text
text is processed by the conversation agent (brain) and responds
TTS converts text to output speech
audio of speech gets played

[scarlet warren]

ollama is a great option for a local llm yes.

[solemn turret]

Ok, so then simply using HA Assist as is, to replace my Google Nests, Alexa, etc isnt that straight forward. To get anywhere close id need to run my own local server, Ollama or something else. And likely even need to get an AI Accerlator of some kind like google coral, halio8 etc to even make it worth the efficiency im assuming. So were talking money and time to have all this running in parrelel

[scarlet warren]

a lot of people use cloud services like openai or gemeni api's instead of running local.
ollama does NOT use a TPU acclerator like a coral. in general for best results you run it on a GPU.

I run ollama on a separate server to my HA which has a GPU and is used for AI/Transcoding etc...

[solemn turret]

is openai\gemini have unlimited tokens? or are they rate limited when u leverage with assist?

[ivory forum]

There's no edge devices for AI yet. You'd need GPU, or something like Mac Mini.
However, LLM won't solve all your problems - it rather gonna add some.
This all makes sense, if you remember, that with amazon/google each request is sent to their servers and is processed there. Something should process plain speech...
With HA it's either LLM (self-hosted or cloud) - or extensive set of local rules. But those are to be written by you. That's why most of guys tend to use LLM - it's easier to setup to have mediocre results. But it's harder to master (I'd say - impossible, because of hallucinations).

[ivory forum]

You will have to pay. 🙂

[solemn turret]

uhg lol

[scarlet warren]

I think the api set up is pay as you use. but i am not totally sure as I don't use them.

[solemn turret]

well so much for doing any of all this just to be able to get info back on some sensors

[scarlet warren]

you can create custom voice commands with custom responces using automations.

[ivory forum]

Gemini has unpaid plan, but it's like for 30 requests a month... It was giving me troubles every day.

[solemn turret]

Can you link me to examples of this by chance?

[ivory forum]

For simple sensors - it will use local approach. If local Hassil engine can parse sentence - LLM won't be involved.

[solemn turret]

https://www.home-assistant.io/voice_control/custom_sentences/

[ivory forum]

Just go to automations and add trigger "sentence"

[solemn turret]

Is this what were talking about?

ok

[scarlet warren]

yeah you create an automation andf use a sentence trigger and then use set conversation response to tell it what to respond with. which can be templated.

[stone zinc]

Unless you're getting really long answers from OpenAI though we're talking pennies per day on usage for basic tasks like this