I am doing the work to get HA setup optimally before my VPE arrives. I have been using Assist (Overview, upper right hand corner) for testing. Assist controls devices, etc as expected. I am running Ollama locally and the radio button for Prefer Handling Commands Locally is On in the Ollama Voice Assistant configuration. If I prompt by selecting Ollama in Assist it works correctly. If I prompt for a general knowledge response in Home Assistant (not Ollama) it always fails. It never attempts, as far as I can tell, to fallback to Ollama. Either I don't have something configured correctly or I am misunderstanding how fallback is supposed to work. I am referring to the feature added in 2024.12.. Any advice would be appreciated!
#Assist local does not fall back to LLM
distant ingot
cobalt cosmos
- Set up a LLM integration and select "Assist" (as opposed to "No control") in the "Control Home Assistant" option
- Set up a pipeline with a LLM conversation agent and make sure the satellite uses that pipeline
- Turn on the "Prefer handling commands locally" option for that pipeline and save it
- Ask the satellite to control your home ("Turn on the kitchen lights") - it should not access the LLM and the light should be turned on using the default conversation agent
- Ask the satellite to control your home using non-standard language ("It's very dark in the living room. Is there anything you can do about that?") - it should control your lights via LLM tools
- Ask the satellite about some general knowledge ("How many moons does Saturn have?") - it should respond using the LLM training data
All the above capabilities depend on the LLM you're using. Some are better, some are worse. Ollama is known to be a bit stubborn to use tools, so for testing you can try with OpenAI (e.g. gpt-4o) and then migrate to Ollama to see if you get the same results
Also, this topic is better suited for #1284966915055288530
distant ingot
Thanks! I will try this. Apologies. I had intended to post this in voice-assisants. Thanks again.
distant ingot
After reading through this again and reviewing my configuration I think I understand the issue. It appears that fall back is only designed to work with a pipeline and a satellite. I have been using the Ollama integration locally and, when I added it I selected "Assist.". "Prefer handling commands locally" is enabled in the Ollama integration. My VPE should arrive next week. I have been testing exclusively with the chat interface available in Overview, upper right hand corner. Based on what you described above it appears this interface is not designed for fallback? Thanks!
cobalt cosmos
The browser or companion app are slimmed-down satellites. I don't think i understand what you mean by "fallback". Can you desecribe exactly what you are trying to do, along with the result you expect and what actually happens instead?
distant ingot
Apologies that I am not being clear. I am trying to get as much of the configuration work completed as I can before the VPE arrives. I have upgraded my HA to a new, sufficiently powerful NUC to run voice locally. I have installed a dedicated Seeed Studio J4012 with the Jetson Orin NX 16 module to run Ollama locally. I have setup the Ollama integration in HA as described above. I have been using and testing the @inner jungle weather integration for LLM to get a weather forecast. I have been exposing the desired entities to Assist and creating custom sentences to replicate the phrases we currently use with Alexa. Through this configuration and testing I have been using the browser interface. I have Home Assistant set to Preferred in the browser interface. Using the browser interface I can issue control commands. Turn on X, turn off X, Pause X, etc. In the browser interface, if I select Ollama (from the dropdown), I can ask "What's the forecst for tomorrow?" using the @inner jungle Blueprint and it responds as expected. If i ask the web interface, with Home Assistant as Preferred (not Ollama), "What's the forecast for tomorrow?", it resonds "Sorry, I am not aware of any device called forecast for tomorrow."
inner jungle
Yes, that's expected, you need an LLM to use that script blueprint
On the ollama pipeline you can optionally set that up prefer local handling of the commands
Then it will try first without the LLM, and if that falls it will try the LLM
It's a bit confusing in your case, as your LLM is also local
distant ingot
Interesting. Thanks very much for your work on the Blueprint. I think I maybe getting some terms confused. In the Ollama Conversation Agent configuration I have it set to Prefer Handling Commands locally. My understanding, which must be incorrect, Is that if I asked a general knowledge question of Home Assisant (not Ollama) it would fallback to Ollama.
I presumed the local conversation agent is the one set to Preferred, which is Home Assistant. Sounds like that's not the case?
inner jungle
That's not how it works. If you set the Prefer handing commands locally option, you should use the Ollama voice pipeline. It will then first try without Ollama, and only if that doesn't work it will use Ollama
So that's the Voice pipeline you should use (and probably also want to set are preferred)
cobalt cosmos
Another way to put it: having "prefer handling commands locally" set on a LLM pipeline will first try to handle the commands without LLM, and then fall back to LLM if that fails, regardless of what your preferred pipeline is. The preferred pipeline simply makes one pipeline or another the one to use by a satellite if you have not explicitly chosen any pipeline on that satellite.
distant ingot
Thanks to you both for explaining it. In my example when using the Assist (satellite) and asking "how far from the earth to the moon." I should chose (and optionally set as preferred) the Ollama Conversation agent. With "prefer handling commands locally" set, If I ask it to "Turn off X" it will forward that request to Home Assistant rather than the LLM attempting to "Turn off X." Is that correct? Does this also effect the recommendation for the number or entities you can expose. The docs recommend approximately 25 if you using the LLM to control HA. If you are using Home Assistant as the conversation agent then, I presume that limit isn't an issue? In the Ollama integration do you set it to Assist or No Control? Thanks very much to both of you for helping explain the details of the architecture!
inner jungle
If you set Ollama to no control, it won't be able to access scripts like the weather forecast script
As it needs access to the exposed entities to find and use such tools
distant ingot
Got it.
So, with "prefer handling commands locally" set it understands forward Turn off X to Home Assistant rather than attempt that. Nonetheless, it is aware of all the exposed entities. Since it is aware of the exposed entities the recommendation of around 25 is still valid.
distant ingot
I used Debug, Run Text Pipeline and now it makes sense 🙂
Apologies for my misunderstanding of the architecture.
inner jungle
No worries, happy to help
distant ingot
I appear to have run into an interesting intents condition as it relates to Temperature. I added a Zigbee temp sensor and exposed it to Assist and added an alias "outside". The phrase we have been using with Alexa is "what's the temperature outside." In Assist this fails with no_valid_targets. The phrase "what's the outside temperature" works as expected. I would expect either phrase to return the same results. What am I missing? Thanks!
What's also interesting is "what's the temperature downstairs" returns the expected results..
distant ingot
OK. Thanks!
inner jungle
I appear to have run into an interesting intents condition as it relates to Temp...
what's the temperature outside basically does this what's the <name> and then returns the state of that entity
if you give the entity the name (or alias) temperature outside the phrase what's the temperature outside will also work