Ollama, VA, & HA

[keen blade]

Trying to get HA to work with Ollama and VA.

List of things I’m trying to do.

Ask VA about devices. ❌
Ask Ollama about devices. ✔️
Ask questions for Ollama thought VA. ❌

Where do I start?

[misty ruin]

Here: https://www.home-assistant.io/voice_control/voice_remote_local_assistant/
Do you have a GPU?

[keen blade]

I think so. I’m at the gym right now and I will look when I get home. I know it’s an Nvidia.

[past cape]

are you just looking for a guide to set this up or your running into an issue?

[keen blade]

Setting up. I’ve got Ollama in Assist to see devices but not VA.

[past cape]

bit confused what you mean your LLM see the devices, HA and VA act in tandem and HA is what retains your devices and your VA simply looks at what is exposed in your HA to reference the entity ID for each call to command. So if you have your VA setup to only route commands through your LLM and not enabled the feature to run your commands locally first?

I think if you just go to your expose screen, you can setup aliases and should work without much problem

[light plume]

I'm curious why people mostly writing questions about their setup when they're not at home... 🙂

[keen blade]

I have an Nivida Quadro K5200. The GPU is a GK110B.

[keen blade]

I do a good bit of thinking while between sets. lol

[light plume]

Probably your Pipeline setup is crooked. Or, as others said, your LLM is choking. Case is, that when you build it into pipeline, HA is adding system prompt to every message, that contains your exposed entities, and tools, exposed by HA. That adds up to the context a lot, and model requires more RAM and more time to process, comparaboe to using it directly.

[keen blade]

Sounds like I need to go through and only expose what I want my LLM to see. The PC that's running my LLM does have 64gb of ram and a Quadro K5200. It was an editing PC at my work.

[misty ruin]

What model do you use? Are you sure the GPU is being used?

[light plume]

RAM itself isn't that important, if you run your Ollama with GPU support. Model should run on GPU VRAM. Make sure it fits there (don't use big model, take something 3B Q4 for starters).

[keen blade]

llama3.2 and no clue.

[past cape]

Does ollama run native or in a docker?

If docker ensure u are using the --gpus=all foe the argument

[misty ruin]

No one sane uses docker run 😄

[keen blade]

Docker Desktop

[past cape]

Run a voice command and what your gpu levels to see if it's being used, not sure what GPU monitor you use

[last tiger]

Because at home, there are many other things that occupy your time, but you can find a few minutes here and there during the day to ask questions and check for replies to them.