#Voice PE responding on wrong device.

1 messages · Page 1 of 1 (latest)

spice fractal
#

I have two PEs (we'll call them 1 and 2). They are on different floors so only one can hear me at a time. If I tell #1 to do something that Assist can handle, like "Turn on the rec room light", it does it and says something to tell me it turned on the light. If I tell #1 something like "Tell me about Japan", it plays the response on the other PE (#2). The LED ring on #1 spins like it is the one replying, and I can tell it to "stop" and the response that is playing on #2 stops. Then the same response starts playing on #1.
This only appears to happen when the processing falls back to my locally hosted Llama 3.2 LLM. Simple commands execute properly and the response/confirmation plays back on the correct PE. Any ideas what might cause this?

neat knot
#

Well this is a new one. @sly tangle any thoughts?

rough iron
#

Huh, I have 5 Koalas in my home, and didn't see that behaviour...

spice fractal
#

Yep, just tested again but the other way around. Same thing. Answers from the LLM are going to the wrong PE. Weird.

rough iron
#

Re-add them maybe

spice fractal
#

I removed them, factory reset each one, and re-added them. First one worked fine. As soon as I added the second one it started happening again.

spice fractal
#

Further testing: I completely removed the Whisper, Piper, OpenWakeword addons (erasing data too). Also made sure Wyoming was removed. Removed and reset the PEs. Rebooted for good measure. Reinstalled and re-setup everything. No change. Local commands work great. LLM responses go to the other PE.

spice fractal
#

Possibly solved: I originally had them on a VLAN for IOT devices which was different than HA's VLAN. Should all be set up correctly because I've never had issues with any devices before, and the local commands work correctly. But I switched the PEs to the same LAN as HA and now the LLM responses reach the correct PE.

deep turtle
#

Attempt to prompt LLM not to use broadcast tools and then conduct testing

deep turtle
#

I typed in the browser dialogue: * tell a joke *, but LLM used a broadcast tool to tell a joke on my PE and then replied with a text message saying it told a joke.😂

sly tangle
rough iron
rancid flower
#

My guess, because of some glitch every response is broadcasted. But not to the device on which the command is issued (as with a normal broadcast request)

rancid flower
covert owl
#

I just encountered this as well with a similar set up as OP with both the PE on a dedicated IOT VLAN. Scared the wife when asking a question that sent to LLM and it started speaking in the living room PE mostly and part in the office PE

spice fractal
#

Ok I just ran into a related issue. I typed a question into Assist on my phone app (how many feet are in a mile). One of my voice PEs responded with the answer (not the phone) and then the answer popped up in the assist chat on my phone. Also tried a non-llm command (turn on the master bathroom light). This turned on the light and the response (turned on the light) showed up in assist on my phone. The voice PE did not say anything.
So the issue is similar to before. Commands that are handled by the LLM are being verbalized on the wrong device. In this case my phone, HA, and the PEs are all on the same network.
I turned off my phone's wifi and tried another LLM question (how many feet are in a meter) using only cellular data. This time the PE didn't say anything and the answer popped up on my phone.

rancid flower
#

does only one Voice PE resonds, or all of them (assuming you have more)

#

@spice fractal

spice fractal
#

Just tested it again by asking Assist on my phone some LLM only questions. All three of my PEs respond at the same time . After that my phone responded with the same answer.

coral mesa
#

Sounds like the LLM is grabbing the announce/broadcast tool and thinking it needs to use it to answer 😄

rough iron
spice fractal
#

Lol I wouldn't put it past the LLM meddling with things in an unexpected way. This new issue is related, but it is slightly different. The first issue was the response coming out of the wrong PE first, then coming out of the original PE. But in this case (Assist command on the phone), the response is definitely being broadcast to all PEs simultaneously.

rancid flower
#

It's the same issue, as broadcasts aren't sent to the device on which you issue the command. If you want to broadcast that dinner is ready, it doesn't make much sense to announce that on the device you're standing next to

#

So your issue is that the LLM is broadcasting everything. (Which I already said 2 weeks ago 😉)

#

Tell your LLM to only use the broadcast tool when it's specifically asked to broadcast the result

spice fractal
#

Makes sense. I've tried different variations of "Don't use the broadcast tool" in the prompt and reloaded it, but the issue remains. Any way to completely disable the broadcast tool, or maybe just cut its link to the LLM?

#

I also noticed that I am being given two different LLM responses. They are both similar, but the PEs say it one way and the text response in the PE chat says it differently. Sometimes the responses are the same, sometimes they are different. So it appears to be processing a single Assist request twice.

rancid flower
#

Which LLM are you using?

sterile epoch
#

It could be processing it once, but the LLM is just responding differently. If you had it perform some other intent, it would generate that intent call, then generate a response to you. It’s doing the same here but one of those is the broadcast message.

spice fractal
#

LLama3.2 - just noticed that the PEs are exposed to Assist. I'll try removing them from the exposed list to see if that helps tonight, unless it breaks something else.

rough iron
gilded flame
#

I had the same setup and issue as OP with 2 devices. I eventually gave up llama3.2 and used samantha-mistral as a test and it works fine except for shes annoying. but i think lamma3.2 is the issue because the moment i go back to that, it starts broadcasting to the other device as well. edit: llama3.1 works just fine as well.

spice fractal
#

Un-exposed all the assistants. I forgot they were automatically added when I reset them. No change. Tried dropping to llama3.1. No change. Will try a different LLM tomorrow.

deep turtle
#

For locally run LLMs, you are limited to using Qwen2.5. However, if you are open to a modest billing, the more cost-effective 4o-mini would be an excellent alternative.

spice fractal
#

Update time. This was previously planned, but I rebuilt my LLM box and started from scratch since some new hardware arrived. It runs Ubuntu with Ollama, Piper, and Whisper in docker containers. Previously Piper and Whisper were HA addons on the HA server, so I removed them. Nice speed boost, and recognition is better since it runs larger whisper and piper models. Anyway, the problems with Llama3.2 remained. Dropping to 3.1 fixed the issue of responses being broadcast to all devices. Weird, but things are running great now. I guess Llama3.2 really wants to broadcast itself.

supple echo
#

I have issues with ChatGPT responding in wrong devices. It’s so annoying

supple echo
mossy arrow
#

so I had a theory on this: i dont have alot of expierence with llms and could be totally off and maybe it doesn't even make sense but here we go...

the llm is told its voice assistant and thus wants to "talk back" and instead of doing a standard response its sending a TTS action which is finding the wrong device. perhaps you can add to the prompt something like "respond only in plain text that will work with TTS" or something to that effect.

perhaps I am barking up the wrong tree...

mossy arrow
wispy oar
#

How're your results?

wispy oar
deep turtle
# wispy oar How're your results?

I only encountered the misuse of broadcast tools with llama3.X, and I have stopped using it. Instead, I am now using Qwen2.5 and 4o-mini, where I haven't encountered such issues

wispy oar