"Prefer handling the commands locally" do not work for my native language | Home Assistant | Page 1

dreamy ember Feb 13, 2025, 4:29 PM

#

Hi all,

I'm running latest HA as HAOS on an iNUC, Assist PE, local Whisper (GPU) & Piper. Conversational Agent is OpenAI.

I have the "Prefer handling the commands locally" activated. Testing this with English works great but when switching to native language (Norwegian) it doesn't work. Tested both with text and speech. Flipper HA language between EN and NB, no changes.

I've checked the aliases, all is good.

Testing in my home office with a single light entity.

Any tips to look next?

#

"Prefer handling the commands locally" do not work for my native language

reef cedar Feb 13, 2025, 4:31 PM

#

Please describe what "doesn't work" means. 🙂
Doesn't local part work, or LLM doesn't?

dreamy ember Feb 13, 2025, 11:24 PM

#

The command will not run Local. From debug, I can see "Processed locally" is false. Here is a screenshot as an example - this time I started from conversation (non-verbal).

My phrase is in English "Turn on the lights in the office".
According to intents/sentences/nb/light_HassTurnOn.yaml https://github.com/home-assistant/intents/blob/main/sentences/nb/light_HassLightSet.yaml this should be straight forward

language: "nb"
intents:
 HassTurnOn:
   data:
     - sentences:
         - "<skru_på> [<alle>] <lys> [<i_på>] <område>"
         - "<skru_på> [<alle>] <område>[s][ ]<lys>"
         - "<skru_på> <område> [<alle>] <lys>"

This one should kick'd in: "<skru_på> [<alle>] <lys> [<i_på>] <område>"
Equal for EN is: "<turn> on <name> <light> [<in_area_floor>]"

GitHub

intents/sentences/nb/light_HassLightSet.yaml at main · home-assista...

Intents to be used with Home Assistant. Contribute to home-assistant/intents development by creating an account on GitHub.

#

#

I can add also, setting up a new profile where I use all for Norwegian, it do work...

Really stuck....

dapper tendon Feb 14, 2025, 7:20 AM

#

the 2 sentences you're testing with are different ("skru på" vs "skru av", which i'm guessing means "turn on" vs "turn off")
i recommend that you go to https://my.home-assistant.io/redirect/developer_assist/ and test each sentence to see if it can match anything (it won't be executed). my guess is that the actual sentence you're saying is not matched by the local conversation agent and is thus passed to the LLM agent. if the dev tools shows that it does match a built-in sentence, then we can go on to to see why the LLM is taking over

maybe @rich salmon can also help here with the matching part

#

oh, one more thing to check: when you configure the pipeline and check out the LLM conversation agent's settings, Control Home Assistant is set to Assist rather than No Control, right? it is a bug that you are allowed to see/toggle Prefer handling commands locally when the agent is set to not control anything, so that may be the source of your troubles

dreamy ember Feb 14, 2025, 9:01 AM

#

dapper tendon the 2 sentences you're testing with are different ("skru på" vs "skru av", which...

Hi @dapper tendon As I expected, running the sentence in the Dev-Tools gave match, and matches my findings in my earlier post.

dreamy ember Feb 14, 2025, 9:13 AM

#

dapper tendon oh, one more thing to check: when you configure the pipeline and check out the L...

Hi @dapper tendon All my Assist setups are set to "Assist"

As earlier described - I did set up an Assist profile with all local, then all run fine, both conversation by text and voice, on Onju and PE. But when involving an LLM (I normal use OpenAI but do also have Claude & Gemini available) I can't get it to use local when running Norwegian.

My other Assists which is set up for English, run local just fine. Something here with LLM and Norwegian/non-English (not tested with other languages).

dapper tendon Feb 14, 2025, 9:47 AM

#

can i please ask you to add an issue for the openai_conversation integration? https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A"integration%3A+openai_conversation"

#

or, if all of your LLM agents behave the same (OpenAI, Claude, Gemini), then tag the issue for the conversation integration

ocean terrace Feb 14, 2025, 11:08 AM

#

My guess, it matches 2 entities, and therefor doesn't know which one to target, and that's why it sends it to the LLM

#

I gave two lights the alias Test light and when using Assist (without LLM) that errors

#

#

when using LLM with local preference, it sends it to the LLM

#

in this case it gives a simlar response, but based on the LLM and it's mood the response will differ

#

oh wait, scrap all that, you are using an area as target

dreamy ember Feb 14, 2025, 1:59 PM

#

Gemini....:

#

Damn it! Figured out what caused it (now to figure out how to resolve it)... I run external Whisper (GPU) on a remote docker.

Switching to Home Assistant Cloud did the trick, now it works Local !

green harbor Feb 14, 2025, 2:03 PM

#

huh weird, wonder why the STT would affet it

dreamy ember Feb 14, 2025, 2:05 PM

#

Did the same test now using Google Cloud for STT, back to "false" again

green harbor Feb 14, 2025, 2:05 PM

#

is it hearing the same string each time?

dreamy ember Feb 14, 2025, 2:06 PM

#

green harbor is it hearing the same string each time?

Yeap

#

This is then related to STT

green harbor Feb 14, 2025, 2:06 PM

#

doesn't really make sense though, the STT just converts speech to text, it isn't responsible for intent handling

#

from the perspective of the conversation agent it shouldn't know or care which STT is being used, it just receives the text

dreamy ember Feb 14, 2025, 2:07 PM

#

Agree, but in reality (at least here) this is it. Weird is that with ChatGPT (Open AI) it wors in English

green harbor Feb 14, 2025, 2:09 PM

#

so if you type your command in the assist window, with all those settings in place, including the "bad" stt, does it still work?

dreamy ember Feb 14, 2025, 2:09 PM

#

Test ENGLISH with ChatGPT/OpenAI:

green harbor Feb 14, 2025, 2:09 PM

#

assist window should be using the exact same pipeline

#

Yeah the issue seems language specific, but it seems like either the intent has some issue, or somehow the particular STT you are using is doing something weird, like maybe inserting an invisible special character or something perhaps that the intent handler can't understand, that'd be my guess

dreamy ember Feb 14, 2025, 2:11 PM

#

Test NORWEGIAN (different pipeline though):

#

Test NORWEGIAN (same pipeline as previous Norwegian) with Home Assistant Cloud:

green harbor Feb 14, 2025, 2:13 PM

#

maybe you should delete the pipeline entirely and recreate it, maybe something is broken in it

#

wouldn't remove your openAI stuff or anything, it's just the glue that ties them together

dreamy ember Feb 14, 2025, 2:14 PM

#

To me, this at least proves that the combination LLM and at least Norwegian fails for using Local

green harbor Feb 14, 2025, 2:14 PM

#

yah but the pipeline could have something broken in it, if you delete it entirely and recreate it that could rule that out

#

otherwise yeah, maybe that language has trouble somewhere with the STT

#

the Cloud STT is a bit more robust, maybe it handles your language better 🤷

#

this is also super silly, but I notice in one case "skruv" is lowercase and in the other it is uppercase. That technically shouldn't matter, but it is one difference I see

dreamy ember Feb 14, 2025, 2:17 PM

#

Made a new Pipline, tested with HA Cloud and Whisper - HA Cloud let it run locally, not when changing to Whisper

#

Also tested changing STT to "faster-whisper" which run locally on my HAOS - same, still not local.

#

So definitly something when using any LLM combined with (at least) Norwegian

green harbor Feb 14, 2025, 2:19 PM

#

so if you remove the LLM and just do the Home Assistant agent, with any of these STT, does that work correctly?

dreamy ember Feb 14, 2025, 2:23 PM

#

Not entirely correct; I can use ANY conversational agent incl any LLM.

The difference is:

STT set to remote or local Whisper, do not run local on Norwegian**, BUT, run local if English is selected.

To get Norwegian to work, STT need to be set to Home Assistant Cloud.

#

Here is the test with Home Assistant Cloud and "turn off the lights in the office" and after that "turn them back on"

Run exactly is it should, BUT, not using Whisper.

Could this be related to Whisper (as it happens with both remote and local agent) ?

#

My Wyoming. faster-whisper-ct2 is the remote powered by an RTXA4000 GPU and the Large model 🙂 wheres whisperis the local running from add-on.

#

No, hold-on, didn't work with Google Cloud either, so it is not related to Whisper...

#

...then the only thing as far as I understand is causing this is (at least) the Norwegian language.

Anybody that can test the same from any other country than one speaking native English?

dapper tendon Feb 14, 2025, 2:41 PM

#

I'll test in a second (hopefully i can revive whisper), but please do me a favor: from the debug log, copy (actually copy, don't rewrite) what the STT has transcribed in a case where it does not process locally and paste that in the Assist dev tool. let me know if the sentence matches

#

i suspect there are some extra/odd characters in what the STT transcribes. I'm hoping that they get passed to the frontend

dreamy ember Feb 14, 2025, 2:45 PM

#

Here's the test

dapper tendon Feb 14, 2025, 2:45 PM

#

i'm stumped. @lusty tree please take a look at this, as you know the pipeline components much better than anyone

dreamy ember Feb 14, 2025, 2:51 PM

#

hehe, good that you're "out" also 🙂 To make it more intersting - I have a remote localtion (summer house) with full HA setup - did the same test there 30 mins ago and exact same result. There I run HA in Docker (Debian).

There I only have faster-whisper as an add-on

final solar Feb 14, 2025, 3:01 PM

#

I checked, it works as expected in german.

dreamy ember Feb 14, 2025, 3:03 PM

#

final solar I checked, it works as expected in german.

Harry, can you pls share screenshots here of the pipeline setup where it works with German language, LLM as conversational agent and Whisper or Google Cloud as STT and the debug where it works? So 2 screenshots please?

final solar Feb 14, 2025, 3:05 PM

#

google cloud, whisper and HA cloud.

#

dreamy ember Feb 14, 2025, 3:07 PM

#

The screenshot show Home Assistant Cloud as STT?

final solar Feb 14, 2025, 3:07 PM

#

one is HA cloud the other is whisper and google as STT

reef cedar Feb 14, 2025, 4:08 PM

#

Could it be that Whisper is producing some (invisible) symbols or some visible similar, but different symbols in the transcribed text, which leads to unmatched sentence?

dreamy ember Feb 14, 2025, 4:09 PM

#

Both whispers? The add-on and the remote? And this also happens with Google Cloud...

green harbor Feb 14, 2025, 4:12 PM

#

It is indeed possible, the whispers would behave the same regardless of being local or remote, they use the same internal model. It's possible Google Cloud's model behaves similarly

#

I think you'd have to analyze the string potentially and see if it is using a slightly different unicode character for instance

dreamy ember Feb 14, 2025, 4:17 PM

#

Sorry guys, I don't buy it - it can't be. Google Cloud and Whisper do the same "mistake" ?

At the summer house I used "Skru av lysene i spisestue" which means "turn off the lights in the dining room" and this sentence do not contain any Norwegian characters.

How would I troubleshoot something like this?

#

I now even pulled the log from the PE to see what is read from STT: "Skru av lysene på kontoret." This is plain text.

reef cedar Feb 14, 2025, 4:49 PM

#

dreamy ember Sorry guys, I don't buy it - it can't be. Google Cloud and Whisper do the same "...

It might be not Norwegian characters, but some other characters, even invisible ones. Like something instead of whitespace, that looks like whitespace but has different Unicode, so split() doesn't work on it, so it breaks matching.

dapper tendon Feb 14, 2025, 4:53 PM

#

dreamy ember Sorry guys, I don't buy it - it can't be. Google Cloud and Whisper do the same "...

Not the same, but similar, or with the same outcome - extra chars. I'm still waiting for Supreme Commander Mike to chime in 😅

lusty tree Feb 14, 2025, 6:09 PM

#

This might be a combination of a few things. The language of a pipeline is usually set by the conversation agent, but LLMs report supporting "all languages" so we have to do more digging to guess the intent language. We currently look at the STT and then TTS languages, and I'm wondering if STT being "no" vs. the intent language being "nb" is what's causing it to skip over local processing.

dreamy ember Feb 14, 2025, 6:34 PM

#

lusty tree This might be a combination of a few things. The language of a pipeline is usual...

Thanks @lusty tree any way I can troubleshoot this ?

#

Btw, do the language set for the user (user profile) in any way interfere or affecting what we set in the pipeline configurations ?

lusty tree Feb 14, 2025, 8:40 PM

#

dreamy ember Btw, do the language set for the user (user profile) in any way interfere or aff...

No, we just consider the pipeline information.

dreamy ember Feb 14, 2025, 8:42 PM

#

Will you be looking into this? I really dunno how I can assist in this atmo

lusty tree Feb 14, 2025, 8:42 PM

#

@dreamy ember I tested this locally with "skru på lysene på kontoret" and it works fine for me here 🤷‍♂️
It runs the local intent and then goes to the LLM if I add extra stuff.

dreamy ember Feb 14, 2025, 8:42 PM

#

U got to be shitting me! 🙂 What what?

lusty tree Feb 14, 2025, 8:43 PM

#

Let me try with Whisper...

dreamy ember Feb 14, 2025, 8:43 PM

#

lusty tree <@861504357513297921> I tested this locally with "skru på lysene på kontoret" an...

Could you pls share screenshots of pipeline config ?

dreamy ember Feb 14, 2025, 8:43 PM

#

lusty tree Let me try with Whisper...

What did u test with?

lusty tree Feb 14, 2025, 8:45 PM

#

Ah ha! It's something with Whisper. I tested originally with HA cloud.

dreamy ember Feb 14, 2025, 8:46 PM

#

HA Cloud always worked here also. Test with Google Cloud, will not work either

lusty tree Feb 14, 2025, 8:54 PM

#

OK, got it figured out and fixed. Just need to write up a test and make a PR. Thanks 👍

dreamy ember Feb 14, 2025, 8:56 PM

#

Wow! Great! Something I can test locally ?

lusty tree Feb 14, 2025, 9:35 PM

#

You can if you're able to pull in this PR locally: https://github.com/home-assistant/core/pull/138560

dreamy ember Feb 14, 2025, 10:02 PM

#

lusty tree You can if you're able to pull in this PR locally: <https://github.com/home-assi...

This will work on more than lights I guess? It will fix the "use local if intents exist" ?

lusty tree Feb 14, 2025, 11:38 PM

#

Yes, it should fix the general problem.

dreamy ember Feb 14, 2025, 11:43 PM

#

lusty tree Yes, it should fix the general problem.

Do I need to change both files or is it enough with the homeassistant/components/conversation/default_agent.py ?

lusty tree Feb 15, 2025, 1:29 AM

#

Just the default agent file should be enough.

dreamy ember Feb 15, 2025, 4:44 PM

#

lusty tree Just the default agent file should be enough.

Shouldn't I be able to copy the entire conversation folder from the components folder in dev and put it within my custom_componentsand do a reboot ? I can see your change have been merged to dev already

dapper tendon Feb 15, 2025, 4:53 PM

#

yes, you can

dreamy ember Feb 15, 2025, 5:03 PM

#

... but it didn't work

dapper tendon Feb 15, 2025, 5:04 PM

#

i'm guessing you did restart, right?

dreamy ember Feb 15, 2025, 5:05 PM

#

Indeed I did

#

#

My structure

dapper tendon Feb 15, 2025, 5:15 PM

#

by "didn't work" you mean you get the same issue, right?

#

i.e. Whisper + NO/NB does not process locally

dreamy ember Feb 15, 2025, 5:17 PM

#

No, it didn't read the default_agent.py from the custom_components - and I didn't get an error in the log either at boot

#

I did only copy the entire folder conversationfrom core/homeassistant/components/ in branch dev to my custom_components so I have custom_components/conversation/ locally

dapper tendon Feb 15, 2025, 5:30 PM

#

are you sure your Samba settings are correct? I see you're using Windows to copy it, so it could be a permissions issue, maybe

dreamy ember Feb 15, 2025, 5:32 PM

#

Never had it before - I've even made my own custom integration which works just fine which I did copy straight over

#"Prefer handling the commands locally" do not work for my native language