Natural Language Processing takes long time when no result is found.

[vocal maple]

the Natural Language Processing of a question when no anwser is found is taking more than 16 seconds while if a intent is found it takes under a second. How can i decreate that long sorry i dont understand it responce.

[queen vine]

Are you sure the STT is working fine when that happens? Are you speaking loud enough? There is a 15s timeout for the VAD which seems to come into play in this scenario

[vocal maple]

Yes everything is working fine. I only found the fallover assistant hacs that can start a other agent when no responce is there. only waiting more than 15 secs is to much.

[subtle crescent]

So you are using failover agent, are you using it with an LLM?

[vocal maple]

No, i want to use a failover with an llm but it takes to long due to the long timeout of the home assistant agent

[subtle crescent]

you may need to debug your assist pipeline and see where the time is going

[ornate sandal]

I have the same issue but even longer delays which leads me to think the NLP timed out instead of gracefully determined the input sentence has no matching intent.

This is the result of a debug assist pipeline run. I specifically ran it in text mode so it is clear that there are no issues with STT and TTS but rather with intent matching.

~50 seconds to determine that there's no intent for that (deliberately) incorrect input sentence.

Compare this to a correct sentence that got matched in sub 4 seconds (I think this is still slow, but nothing compared to 50s).

[queen vine]

Oh wow! On what HA version is this happening? Is there any open issue about it?

[vocal maple]

I have the delay time from the debug

Natural Language Processing 16,64s ✅
Engine
conversation.home_assistant
Language
nl
Input
Hoe lang moet ik een ei koken?
Response type
error
Error code
no_intent_match

[vocal maple]

I also have 15 sec time out when i use the assist debugger in development tools

[subtle crescent]

nothing in system logs?

[vocal maple]

tail config/home-assistant.log does not log anything when i run the assist debugger

[ornate sandal]

same for me (HA Container though and monitoring through 'docker logs...')

[ornate sandal]

haven't had the time to raise an issue yet

[subtle crescent]

Is it a custom intent taking a long time, or just all intents?

I tried several requests to the HA assist pipeline, some that work and some that would not, and didn't notice any significant delays, most were under a second.

wonder if it's a language specific thing?

[ornate sandal]

I can only dream of such snappiness. It doesn't matter which intent. If I use a sentence that doesn't work it takes ages. I tried with a different language. It's still very slow when using a pipeline in english (~30-40sec) but a tiny bit faster than in german. Is there a way to debug the pipeline more verbosely? I would love to understand what it is doing in those 30-50 seconds.

My HA is running on a 4-core 1.3GHz CPU with 4GB RAM (Pine Rock64 -> Cortex A53). Not the fastest but not too slow. Resource monitor also doesn't show radical spiking in CPU or RAM usage while those 30-50 seconds tick.

[subtle crescent]

so no custom intents or anything?

I think in the past I had a bad custom intent that I saw messing things up or causing heavy compute due to it not having any domain restriction, so it was trying to look through my 300-400+ entities every request 😄

[queen vine]

4GB is slightly low, depending on what else you are running on it. But again i ask you to add an issue about this. It takes WAY too much

[ornate sandal]

no custom intents on my HA instance (custom_sentences folder) but only sentence triggered automations

[ornate sandal]

RAM usage looks fine though. It's a solid 50% usage all the time. Anyway, I'll raise an issue. Time to get rid of that and to make it work like it should BEFORE the assist HW comes out.

[unique thistle]

Yes, intent matching will definitely take longer when a response isn't found. This should get faster in 2024.12, but for now it's very slow in languages like German and Dutch.
What's happening is that the second intent matching pass is done in an open-ended manner so that a proper error message can be generated. If you say "turn on magic lamp" and there is no entity named "magic lamp", the second pass will try matching things like turn on * (where * is a wildcard). This produces a lot of matches, and we filter through them to make a best guess about what you were trying to do :/

[ornate sandal]

Thanks for the insights, that definitely helps understanding the inner workings. But when using an english pipeline, I still have 30 seconds of response time. So it will not get that much better in 2024.12 for me at least.

[unique thistle]

How many entities do you have? The second pass also includes all entities 😬

[ornate sandal]

All entities or all entities shared with Assist?

[unique thistle]

All entities in HA. We have to do this because one possible error condition is that you don't have something exposed.

[ornate sandal]

2100 entities. 89 shared with Assist.

[unique thistle]

Lol, OK well that's the problem 😄
We need to find a way to not push through every possible entity.

[ornate sandal]

Now we're getting somewhere 😄

[vocal maple]

Its a dutch language thing. English pipeline is responding fast.

[unique thistle]

It's going to depend on the language and on the number of entities in HA.

[vocal maple]

But why is dutch 12 secs longer. I have one custom intent in dutch

[unique thistle]

The Dutch sentence templates seem to be much more complex than the English ones. I see a lot of cases where whitespace is optional and there are multiple possible word endings.

[queen vine]

I have ~1700 entities, 145 of which are exposed. It takes 3s tops in RO for unmatched sentences

And there are some complex sentence templates in RO. I should know, i made them 😅

[fickle gate]

Hungarian is the same 😦 So many variations, even the tests take a long time to run in our language 😦 I understand now why it takes so long (15-30 seconds); I also have 1490 entities.

[ornate sandal]

Glad that my 2100 (now 2000, I got rid of some leftovers and dead weight) are not that far off from others. I was a bit worried after Mike's first response ("lol"). Considering that the number of entities is submitted through https://www.home-assistant.io/integrations/analytics/#statistics, it should be roughly clear what the typical HA user has and what needs to be considered when performance-testing features.

[fickle gate]

I know a lot of people who immediately turn off anything like this, and typically, these are exactly the hardcore users.

[ornate sandal]

Sad but true

[queen vine]

Make that 2700 after removing the Enabled filter. Not sure why i have 1000 disabled/unavailable/hidden entities though and i don't think they matter to Assist 🤔

[sudden lark]

Yes, in Dutch we normally add words together without a space, but this is also frequently done wrong (both while writing manually, but also by STT)
Also we have no real fixed word order, so we can say turn on ceiling light in kitchen but also turn in kitchen ceiling light on or in kitchen turn ceiling light on

And we can also use our verbs in different ways. zet lamp aan (turn light on) can also be said as lamp aan zetten

[ornate sandal]

Z2M is a main factor for me (3500 entities incl those disabled). Lots of entities that are disabled by default (good!).

[vocal maple]

My ha vm has 3.5 gig allocated 2 gig used max

[vocal maple]

Dropped down to 17 intents Natural Language Processing
11,82s need to look to my automations

Anyway i dont see high memory or cpu load when quering a foobar question with no result.

[ornate sandal]

same here. no significant change in resource consumption when the intent matching has to check all entities.

[unique thistle]

I wrote a script to count up the possible sentences per language, and...I think I'm seeing the scope of the problem 😄 (assuming I didn't make a coding error, but I've checked it many times now)
This is just counting the possible forms of a sentence, not all the different list or range values. So it would be something like "set {name} brightness to {brightness}", "set the {name} brightness to {brightness}", etc.
For English, there are 403,487 possible forms with HassDecreaseTimer having the most at 146,520.
Here are a few others:

• German: 7,375,726,420 (worst is HassLightSet with 5,212,372,684)
• French: 1,129,333,771,583 (worst is HassDecreaseTimer with 717,812,820,032)
• Hungarian: 97,303,707,498 (worst is HassLightSet with 67,855,224,000)
• Dutch: 191,382,248,828 (worst is HassTurnOn with 90,419,284,302)
• Romanian: 3,134,779,663 (worst is HassIncreaseTimer with 2,530,620,818)

I keep thinking I made a coding error, but every time I spot check a template the math works out. Combinatorics is crazy.

[vocal maple]

omg the thing i love about the dutch language now is some thing that stabs me in the back.

[fickle gate]

I'm a bit surprised that HassLightSet makes up 70% of Hungarian, but it's true that it has the most possibilities and is the one we use the most. I actually spent days optimizing it to work with free word order and to avoid having too many variations.

By the way, I'm currently working on whether the suffixes can be reduced, but after 5 hours of work, I haven't made much progress yet.

Would you consider sharing the script so I can see how much a modification helps?

[unique thistle]

Sure, I'll upload it to the intents repo after finishing one more script that reports how much time is spent on each sentence template 👍

[vocal maple]

Perhaps create a alternative limited instruction methods for creative languages like dutch. But train a LLM to teach users to use the right format.

[queen vine]

so that means that French has THREE ORDERS OF MAGNITUDE more possible sentences than German (i.e. 1 trillion vs 7 billion, and that's without factoring in all exposed entities, areas, aliases etc.), yet it hasn't got the same problem with slowdown on failed intents. am i getting this right?

[ornate sandal]

I just tested french vs german and used a sentence that I thought would result in no_intent_match for both sentences ("I don't speak <language>"). for french I got no_intent_match in 14sec for german no_valid_targets in 70sec.

here some results for a matched intent but no matched entity (HassTurnOn -> no_valid_targets). german 43sec, french 12sec.

french seems to be faster all the time

[sudden lark]

I've noticed the same thing in Dutch, it mentioned it can't find a specific device instead of mentioning it doesn't understand

@unique thistle

intent:
  name: HassListAddItem
slots:
  item: ich spreche ke
details:
  item:
    name: item
    value: ich spreche ke
    text: ich spreche ke
targets: {}
match: false
sentence_template: <item> (auf|in) <meine_liste>[ (setzen|schreiben|nehmen)]
unmatched_slots:
  name: deutch
source: builtin

It tries to add ich spreche ke on the list Deutch
It seems to ignore the (missing) whitespace between <item> and (auf|in) breaking up kein in two words

[ornate sandal]

Reminds me of https://github.com/home-assistant/intents/discussions/2193#discussioncomment-9670019

[fickle gate]

How can you do something like this? 🙂

[sudden lark]

It's the debug of the Voice pipeline

[fickle gate]

What I have is incredibly fast (image 1), and I added an area error as well as an entity error.

[woeful anvil]

trying the same, but parsing the sentences with the intentfest script in my dev-container - first with hassil 1.7.4 and second with hassil 2.0.2. Wondering why hassil 2.0.2 is slightly slower 🤨

DE -> 0.342s (hassil 1.7.4)

$ time python3 -m script.intentfest parse --language de --sentence 'ich spreche kein deutsch'

real    0m0.342s
user    0m0.316s
sys     0m0.025s

FR -> 0.457s (hassil 1.7.4)

$ time python3 -m script.intentfest parse --language fr --sentence 'je ne parle pas francais'

real    0m0.457s
user    0m0.432s
sys     0m0.025s

DE -> 0.352s (hassil 2.0.2)

$ time python3 -m script.intentfest parse --language de --sentence 'ich spreche kein deutsch'

real    0m0.352s
user    0m0.330s
sys     0m0.022s

FR -> 0.709s (hassil 2.0.2)

$ time python3 -m script.intentfest parse --language fr --sentence 'je ne parle pas francais'

real    0m0.709s
user    0m0.679s
sys     0m0.029s

[woeful anvil]

further the results are different - in FR you get "no intent matched", but in DE you get "no valid targets" 🤔
you can use the sentence "no intent match" in both languages, they results in no_intent_match response (not because of the sentence itself, but because it just not match anything)

[ornate sandal]

DE 70s, EN 14s, FR 12s

don't know why german is so bad on my machine

[woeful anvil]

Which system/machine do you use?

[woeful anvil]

found ist 😄

[ornate sandal]

this one. CPU and RAM are not exhausted before, during or after the NLU is doing its job

[woeful anvil]

@ornate sandal did you meanwhile created an issue report about it, as suggested by @queen vine above?

[ornate sandal]

Yes, I did. https://github.com/home-assistant/core/issues/130969

[sudden lark]

See my posts, it is matching it to the add to list intent

#1308109001124347955 message

[ornate sandal]

I know that english is a rather simple language but why is it so much better in permutations than other languages? Mere 400k permutations (EN) compared to 7 billion (DE) or 1 trillion (FR) seems weird to me.

[queen vine]

It depends a lot on how the sentence templates were written, and i've sacrificed readability for performance in EN (which did make it harder to contribute)

[fickle gate]

Just by removing the uninflected forms from a few places, the situation improved a lot. I also replaced some permissive parentheses () with []—this also made a significant difference. Additionally, there was a duplication in the most problematic sentence.

Current status: hu: 31,174,499,190

The other insane part is the question for sensor_HassGetState: <what_is_the_class_of_name>, but to address this, I’ll need to go through all the questions. That’s next on the list.

[ornate sandal]

@unique thistle What's the release schedule for https://github.com/home-assistant/intents? There have been massive improvements in DE by @woeful anvil in the recent PRs that I would love to test (sentences reduced by 97%).

[vocal maple]

wot.. enable beta channel

[woeful anvil]

beta channel will not bring them now. the intents repo needs to release it first, afterwards it get bumped in HA core.
to test the sentences, just create a config/custom_sentences/<language-code> folder and copy the response/<your_lang_code/*.yaml and sentences/<your_lang_code/*.yaml from the intents repo

[ornate sandal]

from 70sec down to 32sec

[vocal maple]

responses and sentences both in config/custom_sentences/<language-code>?

[ornate sandal]

alright, tested a few sentences and overall the reduction in possible sentences by 97% resulted in ~50% faster responses. Still unsure though why French is still considerably faster (while not yet optimized and wayyyy larger than the pool of DE sentences).

[woeful anvil]

you can also create sub-folders for response and sentences but they are not needed

[woeful anvil]

how many entities do you expose to the assist and how many do you have overall?

[ornate sandal]

I exposed 90 out of 2002 entities

[woeful anvil]

could you try "Schalte das blbablbabla Licht aus" (i mean really blablabla, so it find a possible intent, but no matching entity)

[ornate sandal]

no_intent_match and no_valid_targets both have come down from ~70sec to ~30sec

[vocal maple]

My neural processing went up to 80 seconds

For dutch

English is 2.5 second

Back to 9secs when i removed the yamls

[queen vine]

aka half a fck ton, which is still A LOT. there's got to be something else wrong, Michael did a great job reducing the number of sentences

[fickle gate]

Can't the current built-in intents interfere?

[ornate sandal]

absolutely. I'm grateful for any reduction that brings me closer to a responsive system. tbh I hadn't expected a 50% reduction (especially since I still don't know how FR can be faster with multiple fck tons of more sentence permutations). Michael did great here and IMHO this thread already lead to some valuable changes everyone will benefit from. Gotta love open source community projects!