M5 Stack Atom Echo - OWW wakeword troubleshooting | Home Assistant | Page 1

tribal wren Jul 20, 2024, 8:20 AM

#

in the device page in the HA ESPHome integration, what options do you have for assist pipeline and what is selected?

#

#

and also in HA>>settings>>Voice Assistants>>Home assistant cloud (assist) settings , what do you have set for wakeword?

tiny tartan Jul 20, 2024, 8:50 AM

#

(Sorry for slow replies, I’m out shopping).

Yes, I have set my local Assist as preferred, and it shows up on the Echo configuration.

I have selected “ok nabu” as wakeword

tribal wren Jul 20, 2024, 8:58 AM

#

ok no problem, if you want to drop a message in here when you can go through things and test with the device , give me a ping @tribal wren

tiny tartan Jul 20, 2024, 9:27 AM

#

@tribal wren back now

#

I get this in the HA log:

2024-07-20 11:28:09.871 WARNING (MainThread) [homeassistant.components.esphome.manager] Voice assistant UDP server was not stopped

tribal wren Jul 20, 2024, 9:32 AM

#

anything in openwakeword logs? also what version of HA do you have installed ?

tiny tartan Jul 20, 2024, 9:32 AM

#

openwakeword.log:

Jul 20 08:44:59 nuc systemd[1]: Started OpenWakeWord.
Jul 20 08:45:00 nuc docker[1849078]: INFO:root:Ready
Jul 20 08:45:00 nuc docker[1849078]: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

#

Home Assistant 2024.7.2

#

Running with /usr/bin/docker run --name=openwakeword-omw -p 10400:10400 rhasspy/wyoming-openwakeword --preload-model ok_nabu

tribal wren Jul 20, 2024, 9:34 AM

#

it could be a network issue ... but that would be odd as API is connecting. I would say that the device isn't able to connect to openwakeword . the fact that you can press the button and issue a command says that the pipeline is ok ... so the device is streaming audio so the issue has to be with the openwakeword config

tiny tartan Jul 20, 2024, 9:39 AM

#

There is nothing in the whisper log either. So I'm not able to give Assist commands with button press either

#

Piper works when I test it in the Assist config dialog

tribal wren Jul 20, 2024, 9:42 AM

#

can you issue a command by typing in the assist box and it works ?

#

eg. turn x on

#

top right hand icon from dashboard

tiny tartan Jul 20, 2024, 9:44 AM

#

Yes, that works

#

In Norwegian 🙂

tribal wren Jul 20, 2024, 9:45 AM

#

i'm guessing your not Norwegian ?

tiny tartan Jul 20, 2024, 9:45 AM

#

I am 🙂

tribal wren Jul 20, 2024, 9:45 AM

#

oh good thats ok then lol

tiny tartan Jul 20, 2024, 9:45 AM

#

Sorry if I confused you

tribal wren Jul 20, 2024, 9:45 AM

#

is the esp and HA server on the same vlan / subnet ?

tiny tartan Jul 20, 2024, 9:46 AM

#

Yes.

#

Other esps works

#

The wyoming servers are listening on 0:0:0:0:<port> and :::<port>

#

So looks like both IPv4 and IPv6 are up

tribal wren Jul 20, 2024, 9:48 AM

#

i'm not too familiar with docker, i've never had success getting things to work. but at this point it has to be openwakeword issue as that is the first thing that is the first step ... then it goes to whisper and ends with piper

tiny tartan Jul 20, 2024, 9:49 AM

#

I'm fairly familiar with Docker. We run some large Kubernetes clusters at work.

tribal wren Jul 20, 2024, 9:50 AM

#

see my oww logs look like this ...

DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 2723606075372398
DEBUG:wyoming_openwakeword.handler:Client disconnected: 2723606075372398
DEBUG:wyoming_openwakeword.handler:Client connected: 2723612556999125
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 2723612556999125
DEBUG:wyoming_openwakeword.handler:Client disconnected: 2723612556999125
DEBUG:wyoming_openwakeword.handler:Client connected: 2723618706532943
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 2723618706532943
DEBUG:wyoming_openwakeword.handler:Client disconnected: 2723618706532943
DEBUG:wyoming_openwakeword.handler:Client connected: 2723623927668678
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 2723623927668678

tiny tartan Jul 20, 2024, 9:50 AM

#

Yeah, looks like the Echo is not connecting

tribal wren Jul 20, 2024, 9:51 AM

#

and when wakeword detected

DEBUG:root:Triggered o_k_Col_in (client=2723680115831680)
DEBUG:wyoming_openwakeword.handler:Client disconnected: 2723680115831680
DEBUG:wyoming_openwakeword.handler:Client connected: 2723690110187261
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 2723690110187261

#

o_k_Col_in is my wakeword... it's actually ok colin

#

I'm not sure if the containers need to be configured as host mode , that used to be the case before they changed to using the API

tiny tartan Jul 20, 2024, 9:54 AM

#

Not according to the documentation, but I'm happy to try that!

#

Yes - that worked!

#

Did that for whisper and openwakeword

#

Not required for piper

#

Hmm - worked once at least…

#

Whisper recognised a command, and piper replied

#

But I think that was triggered by button press and not wakeword

#

Whisper log

Jul 20 11:57:00 nuc docker[1863583]: INFO:__main__:Ready
Jul 20 11:58:12 nuc docker[1863583]: INFO:faster_whisper:Processing audio with duration 00:04.570
Jul 20 11:58:17 nuc docker[1863583]: INFO:wyoming_faster_whisper.handler: Slå på lysa.

#

But still nothing in openwakeword log

#

Should the button press behave similar as wakeword?

#

Not able to get anything through again :/

tribal wren Jul 20, 2024, 10:12 AM

#

the button press bypasses the wakeword .. so you just issue the command

#

looks like you are on the right track... i know it's not fixed but a step closer .. 🤷‍♂️

tiny tartan Jul 20, 2024, 10:14 AM

#

Yes, some of the parts are working at least 🙂

#

Looks like the Echo crashed though: https://dpaste.org/3VbbB

#

Also can't get button press to work

#

It only worked once

tribal wren Jul 20, 2024, 10:16 AM

#

yup looks like it maybe got stuck in the loop. are you running the containers on the same pi that HA is on ?

tiny tartan Jul 20, 2024, 10:17 AM

#

Yes, but they run on a NUC, so should have plenty of resources

#

Hmm - maybe running a bit low on the memory side. Perhaps use a smaller language model

#

I need to take a break from debugging. Thanks for the help so far!

tribal wren Jul 20, 2024, 10:21 AM

#

no probs. just found this don't know if it helps
If you are running HA in a Docker container and want to use other services (like Whisper/Piper/OWW) in their own containers, the main gotcha is networking.
The way to go about it is:
use host mode networking for the HA container (to allow incoming UDP connections on random ports, as ESPHome does)
forward ports for each voice service in its container to localhost on the host network
in HA, when setting up Wyoming integrations for each service, use the loopback interface and the designated port for each service (e.g. 127.0.0.1/10200)

#

obvs can forget the udp ports part

tiny tartan Jul 20, 2024, 10:24 AM

#

Current setup is:

HA container: host mode networking (has always beenn that)
Whisper container: host mode networking (started with port forwarding)
OpenWakeWord container: host mode networking (started with port forwarding)
Piper container: port forwarding (this works)
Wyoming integration: localhost:<port>

#

So but it should work with my initial setup as well. I have only seen one connection to Whisper, which was after switching to network host mode. I have not seen any conncetions to OpenWakeWord, neither with port forwarding nor network host mode

tiny tartan Jul 20, 2024, 11:13 AM

#

It really feels like a network issue. I have enabled some more logging in HA, and I get this every 5 seconds:

2024-07-20 13:10:44.167 DEBUG (MainThread) [homeassistant.components.assist_pipeline.pipeline] Timeout during wake word detection

And if I press the button on the Echo, I get

2024-07-20 13:10:37.954 DEBUG (MainThread) [homeassistant.components.assist_pipeline.pipeline] wake-word-detection result None

#

No errors from homeassistant.components.wyoming - it would be helpful if it had some info/debug logging when it connects though

#

Entity state seems a bit off for whisper. It did work once though. I don't know if this is meaningful to look at:

#

For comparison - piper (which has worked the entire time):

#

And openwakeword (which I have still not been able to connect to)

#

The "Assist in progress" does not activate when I press the button though, and the color of the light on the Echo does not change to green either:

#

So I'm not sure if the problem is on the Echo

tribal wren Jul 20, 2024, 1:20 PM

#

It might be worth testing with on board microwakeword which runs on the device. I'm not at my desk at the moment but can dig a config out for it later. There has been a new version that was released last week which should run on the echo.

tiny tartan Jul 20, 2024, 2:35 PM

#

Oh, cool. Last I read about microwakeword, the echo was not powerful enough

#

I also wonder if I should disable the wakeword and try to get it work properly with just the button

#

OK, if I just disable wakeword from HA, whisper works fine

#

Also with port forwarding.

#

So it's only wakeword that I struggle with

#

Will it always listen for 15 seconds?

Jul 20 16:41:45 nuc docker[1882590]: INFO:faster_whisper:Processing audio with duration 00:15.000
Jul 20 16:41:50 nuc docker[1882590]: INFO:wyoming_faster_whisper.handler: Skru på lysefarbeidsrommet.
Jul 20 16:42:12 nuc docker[1882590]: INFO:faster_whisper:Processing audio with duration 00:15.000
Jul 20 16:42:17 nuc docker[1882590]: INFO:wyoming_faster_whisper.handler: Skru på lyset i dette rommet.
Jul 20 16:42:37 nuc docker[1882590]: INFO:faster_whisper:Processing audio with duration 00:15.000
Jul 20 16:42:42 nuc docker[1882590]: INFO:wyoming_faster_whisper.handler: Skru på lyset i arbeidsrommet.

#

Ah, enabled debug logging for openwakeword with --debug - now I see the connects!

Jul 20 16:47:32 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Client connected: 3087922503079539
Jul 20 16:47:32 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 3087922503079539
Jul 20 16:47:37 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Client disconnected: 3087922503079539
Jul 20 16:47:38 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Client connected: 3087927820918419
Jul 20 16:47:38 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 3087927820918419

#

So either I've not configured the wakeword correctly, or my pronociation sucks

#

Ruled out network issue then!

#

I'm running with --preload-model 'ok_nabu'

#

Triggering with button stops working again when I enable "Use wake word" in HA

#

With wakeword enabled, the echo blinks red after pressing the button.

#

Aha! I switched to "Hey Jarvis", and now wakeword works!

#

Jul 20 16:57:04 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Jul 20 16:57:04 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 3088494452179285
Jul 20 16:57:09 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Client disconnected: 3088494452179285
Jul 20 16:57:09 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Client connected: 3088499588237177
Jul 20 16:57:09 nuc docker[1888150]: DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 3088499588237177
Jul 20 16:57:13 nuc docker[1888150]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=3088499588237177)

tribal wren Jul 20, 2024, 2:59 PM

#

well ... thats a good sign 🙂 🎉

tiny tartan Jul 20, 2024, 2:59 PM

#

Button press seems to be disabled when wakeword is enabled though

#

I can live with that

#

I suspect there is something wrong with --preload-model 'ok_nabu'

#

Have to make dinner. This was fun! Thanks for the help with the echo image yesterday

tiny tartan Jul 20, 2024, 3:36 PM

#

Important to run the wyoming containers with —debug when debugging 😅

tribal wren Jul 20, 2024, 3:37 PM

#

you're welcome, glad you have something sorted now, at least you know it works. 🙂

tiny tartan Jul 20, 2024, 3:39 PM

#

I can probably figure out the problem with ok_nabu model on my own. But I had not managed to build a working esp image without your help. Much appreciated

#

Reading about microWakeWord for Atom Echo now - seems a bit premature

#

Probably something I'd rather play with later this year

#

Found https://github.com/kahrendt/microWakeWord/releases/tag/v2.1_models

#

If you want to test these out now, you must be on the dev branch of ESPHome and use an external component

#

The problem with ok_nabu is my goddam bad English pronouciation. Switched back, managed to to trigger the wakeword once. That's about 99% failure rate for me. hey jarvis has 99% success rate for me so far…

tribal wren Jul 20, 2024, 5:51 PM

#

well your english typing is better than mine and i'm from the UK lol
you can make your own wakewords for openwakeword https://colab.research.google.com/drive/1q1oe2zOyZp7UsB3jJiQ1IFn8z5YfjwEb?usp=sharing#scrollTo=1cbqBebHXjFD

#

it takes about an hour to generate , but you can spell things phonetically to match how you pronounce

tiny tartan Jul 20, 2024, 8:41 PM

#

Yeah, I have that on my todo list. I read somewhere that only English is supported though? I was thinking to wait until I can create a wakeword in Norwegian. Also need to figure out what it should be

#M5 Stack Atom Echo - OWW wakeword troubleshooting