If I haven't spoken to my voice assistant in a while (a few minutes or so), there's about a 50% chance that, when I say the wake word, the Voice PE actually listens. The rest of the time, it flashes a red circle or doesn't change the LED state at all. When it does this, I have to activate it three or more times (up to seven!) before it will actually accept voice commands.
It's recognizing the wake word because I hear the wake word sound effect each time I say it. It's almost like it's having trouble connecting to the server or something, but my Wi-Fi coverage is good and it never actually reports as offline.
I'm not very experienced with ESPHome devices, but I'm happy to provide logs if someone can direct me to where/how to retrieve them.
I can't seem to find others who have this problem, so I'm baffled by what could be going wrong. This is a huge annoyance to the smooth operation of my device, by far my #1 issue.