Streaming audio to voice assistant preview edition in real-time.

[mortal stratus]

Hello, I am trying to integrate TTS API to voice assistant but even though I use streamed response which I send to voice assistant in chunks, it still waits until stream ends. Is it possible to play stream in real-time?

I use logic like this in MQTT wyoming custom python script:

await self.write_event(AudioStart(rate=self.rate, width=self.width, channels=self.channels).event())

async for chunk in bytes_iter:
  await self.write_event(AudioChunk(audio=bytes(to_send), rate=self.rate, width=self.width, channels=self.channels).event())

await self.write_event(AudioStop().event())

Voice assistant waits until AudioStop() event is written. So even though TTS starts sending audio in 200ms, it sometimes waits for like 10 seconds before it starts playing which is kinda annoying.

Streaming audio to voice assistant preview edition in real-time.

[sharp pilot]

this question is probably better off in the developer support section #1257019582112334014 to see the developer channels you need to self assign the developer role in the id:customize section.

[hollow oasis]

Share a link to your project's full code. Or specify which API you want to integrate.
Then it will be easier to advise you.

It is absolutely possible to implement streaming.

[mortal stratus]

https://github.com/LadaBr/cartesia_tts/blob/main/custom_components/cartesia_tts/tts.py

I checked Google cloud and open ai TTS integrations and both of them fetch entire TTS audio before it is sent to satellite. So I feel like it isn't possible

Also for some reason voice assistant LED ring stopped spinning after last update during TTS playback. Previously there was spinning white circle during TTS but now it just completes task, turns LED ring off and then plays TTS audio

Is there any documentation for voice assistant (preview edition) development?

[hollow oasis]

I don't have a test HA for convenient integration development, so I put together an external server layer.
There is quite a lot of room for improvement, but I hope it will help you. The main task is to split text from LLM into short sentences. Instead of implementing your own solution (sentence_boundary), you can use the ready-made sentence_stream library.

I can also suggest digging through my repositories; hass-edge-tts will probably be the closest match for your task. https://github.com/mitrokun?tab=repositories

As far as I could see, last month's update to the 11labs integration also uses streaming and smart text segmentation.

[hollow oasis]

switched from using the library to direct queries and added language settings

[mortal stratus]

I can see same implementation there.

            await self.write_event(
                AudioStart(
                    rate=self.speech_tts.sample_rate,
                    width=self.speech_tts.sample_width,
                    channels=self.speech_tts.channels,
                ).event(),
            )

            async for chunk in self.speech_tts.synthesize(text, voice_id=voice_name, language=language):
                if chunk:
                    await self.write_event(
                        AudioChunk(
                            audio=chunk,
                            rate=self.speech_tts.sample_rate,
                            width=self.speech_tts.sample_width,
                            channels=self.speech_tts.channels,
                        ).event(),
                    )
            
            await self.write_event(AudioStop().event())

Voice assistant waits for AudioStop().event() before it starts playing

homeassistant library tts source code even says this:

it doesnt iterate chunks and expects all data as bytes

[hollow oasis]

You're a bit confused. HA fully supports streaming. However, due to limitations of local synthesis engines or providers, we have to create splitting mechanisms. If the provider being used is capable of independently accumulating/splitting/delivering audio chunks, then there's no problem using one generator for sending text data and one for receiving. If you're interested, I can show you a project using grpc working in this mode.

As for my server implementation, to the end user, it will appear as streaming. The AudioStop().event() wait is not global.

[hollow oasis]

We use a different method for streaming mode - async_stream_tts_audio.
https://github.com/mitrokun/hass-edge-tts/blob/main/custom_components/edge_tts/tts.py#L166
The integration must support both

[mortal stratus]

i didnt see that method in base class

[hollow oasis]

Before vibecoding, it is better to spend more time gathering information.

[mortal stratus]

What do you mean? I literally checked source code. It really is not my fault that TextToSpeechEntity is missing async_stream_tts_audio method ...like why would it have all other methods with raise NotImplementedError() or None return value and not this one...