#Performance tips for Text to speech (custom voices)

1 messages · Page 1 of 1 (latest)

hidden bluff
#

While eleven_turbo_v2 is basically real time, I've found it doesn't support custom voices with the style attribute. So we use eleven_multilingual_v2 for our custom voices (created in VoiceLab).

Are there performance tips for speeding up custom voice generation?

What we've tried:

  1. Streaming API (/v1/text-to-speech/{voice_id}/stream) - this didn't work as expected. It generates most of the audio before starting the stream.
  2. Websocket API (/v1/text-to-speech/{voice_id}/stream-input) - this starts sending audio much faster, but you can't expect it to be "real time", as the audio isn't guaranteed to be sent fast enough, so the user could get audio lag/skipping, which is not a great UX.
  3. Reducing API latency (helpdesk article) - setting optimize_streaming_latency helps some, but doesn't quite get us to "real time" yet.

What we're planning to try:

  1. Divide up our text into sentences and parallelize the calls to see if we can get the full audio back faster, and stitch it together after the fact.
  2. Open the websocket connection as early as possible and start streaming text to it before we have our full transcript ready.

Anything else we could try to get closer to "real time" for custom voices? Willing to try literally anything 🙂

Loving the product, thanks!

patent yarrow
#

you've pretty much hit on all the possible latency optimizations - I tend to vary between the "split into sentences and parallelize" and "open websocket connection early" approaches depending on the situation

#

but no, there isn't much to be done beyond that - multilingual v2 is just very slow

#

especially if you enable style

#

if all you need is english, you could try out eleven_english_v2 if you have the alpha access

#

that one is fast enough to be realtime even with style on

polar pier
#

I'm also trying to make this as performant as possible. Does changing the output format on a TTS request reduce latency. E.g mp3_44100_128 -> mp3_44100_64?

patent yarrow
#

I did like, 25 runs, and the difference between formats was less than the run to run variance

hidden bluff
#

this is great info @patent yarrow thanks!

will try out eleven_english_v2 -- is there any info on getting alpha access? hadn't heard of it yet afaik 🙏

patent yarrow