websocket | Cartesia | Page 1

celest lantern Oct 7, 2024, 4:54 PM

#

Hi I am using websocket tts and found some words are cut off and it happens every often. One example is like following (oh boy cuts off)
I wonder is this expected? any suggestion to improve?

https://aideate-media.com/audios/63/82/fa/6382fa5e5e6dbedddcde62480323c788.wav
transcription : REALLY?! You mean, a REAL treasure hunt?! With maps and riddles and secrets?! OH BOY! Count me IN! When do we start?!

brittle stone Oct 7, 2024, 9:55 PM

#

Hi @celest lantern - can you send an example snippet of the code you're using for generation?

celest lantern Oct 7, 2024, 10:09 PM

#

sure

#

        logger.debug(f"Generating TTS: [{text}]")

        try:
            if not self._websocket:
                await self._connect()

            if not self._context_id:
                self._context_id = str(uuid.uuid4())
            self._transcription.append(text)
            verbal_text = self._get_verbal_text(text)

            msg = {
                "transcript": verbal_text if verbal_text else " ",
                "continue": True,
                "context_id": self._context_id,
                "model_id": self._model_id,
                "voice": {"mode": "id", "id": self._voice_id},
                "output_format": self._output_format,
                "language": self._language,
                "add_timestamps": False,
            }
            try:
                await self._get_websocket().send(json.dumps(msg))
            except Exception as e:
                logger.error(f"{self} error sending message: {e}")
                await self._disconnect()
                await self._connect()
                return
            # yield None
        except Exception as e:
            logger.error(f"{self} exception: {e}")```

#

this is how i send transcript

unreal otter Oct 8, 2024, 5:02 PM

#

I just heard your generation and it is ok, no cut out. You can hear it is "spelling" B O Y.

#

(not a cartesia employee here, just been using cartesia since they release)

#

From what I know, you would need to preprocess your text to make it "boy" or, provide pronunciation tags as described in: https://docs.cartesia.ai/user-guides/custom-pronunciation-guide

Cartesia

Custom Pronunciations — Cartesia

Sonic supports inserting phonetic transcriptions for words that are hard to get right, like proper nouns or domain-specific terms.

celest lantern Oct 10, 2024, 5:09 PM

#

@brittle stone in terms of text to speech quality, we are seeing quite big difference when using SSE by sending whole text vs using websocket streaming in text in chunks. Is this something expected? the quality here I mainly referring to (1) if there is any words/text skipped (2) how natural the speech is, any chopping feeling

the way we streaming in text taking https://docs.cartesia.ai/reference/web-socket/stream-speech/working-with-web-sockets as a reference
a) make first chunk longer to be at least 4 words
b) sending by words in each chunk (LLM streaming out text could break words in between, we add buffer our side to make sure sending complete word. Not sure if cartesia side has similar logic to handle if a word is separated in two continuous chunks)

And lastly here is all for English voice case.

Cartesia

Working with WebSockets — Cartesia

Connect to Cartesia over a WebSocket and generate speech from a transcript using a given voice and model. The audio is streamed out as Base64-encoded raw bytes.

brittle stone Oct 10, 2024, 6:22 PM

#

Hi @celest lantern - hm just want to confirm if you're using continuations? Here's a quick rundown of the concept. On our end we recommend buffering to the first point of punctuation (or first sentence if that's easy) and then submitting word by word from there.

celest lantern Oct 10, 2024, 6:23 PM

#

@brittle stone yes, I am using continuations.

brittle stone Oct 10, 2024, 6:23 PM

#

Could you send an example of the two clips? The one generated with SSE and the one generated with websockets that's of poorer quality

celest lantern Oct 10, 2024, 6:25 PM

#

sure. Let me collect some examples and post here. Maybe take a while. will poke you again once I get the examples

celest lantern Oct 10, 2024, 8:23 PM

#

@brittle stone can I DM you the examples?

brittle stone Oct 10, 2024, 8:24 PM

#

Yep no worries

celest lantern Oct 10, 2024, 10:54 PM

#

@brittle stone separate question, we are also planning to expand to non-english languages. And when using websocket, any suggestion or guidance for how to chunk the worlds/characters? I assume the first chunk still requires a full sentence. for English, we are separate by whitespace to send by words. Wonder what is the best way for non-english cases. Thanks

brittle stone Oct 10, 2024, 11:16 PM

#

@celest lantern What languages? Latin is likely the same, the asian languages might uh.... require more nuance

celest lantern Oct 10, 2024, 11:20 PM

#

our current target language are Spanish and Chinese. and Thai, vietnamese also likely will be but I think Cartesia is not supported yet.
I am trying to find some doc on Cartesia website for guidance, have no luck with that.

brittle stone Oct 10, 2024, 11:24 PM

#

Ah gotcha, yah I'm guessing our multilingual documentation is lacking a bit. Curious what specific questions you have? Is it around the separation for languages like Mandarin?

For Mandarin you can probably use the full stop 。

celest lantern Oct 11, 2024, 12:07 AM

#

my use case is the text is generated by LLM as stream. each chunk of text can be break in middle but all the chunks put together form a complete text. I wonder if directly send those generated text, would it cause and quality drop? or only the first chunk matters?

#websocket