Using continue with websockets leades to disorted voice | Cartesia | Page 1

uneven totem Jan 6, 2025, 10:33 AM

#

📎 message.txt

lean creek Jan 6, 2025, 4:12 PM

#

I would consider first that you are making a mistake before saying something is wrong to others.

#

With that said, your code is broken.

#

(not a cartesia employee)

onyx trench Jan 6, 2025, 5:40 PM

#

Hey @uneven totem - it's possible this is related to a JS Client issue that's been identified, let me take a closer look with the team and follow up. Appreciate the patience!

lean creek Jan 6, 2025, 5:46 PM

#

The other problem in that code is the use of multiple context.

onyx trench Jan 6, 2025, 5:53 PM

#

Oh hm I might be wrong but from his snippet it seems to be using the same context ID across send and continue

lean creek Jan 6, 2025, 6:08 PM

#

I think you are right.

#

One thought here, he has the whole text at hand, shouldn't be better to just send it in a single request?

#

I recall you guys saying it was ok now to send long texts as they will use the same "continuation" logic inside.

onyx trench Jan 6, 2025, 6:12 PM

#

Ah that is correct, @uneven totem if your usecase is one in which you know the full text you want to submit for generation you can just submit it all at once.

#

If not, we'll be fixing the client and following up with an update.

uneven totem Jan 7, 2025, 4:34 AM

#

The reason of doing this becuase of some feature that I want to implement, plus, I did tried submitting it all at once, and the problem that I face is sounds normal at the start, but somewhere near the end or middle, It gets really loud / really low, Due to which the story telling would sound soo bad

#

Shouldn't context Id be same for send and continue ?

#

Maybe I am making a mistake here

onyx trench Jan 7, 2025, 4:35 AM

#

somewhere near the end or middle, It gets really loud / really low, Due to which the story telling would sound soo bad
Hm that's definitely unintended behavior, if you could send us an example clip next time that happens that would be awesome!

#

Nah you're correct the context ID should be the same

#

So two things here:

The JS Client has been updated. If you update to 2.1.3 you should be able to add continue as True with your send and that should resolve your issues. You can see the diff here that was added today.
Unfortunately there is minimal functional difference between continuations vs sending a large input. Qualitatively if you're seeing issues with the large input, you'll likely be seeing them submitting the same transcript over continuations. So in an ideal world we can resolve the issues with the example you send over to us

GitHub

Comparing 2.1.2...2.1.3 · cartesia-ai/cartesia-js

The JavaScript client for the Cartesia API. Contribute to cartesia-ai/cartesia-js development by creating an account on GitHub.

#

To clarify with regards to (1) just incase you want to try it yourself, you just need to mark continue=True in your WebSocketTtsRequest that you pass to send

#

Lmk if you have any issues there

uneven totem Jan 7, 2025, 4:39 AM

#

Okkay, I'll send you audio as well where i got this issue

#

I'll check out updating the version

#

Thanks @onyx trench

uneven totem Jan 7, 2025, 5:42 AM

#

Getting the same issue with the update, Idk why

#

Screenshot_2025-01-07_at_11.13.17_AM.png

#

@lean creek ,Where is the code broken ?

lean creek Jan 7, 2025, 2:45 PM

#

I was wrongly assuming you where using different context_id for the generation, but after further inspection we concluded it was not the case.

lean creek Jan 7, 2025, 2:58 PM

#

uneven totem

That code is missing the continue: true on the first send request.

#

also, on the for loop, you should use websocket.continue instead of websocket.send

uneven totem Jan 9, 2025, 6:57 AM

#

@onyx trench , What should I do to get the playground audio quality for my website ?

onyx trench Jan 9, 2025, 7:14 AM

#

@uneven totem I believe the default sample rate we use on the Playground is 44100, cmiiw @forest orbit

forest orbit Jan 9, 2025, 7:14 AM

#

correct

uneven totem Jan 9, 2025, 7:31 AM

#

And ffmpeg ? @onyx trench , tbh the playground audio is smooth, I am streaming with mp3 format with below ffmpeg settings

#

#

Also the cartesia adds an unessary pause when speaking, this was the text
"The fluorescent lights of the tech support office buzzed faintly, casting a dull glow over the rows of cubicles. Mia sat hunched over her desk, her fingers flying across the keyboard as she tried to debug yet another faulty line of code."

#

Is it me or cartesia ?

#

The pause just before saying "casting a dull glow over the rows of cubicles"

#

@onyx trench

uneven totem Jan 10, 2025, 7:01 AM

#

Using continue with websockets leades to disorted voice

#

Hey @onyx trench , Could you help me in this ?

onyx trench Jan 10, 2025, 7:15 AM

#

Hi @uneven totem - what are the transcripts you sent above? Or did you send one large transcript.

The pause certainly sounds unnatural - while we've seen the model inadvertently do this on occasion I wouldn't expect it. What voice did you use?

uneven totem Jan 10, 2025, 7:36 AM

#

This are the settings and I am sending ASMR lady id

uneven totem Jan 10, 2025, 7:36 AM

#

onyx trench Hi <@1123872587089448982> - what are the transcripts you sent above? Or did you ...

Uhmm, I did send multiple transacript

onyx trench Jan 10, 2025, 7:42 AM

#

Ah that makes sense - the more experimental controls you add to a voice the less stable it will be

uneven totem Jan 10, 2025, 9:02 AM

#

Ohh, Okkayy

#

And can you let me know about how i could achieve a playground type quality using websockets for data.

winter forum Jan 10, 2025, 10:02 AM

#

I also noticed that over WS I receive a bit worse quality than the playground.

And sometimes the voice sounds very weird, for example it's trying to "overplay" some emotions.

I am not using any experimental feature right now.

{
  context_id: contextId,
  transcript: chunk,
  model_id: languageCode === 'en' ? 'sonic-english' : 'sonic-multilingual',
  voice: {
    mode: 'id',
    // "Helpful Woman"
    id: '156fb8d2-335b-4950-9cb3-a2d33befec77',
  },
  output_format: {
    container: 'raw',
    encoding: 'pcm_mulaw',
    sample_rate: 8000,
  },
  language: languageCode,
  continue: true,
}

lean creek Jan 10, 2025, 2:34 PM

#

mulaw is a compression format meant for telephony. In the play ground, the use the regular integer version that requires double the bitrate. That are also using higher sample rate, which also increases the bitrate

#

I think they use 44100

#

I’m using mulaw with sample rate 21050 and it sounds good enough for my use case.

winter forum Jan 10, 2025, 4:01 PM

#

got it, thanks.

uneven totem Jan 13, 2025, 8:33 AM

#

@onyx trench, Can you please let me know these things ?
* what your actual chain for processing the audio is
* do you use hls? what bitrate settings?
* do they do anything to normalize the audio after (eg. make it all the same volume)?
* in general our doesn’t sound crisp at all?

The sound which i generate from the websocket data, is not as good as how the playground sounds like

onyx trench Jan 14, 2025, 12:48 AM

#

Hi @uneven totem - the Playground is just using the JavaScript SDK to call the API under the hood, and it's just using this output format:

container: "raw"
encoding: "pcm_f32le"
sample_rate: 44100

uneven totem Jan 14, 2025, 4:31 AM

#

Okay @onyx trench , But I mean there would be a way you guys are processing the raw format, The audio output i get is not as good as the playground, Also I am using the same output format

uneven totem Jan 14, 2025, 5:27 AM

#

This audio is generated by me using websocket api

#

This is the one generated by playground

lean creek Jan 15, 2025, 1:42 AM

#

You shared an MP3 and are comparing it to a WAV file.

#

MP3 is a compressed file (you can notice the big difference in size) so it will have lower audio quality by its nature.

#

Also, MP3 was develop for Music compression, so it wont do a good job for voice, specially at lower bitrates.

#Using continue with websockets leades to disorted voice