#merging mp3 snippets sounds unnatural

1 messages · Page 1 of 1 (latest)

storm quail
#

I am generating dialog between characters and then merging them together with ffmpeg. The audio has weird clipping and jumping sounds and sounds the audio does not sound very natural. This would seem to be a process that happens all the time on cartesia.

I also notice that on the generated mp3s I get from cartesia, they sometimes have different background noises, so when you merge them they would not merge nicely anyway, so it seems to be 2 issues happening here.

Is there a tutorial,docs, or official way to merge cartesia audio to make a longer audio file?
Here is the code I use now in ruby:

# Create list of files with silence between them
mp3_files = Dir.glob("#{dir}/*.mp3").reject { |f| f == silence_file }
  .sort_by { |file| file.scan(/\d+|\D+/).map { |s| s =~ /\d+/ ? s.to_i : s } }

output_file = "merged_#{id}.mp3"

Tempfile.open('mp3_list') do |tempfile|
  mp3_files.each_with_index do |file, index|
    tempfile.puts "file '#{File.expand_path(file)}'"
    # Add silence after each file except the last one
    if index < mp3_files.length - 1
      tempfile.puts "file '#{File.expand_path(silence_file)}'"
    end
  end
  tempfile.flush

  # Merge the mp3 files using ffmpeg
  cmd = "ffmpeg -f concat -safe 0 -i #{tempfile.path} -c:a libmp3lame -q:a 2 #{output_file}"
  puts cmd
  system cmd
end
native vessel
#

Let me know if you have any questions when reading through / testing

storm quail
#

@native vessel thanks, can you use them for different voices as well? also its says its for realtime. I dont need to do it in realtime, can it be done in a non realtime manner?

#

it looks like its for a single voice

native vessel
#

@storm quail wouldn't recommend trying to stitch different voices together with continuations haha. Uh realtime is the most common usecase because if it wasn't realtime you could just submit the whole transcript at once

storm quail
#

so how are people dealing with this with different voices. its the same problem that is outlined in that page

native vessel
#

Ah I think most of our users don't want to stitch different voices together as part of the same generation 😅 Prosody is relevant to a single person talking

#

Are you trying to make something that's more like two voices talking to each other?

storm quail
#

@native vessel yes, exactly, a story

storm quail
#

@native vessel also I noticed many of the voices are at different volumes.

#

The sonic model seems to have lots of issues that make it not production ready

native vessel
#

also I noticed many of the voices are at different volumes.
If you're just stitching the voices together you can use the /bytes endpoint. This has a field will have not yet publicized in our docs: loudness: float that uses the LUFS scale. If you specify the same number for each generation it should normalize the volume.

Having listened to your usecase, here would be my recommendation.

Generate mp3 files with the bytes endpoint using the loudness field. Sounds like you want to have some more natural pauses in between, I would recommend using break tags for that if you're unable to inject standard silence on your own. Then stitch together the clips with ffmpeg.