My Conversational AI is not responding to any audio file that I sent over via websocket, all I'm re | ElevenLabs | Page 1

boreal wraithBOT Apr 11, 2025, 3:37 AM

#

AI-generated Answer for jiayun_jyun

If your Conversational AI is not responding to audio files sent via WebSocket and you're only receiving ping messages, there are a few things you might want to check:

Audio Format: Ensure that the audio files are in the correct format. Typically, the required format is 16,000 Hz PCM, 16-bit mono LE samples. Make sure your audio files meet these specifications.
WebSocket Connection: Verify that your WebSocket connection is correctly established. You should be using the appropriate endpoint, such as wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}. Ensure that the agent_id corresponds to an active and configured agent.
Audio Transmission: Confirm that the audio chunks are being sent correctly. They should be encoded in Base64 and sent in the user_audio_chunk format. Check that there are no disruptions in the WebSocket connection when sending these chunks.
Agent Configuration: Make sure the agent is properly configured to handle audio input and generate responses. If the agent is not set up correctly, it may not process the audio as expected.
Credits and Permissions: Ensure that you have sufficient credits to initiate the agent call and that all necessary permissions, such as microphone access, are granted.

If these steps do not resolve the issue, consider checking any available logs for errors or reaching out for further technical support.

compact yoke Apr 11, 2025, 3:43 AM

#

what websocket function do i call, websocket.sendText?

boreal wraithBOT Apr 11, 2025, 3:43 AM

#

AI-generated Answer for jiayun_jyun

The issue might be related to how you're sending the audio data over the WebSocket. Based on the provided information, ensure the following:

Audio Format: Verify that the audio data is being sent in the correct format specified by ElevenLabs. The WebSocket expects raw binary data, not Base64-encoded audio. If you're sending Base64-encoded audio, this could cause the connection to close or fail.
WebSocket Handling: When sending audio data, you should use a method that supports sending raw binary data. If you're using websocket.sendText, it might not be suitable for sending binary audio chunks. Instead, use a method like websocket.sendBinary or equivalent in your environment to send raw binary data.
Connection Stability: Ensure that the WebSocket connection remains open and that ping messages are being handled correctly. If the connection closes after sending audio, it could indicate an issue with the format or the way the data is being sent.
Initialization: Confirm that the WebSocket is initialized correctly and that the audio format (MP3 or WAV) is specified during initialization.

If the issue persists, you may need to review your WebSocket implementation and ensure it aligns with the examples provided by ElevenLabs. You can also check out their GitHub page for sample implementations.

compact yoke Apr 11, 2025, 4:10 AM

#

can you give me an example of the correct message?

boreal wraithBOT Apr 11, 2025, 4:10 AM

#

AI-generated Answer for jiayun_jyun

To send audio data to the Conversational AI via WebSocket, you need to ensure the audio is formatted correctly and sent in the expected structure. The audio should be:

PCM 16-bit mono
Sample rate of 16,000 Hz
Base64 encoded

The correct message format for sending audio chunks is typically structured as follows:

{
  "event": "user_audio_chunk",
  "audio_data": "BASE64_ENCODED_AUDIO"
}

Here’s an example of how you might send this message using a WebSocket client:

webSocket.sendText(
    """
    {
        "event": "user_audio_chunk",
        "audio_data": "BASE64_ENCODED_AUDIO"
    }
    """.trimIndent()
)

Make sure to replace "BASE64_ENCODED_AUDIO" with the actual Base64-encoded audio data. Additionally, you should send audio chunks sequentially every 250ms to maintain a natural conversation flow, as the platform handles pauses and silence automatically.

If the agent is not responding, verify the following:

The audio format matches the requirements.
You are sending the audio chunks consistently without long delays.
The agent responds normally to questions via the website UI.
You have sufficient credits to initiate the agent call.

If the issue persists, you may need to check your implementation against examples provided on ElevenLabs' GitHub page or reach out to support for further troubleshooting.

compact yoke Apr 11, 2025, 6:49 AM

#

Can you tell me whats the correct format for sending text transcription too

boreal wraithBOT Apr 11, 2025, 6:49 AM

#

AI-generated Answer for jiayun_jyun

To ensure proper communication with the ElevenLabs WebSocket, you need to follow these steps and format your messages correctly:

Initialization: When you establish the WebSocket connection, the first message you send must include the voice_settings and xi_api_key. This ensures the connection is properly initialized.
Sending Text: Text should be sent in chunks, and each chunk should end with a space to facilitate proper processing. After sending all text chunks, you must send an empty string as an End of Sequence (EOS) message to indicate the end of the input.

Example Message Format:

{
    "xi_api_key": "your_api_key",
    "voice_settings": {
        "stability": 0.5,
        "similarity_boost": 0.8
    },
    "text": "Your text here "
}

End of Sequence: After all text chunks are sent, send an empty string:
```
{
    "text": ""
}
```
Response Handling: Ensure your code processes the WebSocket responses correctly. Intermediate audio chunks may have fields like isFinal, alignment, and normalizedAlignment as null, but these should be present in the final response.

If you're still encountering issues, double-check your WebSocket implementation and ensure you're handling binary audio data correctly. If needed, share more details about your implementation for further assistance.

compact yoke Apr 11, 2025, 6:50 AM

#

whats the websocket.function()) for this

boreal wraithBOT Apr 11, 2025, 6:50 AM

#

AI-generated Answer for jiayun_jyun

The Conversational AI WebSocket is primarily designed to handle audio input, not text transcription. If you are sending audio, ensure the format is correct: PCM 16-bit mono, Base64, with a sample rate of 16,000 Hz. Audio chunks should be sent every 250ms to maintain a natural conversation flow. Silence and pauses are automatically handled by the platform.

For sending audio, you would typically use a WebSocket function like websocket.send() or websocket.sendText() depending on your implementation. The message structure should include the user_audio_chunk field containing the Base64-encoded audio data.

If you are trying to send text instead of audio, the platform does not support this functionality directly for live conversations. The AI is designed to process audio input and convert it into text internally. If you need text transcription, you may need to clarify your requirements or explore alternative solutions.

Let me know if you need further clarification!

compact yoke Apr 11, 2025, 6:51 AM

#

whats the difference between send and sendtext

boreal wraithBOT Apr 11, 2025, 6:51 AM

#

AI-generated Answer for jiayun_jyun

The provided content does not specify the difference between send and sendText or the exact WebSocket function to use. However, it does outline the correct format for sending messages via WebSocket to ElevenLabs.

To send text transcription or initiate a conversation, you need to ensure the WebSocket connection is properly established and send messages in the correct format. For example:

Example Message Format for Conversation Initiation:

{
  "type": "conversation_initiation_client_data",
  "conversation_config_override": {
    "agent": {
      "prompt": {
        "prompt": "You are a helpful AI assistant."
      },
      "first_message": "Hi! How can I help you today?",
      "language": "en"
    }
  }
}

Example Endpoint:

Ensure you are connecting to the correct WebSocket endpoint:

wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}

Sending Text Transcription:

To send text transcription, you would send a message containing the text input. The exact format for this is not explicitly provided, but it should align with the WebSocket API's requirements for user input.

If you're still experiencing issues, verify that your WebSocket connection is correctly initialized and that the agent_id corresponds to an active and configured agent. Additionally, ensure your input is formatted correctly and there are no disruptions in the WebSocket connection.

compact yoke Apr 11, 2025, 7:26 AM

#

Why does the agent only response every few user_audio_chunk sent

boreal wraithBOT Apr 11, 2025, 7:26 AM

#

AI-generated Answer for jiayun_jyun

The agent's response behavior may depend on how audio chunks are sent and processed. If the agent only responds after several user_audio_chunk messages, it could be due to the following:

Audio Chunk Size and Timing: Sending smaller chunks or sending them infrequently may not provide enough context for the AI to process and respond effectively. Larger or continuous audio chunks typically improve the AI's ability to generate responses.
Live Conversation Design: The Conversational AI is optimized for live conversations, meaning it expects a steady stream of audio data rather than sporadic chunks. Sending audio continuously, as opposed to waiting for a user to record and send a single chunk, aligns better with its intended functionality.
Flush Parameter: If the flush parameter is not used, the AI may wait to accumulate enough context before generating a response. This could explain delays in responses.
WebSocket Connection Stability: Ensure the WebSocket connection remains active and stable while sending audio chunks. Connection drops or interruptions could affect the agent's ability to respond consistently.

If you need further assistance, providing details about your implementation, such as the chunk size, timing, and WebSocket configuration, can help identify the issue.

compact yoke Apr 11, 2025, 8:49 AM

#

Livestreaming the audio seems to make the AI not understand anything at all, suspect that it is due to fragmented message

boreal wraithBOT Apr 11, 2025, 8:49 AM

#

AI-generated Answer for jiayun_jyun

It seems like the issue might be related to how the audio is being sent or processed. Fragmented messages or incomplete audio chunks can cause the AI to misinterpret or fail to process the input correctly. Here are some potential causes and suggestions:

Fragmented Audio Input: If the audio is being sent in small, disjointed chunks, the AI might not be able to reconstruct the full context. Ensure that the audio chunks are properly sequenced and complete before being sent.
Audio Quality: Poor audio quality or background noise can also lead to misinterpretation. Using a high-quality microphone and minimizing background noise can help improve accuracy.
WebSocket Implementation: If you're livestreaming audio, ensure that the WebSocket implementation is correctly handling the audio stream. The function you use (e.g., send or sendText) depends on the specific WebSocket library you're using. Typically, send is used for binary data (like audio), while sendText is for plain text.
Testing with Different Devices: If the issue persists, try testing on different devices or browsers. Chrome (latest version) is generally recommended for compatibility.

If the problem continues, it might be worth reviewing the audio streaming setup or reaching out to the support team for further assistance.

compact yoke Apr 16, 2025, 7:33 AM

#

sometimes the ai agent send multiple responses

boreal wraithBOT Apr 16, 2025, 7:33 AM

#

AI-generated Answer for jiayun_jyun

The AI agent's behavior of sending multiple responses could be due to the non-deterministic nature of the system, which can result in slight variations in its output. This is a common characteristic of AI systems, especially when generating responses based on complex prompts or instructions.

To address this, you can try refining the prompts or instructions given to the AI. For example, using more specific or clear directives might help reduce the likelihood of multiple responses. Additionally, testing different configurations or settings may help identify if there are any adjustments that can improve the consistency of the agent's behavior.

If the issue persists, you may want to reach out to the support team for further assistance or troubleshooting.

compact yoke Apr 16, 2025, 7:34 AM

#

which is more recommended for conversasion? Streaming the audio in continuous chunks or record a section/sentence and send it to the agent?

boreal wraithBOT Apr 16, 2025, 7:34 AM

#

AI-generated Answer for jiayun_jyun

For maintaining a natural conversation flow, it's recommended to stream audio in continuous chunks. This approach allows the platform to handle silence and pauses automatically, ensuring a smoother interaction. When sending audio, ensure it is in the correct format: PCM 16-bit mono, Base64 encoded, with a sample rate of 16,000 Hz, and send segments every 250ms. This setup helps the AI manage turn-taking effectively without needing explicit indicators for the "end of turn."

compact yoke Apr 16, 2025, 8:07 AM

#

how can i know if my agent is still 'talking' and not interrupt it? is there a message/event for that in websocket

#My Conversational AI is not responding to any audio file that I sent over via websocket, all I'm re

Example Message Format for Conversation Initiation:

Example Endpoint:

Sending Text Transcription: