client tool calling with wss agent websocket connection | ElevenLabs | Page 1

silver etherBOT Jan 28, 2025, 12:15 AM

#

AI-generated Answer for dicartagena

To establish a WebSocket connection with an agent using the wss protocol, you can follow these steps:

1. Using Agent ID (Public Agents)

For public agents, you can directly use the agent_id in the WebSocket URL without requiring additional authentication. The WebSocket URL format is:

wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>

2. Using a Signed URL (Private Agents)

For private agents or when authentication is required, you need to generate a signed URL from your server. Here's an example of how to do this:

Server-Side Code (Node.js)

const fetch = require('node-fetch');

app.get("/signed-url", async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/get_signed_url?agent_id=${process.env.AGENT_ID}`,
    {
      method: "GET",
      headers: {
        "xi-api-key": process.env.XI_API_KEY, // Your ElevenLabs API key
      },
    }
  );

  if (!response.ok) {
    return res.status(500).send("Failed to get signed URL");
  }

  const body = await response.json();
  res.send(body.signed_url);
});

Client-Side Code

const response = await fetch("/signed-url");
const signedUrl = await response.text();

const conversation = await Conversation.startSession({ signedUrl });

3. Maintaining the WebSocket Connection

To ensure a stable connection:

Keep the WebSocket connection open throughout the session.
Send periodic keep-alive messages (e.g., ping/pong) to prevent disconnections.
Limit the number of times the connection is closed and reopened, as the SSL/TLS handshake adds overhead.

4. Example WebSocket Endpoint

For a public agent:

wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>

For a private agent (with a signed URL):

wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>&token=<token>

Additional Notes

Ensure the WebSocket URL uses the wss:// protocol.
If you encounter issues, verify that the agent_id or signed URL is correctly passed to the WebSocket connection.

Let me know if you need further assistance!

mossy cradle Jan 28, 2025, 12:16 AM

#

with python

silver etherBOT Jan 28, 2025, 12:16 AM

#

AI-generated Answer for dicartagena

To establish a WebSocket connection with an agent using Python, you can follow the steps below. This example demonstrates how to connect to the ElevenLabs WebSocket API for real-time conversational AI.

Prerequisites

Install the required libraries:
```
pip install websockets python-dotenv
```
Set up your .env file with your API key:
```
ELEVENLABS_API_KEY=your_api_key_here
```

Example Code

Here’s a Python example to connect to the WebSocket API:

import os
import asyncio
import websockets
from dotenv import load_dotenv

# Load API key from .env file
load_dotenv()
API_KEY = os.getenv("ELEVENLABS_API_KEY")
AGENT_ID = "your_agent_id_here"  # Replace with your agent ID

async def connect_to_agent():
    # Generate the WebSocket URL
    websocket_url = f"wss://api.elevenlabs.io/v1/convai/conversation?agent_id={AGENT_ID}"

    # Add headers for authentication if required
    headers = {
        "xi-api-key": API_KEY
    }

    async with websockets.connect(websocket_url, extra_headers=headers) as websocket:
        print("Connected to the agent!")

        # Example: Send a message to the agent
        await websocket.send('{"user_audio_chunk": "<base64-encoded-audio-data>"}')
        print("Message sent to the agent.")

        # Listen for responses
        while True:
            response = await websocket.recv()
            print(f"Received: {response}")

# Run the WebSocket connection
asyncio.run(connect_to_agent())

Key Points

WebSocket URL: Use the format wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>.
Authentication: For private agents, include the API key in the headers or use a signed URL.
Audio Data: Ensure audio chunks are in PCM 16-bit mono format, base64-encoded, and sampled at 16,000 Hz.
Keep-Alive: Send periodic keep-alive messages (e.g., a single space " ") to maintain the connection.

Debugging Tips

If the connection fails, verify the agent ID and API key.
Ensure your network allows WebSocket connections.
Check the WebSocket API documentation for additional details.

Let me know if you need further assistance!

mossy cradle Jan 28, 2025, 12:21 AM

#

how to send audio with defaultaudiointerface?

silver etherBOT Jan 28, 2025, 12:21 AM

#

AI-generated Answer for dicartagena

To send audio using the `DefaultAudioInterface in Python with the ElevenLabs Conversational AI SDK, you can follow these steps:

Install the Required Packages: Ensure you have the elevenlabs package installed. If you want to use the default audio input/output, you also need the pyaudio extra.
```
pip install "elevenlabs[pyaudio]"
```

Set Up the Script: Use the DefaultAudioInterface to handle audio input/output. Below is an example script:

import os
import signal
from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface

# Load environment variables for agent ID and API key
agent_id = os.getenv("AGENT_ID")
api_key = os.getenv("ELEVENLABS_API_KEY")

# Initialize the ElevenLabs client
client = ElevenLabs(api_key=api_key)

# Set up the conversation with DefaultAudioInterface
conversation = Conversation(
    client=client,
    agent_id=agent_id,
    requires_auth=bool(api_key),  # Authentication required for private agents
    audio_interface=DefaultAudioInterface(),  # Use default audio input/output
    callback_agent_response=lambda response: print(f"Agent: {response}"),
    callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
)

# Start the conversation session
conversation.start_session()

# Handle clean shutdown on Ctrl+C
signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())

# Wait for the session to end
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")

Run the Script: Save the script as conversation.py and run it in your terminal. Make sure to set the AGENT_ID and ELEVENLABS_API_KEY environment variables before running the script.
```
AGENT_ID=your_agent_id ELEVENLABS_API_KEY=your_api_key python conversation.py
```

This script uses the DefaultAudioInterface to capture audio from your system's default microphone and play the agent's responses through the default speakers. You can customize the callbacks to handle the conversation data as needed.

mossy cradle Jan 28, 2025, 12:23 AM

#

actually i dont need that for testing i want to just call client tools through this websocket connection

silver etherBOT Jan 28, 2025, 12:23 AM

#

AI-generated Answer for dicartagena

To call client tools through the WebSocket connection for an agent in Python, you can establish a WebSocket connection to the ElevenLabs Conversational AI endpoint. Below is an example of how you can set up the connection and send data:

Steps to Connect to the Agent WebSocket

Install Required Libraries:
Install the websockets library if you haven't already:
```
pip install websockets
```
Set Up the WebSocket Connection:
Use the WebSocket URL for the agent. If the agent is private, you will need a signed URL. For public agents, you can directly use the agent_id.
Send Data:
You can send audio or other data to the WebSocket connection.

Here’s an example Python script to connect to the WebSocket and send a message:

import asyncio
import websockets
import json

async def connect_to_agent():
    # Replace with your agent_id
    agent_id = "<your-agent-id>"
    websocket_url = f"wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}"

    try:
        async with websockets.connect(websocket_url) as websocket:
            # Example: Sending a test message
            message = {
                "user_audio_chunk": "<base64-encoded-audio-data>"  # Replace with actual audio data
            }
            await websocket.send(json.dumps(message))
            print("Message sent to the agent.")

            # Listening for a response
            while True:
                response = await websocket.recv()
                print("Response from agent:", response)

    except websockets.exceptions.ConnectionClosedError as e:
        print(f"WebSocket connection closed unexpectedly: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")

# Run the async function
asyncio.run(connect_to_agent())

Notes:

Replace <your-agent-id> with the actual agent ID.
If the agent is private, you need to generate a signed URL using your API key. Refer to the WebSocket - ElevenLabs documentation for details.
For sending audio, ensure the audio is in PCM 16-bit mono format, base64-encoded, and sampled at 16,000 Hz.

This script demonstrates how to establish a connection and send data to the agent. You can adapt it further based on your specific requirements.

mossy cradle Jan 28, 2025, 12:35 AM

#

yeah but do i have to set up to register the client tool or is that done in the agent dashboard?

silver etherBOT Jan 28, 2025, 12:35 AM

#

AI-generated Answer for dicartagena

Based on the provided content, there is no specific information about setting up or registering a client tool for use with a WebSocket connection or whether this is handled in the agent dashboard.

If you are working with the ElevenLabs platform, you may need to configure your agent and test it through the dashboard. You can also interact with your agent using the provided SDKs (e.g., JavaScript or Python) or by deploying the widget.

For further clarification or specific setup instructions, you may need to consult the Help Center or the Product Documentation.

mossy cradle Jan 28, 2025, 12:59 AM

#

i dont need the conversation tested on client side. i just wantto test it on the agent dashboard

silver etherBOT Jan 28, 2025, 12:59 AM

#

AI-generated Answer for dicartagena

To test your AI agent directly on the agent dashboard, you can use the "Test AI Agent" functionality available in the Conversational AI section. This allows you to interact with your agent without needing to set up a client-side tool.

If you want to test the agent's behavior, ensure the following:

The agent is properly configured with a system prompt and first message.
The agent's settings, such as the LLM and voice, are correctly set up.
You can use the "Talk to Agent" feature in the dashboard to simulate a conversation.

No additional registration for client tools is required if you're testing directly on the dashboard. Let me know if you need further clarification!

mossy cradle Jan 28, 2025, 12:59 AM

#

yeah but i want to call the client tools ive put into my code

silver etherBOT Jan 28, 2025, 12:59 AM

#

AI-generated Answer for dicartagena

To call client tools through the WebSocket connection, you can use the parameters object during session initialization to pass custom tool and prompt values. This allows you to override or specify the tools you want to use in your code. Here's an example in Python:

import asyncio
import websockets
import json

async def connect_to_agent():
    # Replace with your agent's WebSocket URL
    websocket_url = "wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<your-agent-id>"

    # Define custom tool overrides
    custom_tool_overrides = {
        "agent": {
            "prompt": {
                "prompt": "Welcome to our service. How can I assist you today?"
            },
            "first_message": "Hello, this is a custom message."
        }
    }

    async with websockets.connect(websocket_url) as websocket:
        # Send initialization parameters
        await websocket.send(json.dumps({
            "parameters": {
                "toolOverrides": json.dumps(custom_tool_overrides)
            }
        }))

        # Listen for responses
        while True:
            response = await websocket.recv()
            print("Received:", response)

# Run the WebSocket connection
asyncio.run(connect_to_agent())

Key Points:

Replace <your-agent-id> in the websocket_url with your actual agent ID.
Customize the custom_tool_overrides object to include the tools and prompts you want to test.
This setup does not require any additional registration for client tools if they are already configured in your agent dashboard.

If you want to test the tools directly on the agent dashboard, you don't need to set up anything in your code. The tools should already be available for use if they are properly configured in the dashboard.

mossy cradle Jan 28, 2025, 11:41 PM

#

does tool call using post request have a time out?

silver etherBOT Jan 28, 2025, 11:41 PM

#

AI-generated Answer for dicartagena

The provided content does not specify whether a tool call using a POST request has a timeout. If you have more details about the setup or specific behavior you're experiencing, feel free to share, and I’ll do my best to assist!

mossy cradle Jan 28, 2025, 11:41 PM

#

yeah i would like to know what time time limit

silver etherBOT Jan 28, 2025, 11:41 PM

#

AI-generated Answer for dicartagena

The provided content does not specify a time limit or timeout for POST requests. If you need detailed information about timeouts or other API-related specifics, I recommend checking the API References or contacting the support team for clarification.

mossy cradle Jan 28, 2025, 11:42 PM

#

how long can a post request wait for a response

#client tool calling with wss agent websocket connection

1. Using Agent ID (Public Agents)

2. Using a Signed URL (Private Agents)

Server-Side Code (Node.js)

Client-Side Code

3. Maintaining the WebSocket Connection

4. Example WebSocket Endpoint

Additional Notes

Prerequisites

Example Code

Key Points

Debugging Tips

Steps to Connect to the Agent WebSocket

Notes:

Key Points: