#Can i use the same ai agent to serve both telephony and web voice chat?
1 messages · Page 1 of 1 (latest)
Yes. ElevenLabs Agents are built to be deployed across channels, so you can use the same agent configuration/ID for both telephony and a web voice widget. The platform explicitly supports deploying one agent across web, mobile, and phone lines, and emphasizes a unified agent across channels. (elevenlabs.io)
How to do it:
- Web voice chat: Embed the widget with your agent-id. Note: the widget currently requires a public agent with authentication disabled. (elevenlabs.io)
- Telephony: Connect your existing numbers via a supported integration (e.g., Twilio or Telnyx) and route calls to the same agent. (elevenlabs.io)
This setup lets a single agent serve both your website’s voice chat and your phone lines. (elevenlabs.io)
Sources:
Integrate | ElevenLabs Documentation
Widget customization | ElevenLabs Documentation
Connect Twilio to ElevenLabs Conversational AI Voice Agents
Integrate | ElevenLabs Documentation
in this case, what is the "User input audio format" and "TTS output format" I should set for this single agent? As from my understanding, for telephony it must need to set the agent to use "ÎĽ-law 8000 Hz", would it affect the web widget or other channels's perfomance?
Short answer
- For telephony: set both “User input audio format” and “TTS output format” to μ-law 8000 Hz. This is explicitly required in the Telephony guides (e.g., Twilio and Genesys). (elevenlabs.io)
- For web voice chat (widget/WebSocket clients): use PCM 16 kHz for user input, and PCM 16 kHz for agent TTS output. ElevenLabs’ realtime STT recommends pcm_16000 for input, and the agent WebSocket metadata examples show pcm_16000 for both directions by default. (elevenlabs.io)
Will a single ÎĽ-law 8 kHz setting affect the web widget?
- Yes, if you keep one agent configured to μ-law 8000 Hz, channels that use that agent’s defaults (like the web widget) will also negotiate/play telephony‑optimized 8 kHz audio, which is lower fidelity than PCM/MP3 typically used on the web. ElevenLabs documents μ-law 8 kHz as “optimized for telephony.” For best browser audio quality, prefer PCM or MP3 at higher sample rates. (elevenlabs.io)
Practical options
- One agent for all channels (simple, lowest effort): set ÎĽ-law 8000 Hz for telephony compatibility and accept narrowband audio quality on the web. (elevenlabs.io)
- Two agents (recommended for best quality): clone your agent and keep one configured for telephony (ÎĽ-law 8000 Hz) and another for web (PCM 16 kHz). The widget docs cover how to embed the web agent; your telephony agent remains configured per its integration guide. (elevenlabs.io)
Notes
- If you connect via SIP trunking, ElevenLabs’ SIP deployment sends/receives G.711 (8 kHz) or G.722 (16 kHz) codecs at the transport layer; this is described as independent of the agent’s direct WebSocket audio format, which can reduce format juggling for that path. (elevenlabs.io)
References
- Twilio telephony setup requiring ÎĽ-law 8000 Hz (input and output). (elevenlabs.io)
- Genesys telephony setup requiring ÎĽ-law 8000 Hz (input and output). (elevenlabs.io)
- Realtime STT supported formats (recommendation for pcm_16000). (elevenlabs.io)
- Agent WebSocket metadata showing pcm_16000 defaults. (elevenlabs.io)
- TTS formats overview noting μ-law 8 kHz is telephony‑optimized. (elevenlabs.io)
- Widget embedding
then, can i connect muliple telephone numbers to one agent?
Yes. Multiple inbound-capable phone numbers can point to the same agent. Assignment is done per phone number, and each number has a single assigned_agent at any given time—so many numbers → one agent is supported, but one number → one agent at a time. See “Assign your agent” for Twilio numbers and SIP trunk numbers, and the phone number object schema showing assigned_agent. (elevenlabs.io)
Notes:
- Verified Twilio Caller IDs are outbound-only and cannot be assigned for inbound routing. (elevenlabs.io)
- For outbound calls you choose which agent handles the call each time, regardless of the number used. (elevenlabs.io)
- Capacity is governed by your workspace concurrency, shared across all numbers/agents. (help.elevenlabs.io)
References:
- Twilio native integration: assign agent, inbound/outbound capabilities. (elevenlabs.io)
- SIP trunking: import numbers and assign to agents. (elevenlabs.io)
- API: phone number model includes assigned_agent (one agent per number). (elevenlabs.io)
- Concurrency and burst capacity. (help.elevenlabs.io)
Sources:
Twilio native integration | ElevenLabs Documentation
Twilio native integration | ElevenLabs Documentation
Twilio native integration | ElevenLabs Documentation
How many ElevenAgents requests can I make and can I increase it? – ElevenLabs
Twilio native integration | ElevenLabs Documentation
SIP trunking | ElevenLabs Documentation
Get phone number | ElevenLabs Documentation
How many ElevenAgents requests can I make and can I increase it? – ElevenLabs
then can i use a same agent to support both voice and text chat? is there any text chat API to blend the chat to my website landing page?
Short answer: Yes.
-
Same agent, voice and text: ElevenLabs Agents support both voice and chat. You can run the very same agent in text-only “Chat Mode,” or allow users to switch between voice and text in the website widget. If you already have a voice agent, you can enable Chat Mode immediately. (elevenlabs.io)
-
Embed on your website: Use the ConvAI web widget. Add a single <elevenlabs-convai> tag and the embed script to your page. The widget is multimodal (voice + text) and can be configured as Voice only, Voice + text, or Chat Mode. Note: as documented today, widgets require a public agent with auth disabled; use the allowlist to restrict domains. (elevenlabs.io)
-
Text chat API options:
- JavaScript SDK (@elevenlabs/client): Create a Conversation and send text messages (e.g., conversation.send_user_message) and handle agent_response callbacks. You can also enforce text_only via runtime overrides. (elevenlabs.io)
- WebSocket API: Connect to wss://api.elevenlabs.io/v1/convai/conversation?agent_id=... and exchange text events if you prefer a lower-level integration. (elevenlabs.io)
-
Audio settings note: Chat Mode is text-only and does not use audio I/O, so telephony-specific audio formats (e.g., ÎĽ-law 8 kHz) do not affect text chat. (elevenlabs.io)
References:
- ElevenLabs blog: Agents now support Chat Mode. (elevenlabs.io)
- Widget customization (embedding, voice/text modes, security notes). (elevenlabs.io)
- Chat Mode guide (SDK overrides, sending text, callbacks). (elevenlabs.io)
- JavaScript SDK docs (Conversation API). (elevenlabs.io)
- WebSocket API docs. (elevenlabs.io)
Sources:
ElevenLabs — ElevenLabs Agents now support Chat Mode
Widget customization | ElevenLabs Documentation
JavaScript SDK | ElevenLabs Documentation
WebSocket | ElevenLabs Documentation
Chat Mode | ElevenLabs Documentation
ElevenLabs — ElevenLabs Agents now support Chat Mode
Widget customization | ElevenLabs Documentation
Chat Mode | ElevenLabs Documentation
JavaScript SDK | ElevenLabs Documentation
WebSocket | ElevenLabs Documentation
For setting up an English Speaking AI Agent (preferred aussie accent), what is the best Femaie voice you would suggest to me for the most natural and response effect. I know when creating an new agent it will use Eric, and i like that but i need a female voice. What would be your suggested voice for a Female in the same quality?
Short answer: Use the Default voice “Brenna” for an Australian-accented female agent. It’s the current Australian female Default voice (replacing “Mimi”) and is suitable for real‑time agents. (elevenlabs.io)
Additional guidance for “natural + responsive”:
- Model for responsiveness: run the agent on Flash v2.5 (ultra‑low latency ~75 ms; recommended for Agents). Voice choice doesn’t change model latency materially, but Flash is the key lever for “snappy” turn‑taking. (elevenlabs.io)
- If you need maximum naturalness over speed, consider Turbo v2.5; it trades a bit more latency for higher quality. (elevenlabs.io)
- Note on voice types vs latency: Default/Synthetic/Instant Clone voices tend to start faster than Professional Voice Clones (PVC). If you pick a PVC from the Library, expect slightly higher TTFB on Flash. (elevenlabs.io)
Finding more Aussie female options:
- Use the Voice Library filters (Language/Accent + Gender) to discover additional Australian English female voices, then add to My Voices. (help.elevenlabs.io)
- Accent comes from the voice itself; pick a voice trained in your target accent (or clone one) for best pronunciation. (help.elevenlabs.io)
Practical setup notes:
- You can set an agent’s voice (and change it anytime); agents fully support custom voice selection. (help.elevenlabs.io)
- To reference a specific voice in API or configs, copy its voice_id from My Voices or list voices via the API. (help.elevenlabs.io)
Optional polish for “responsive feel”:
- Slightly increase the Speed setting (e.g., ~1.05–1.10) if you want snappier delivery; available across voices/models. (help.elevenlabs.io)
If you want, I can help you switch your agent to “Brenna” now or shortlist a couple of Australian female Library voices based on your brand tone. (elevenlabs.io
ok, can you give me a high level programming insutrction for my vide coding agent to understand how to code my website to connect to an elevenlab agent for voice conversation? It seems like it's struggling between websocket and webrtc. I am using next.js and seems like it's using @unreal valley/react sdk
Short answer: in a browser/Next.js app, use WebRTC via the @elevenlabs/react SDK. Only use WebSockets if you must run without WebRTC. WebRTC gives better audio quality and stability, and in the SDK it’s just a single option.
High‑level integration (Next.js + @elevenlabs/react)
- Install the SDK
- npm install @elevenlabs/react. (elevenlabs.io)
- Build a client component for the call
- Use the useConversation hook in a "use client" component, request mic access, then startSession with connectionType: "webrtc" and your agentId. WebRTC in this SDK is PCM 48 kHz under the hood; you don’t need to manage audio formats. (elevenlabs.io)
- Example patterns for a Next.js client component using useConversation are shown in the ElevenLabs “Santa agent” React guide. (elevenlabs.io)
- Prefer WebRTC over WebSockets
- ElevenLabs recommends WebRTC for browser conversations; switching from "websocket" to "webrtc" is a single parameter change in startSession. (elevenlabs.io)
- Handle private vs public agents securely
- Public agents: startSession({ agentId, connectionType: "webrtc" }).
- Private agents: create a Next.js API route that calls GET /v1/convai/conversation/token on your server with your xi-api-key, then pass the returned conversationToken to startSession from the client. Do not expose xi-api-key in the browser. (elevenlabs.io)
- Optional: drop in a prebuilt UI
- The ConversationBar component provides a ready voice+text UI and uses @elevenlabs/react under the hood (WebRTC). (ui.elevenlabs.io)
- Text chat in the same session
- Call sendUserMessage("...") to send typed messages; use sendContextualUpdate("...") for non-turn context. (elevenlabs.io)
- WebSocket fallback (only if needed)
- If you must use WebSockets instead of WebRTC, connect to wss://api.elevenlabs.io/v1/convai/conversation with an agentId (public) or a server‑generated signed URL (private). (elevenlabs.io)
Why your agent was “struggling between WebSocket and WebRTC”
- In the React SDK, you choose explicitly: startSession({ connectionType: "webrtc" | "websocket", ... }). For web voice chat, set "webrtc" to avoid audio/glitch issues; this is the recommended path for browsers. (elevenlabs.io)
Pointers you can give your coding agent
- Use @elevenlabs/react useConversation.
- In client code: request mic permission, then startSession({ agentId, connectionType: "webrtc" }).
- For private agents: fetch a conversation token from your Next.js API route that calls /v1/convai/conversation/token with xi-api-key; pass conversationToken to startSession. (elevenlabs.io)
- Optional UI: use ConversationBar for a turnkey widget. (ui.elevenlabs.io)
References
- React SDK: useConversation API, startSession("webrtc"), security notes, text APIs. (elevenlabs.io)
- WebRTC recommendation and one‑line switch from websocket → webrtc. (elevenlabs.io)
- Next.js/React example (“Santa agent”) showing client usage.
the WebRTC connection to LiveKit (ElevenLabs' real-time audio infrastructure) is failing with a 404. This means your agent is not configured for browser-based WebRTC connections.
The Issue
Your agent agent_3701kh0825knfe2vdxdj49ssk87s appears to be configured for phone calls only (via Twilio), not for direct browser-to-agent WebRTC connections.
How to Fix (in ElevenLabs Dashboard)
Go to your ElevenLabs dashboard → Conversational AI → your agent
Look for "Widget" or "Web Integration" settings
Make sure browser/web widget access is enabled
Check that the agent has public access or generate a proper signed URL for private access
Alternative: Use a Signed URL
If your agent is private, you may need to generate a signed URL from your backend. The ElevenLabs SDK supports this:
await conversation.startSession({
signedUrl: "your-signed-url-from-backend", // Instead of agentId
});
Can you check your agent's configuration in the ElevenLabs dashboard and confirm whether it's set up for web/browser access?
The above is the response from my vibe coding agent when i tested the implementation from my website. I can heard a very brief "hi" from the agent and it got disconnected straight away, do you think his diagnosis is accurate? what advise you can give it to him?
Short answer: the “phone-only” diagnosis is likely incomplete. The most common causes of a brief greeting then drop in the browser are (a) using WebSocket signed URLs instead of a WebRTC token, or (b) trying to use the embed widget with a private agent. Both are configuration/auth issues, not that the agent is “telephony-only.”
What to check and fix
-
If you’re using the React SDK in the browser, start a WebRTC session explicitly and authenticate correctly:
- Public agent: startSession({ agentId, connectionType: 'webrtc' }). (elevenlabs.io)
- Private agent: first create a WebRTC conversation token via GET /v1/convai/conversation/token on your backend, then startSession({ conversationToken, connectionType: 'webrtc' }) from the client. Do not use a signedUrl here—signed URLs are for WebSocket connections. (elevenlabs.io)
-
If you embedded the web widget: it currently requires a public agent with authentication disabled in the agent’s Advanced tab. Also ensure your site’s domain is allowlisted in the agent’s Security tab. (elevenlabs.io)
-
If you previously configured Twilio: keep ÎĽ-law 8000 Hz for telephony (input and TTS output) as documented, but note that WebRTC in the browser uses a 48 kHz PCM path in the client SDK. These settings target different transports and do not by themselves block WebRTC browser usage. (elevenlabs.io)
Why the brief “hi” then disconnect happens
- With a private agent, the browser can connect briefly but will drop if you pass a WebSocket signedUrl while attempting a WebRTC session, or if no valid WebRTC conversation token is provided. Use a WebRTC token for connectionType: 'webrtc'. (elevenlabs.io)
- With the widget, if the agent isn’t public (auth disabled), the connection will fail; make it public or switch to the SDK flow with tokens. (elevenlabs.io)
Concrete guidance for your coding agent
- Use the React SDK and WebRTC:
- Server: call GET /v1/convai/conversation/token with your agent_id and return the token to the client. (elevenlabs.io)
- Client (Next.js/React): await conversation.startSession({ conversationToken, connectionType: 'webrtc' }). (elevenlabs.io)
- If you prefer the widget, make the agent public and add the embed snippet with agent-id; ensure your domain is in the allowlist. (elevenlabs.io)
Note on “LiveKit 404”
- ElevenLabs’ docs don’t mention LiveKit endpoints. The documented, supported browser path is WebRTC via the SDK
how can i tell if an agent is set to be private or public?
Short answer: “Public” means authentication is disabled for the agent; “Private” means authentication is enabled.
How to check:
- In the dashboard: open your agent → Advanced (or Security) → Authentication. If authentication is disabled, the agent is public; if enabled, it’s private. The widget docs explicitly note that widgets require public agents with authentication disabled in the Advanced tab. (elevenlabs.io)
- From SDK behavior: if you can start a session with only agentId (no signedUrl/conversation token), it’s a public agent; if your agent requires a signed URL or token, it’s private. (elevenlabs.io)
- From raw WebSocket: public agents accept wss://…/conversation?agent_id=YOUR_ID; private agents require a server-generated signed URL. (elevenlabs.io)
Note: Even public agents can be restricted by an Allowlist (Security → Allowlist) to only permit specific domains. If that’s set and the origin isn’t allowed, connections will fail. (elevenlabs.io)
Sources:
Widget customization | ElevenLabs Documentation
React SDK | ElevenLabs Documentation
WebSocket | ElevenLabs Documentation
Widget customization | ElevenLabs Documentation
my agent is public but my app is encountering the prompt disconnection issue after the connection, what will be the most likely issue?
Most likely cause: if you’re connecting via the raw WebSocket API, your client isn’t replying to the server’s ping events, so the server closes the socket shortly after the initial greeting. The official WS guide explicitly shows you must handle ping→pong “to keep connection alive.” Fix by either:
- Using the SDK with WebRTC (recommended): startSession({ agentId, connectionType: 'webrtc' }). (elevenlabs.io)
- Or, if you must use WebSocket, implement ping/pong exactly as in the docs and begin streaming user audio immediately after onopen. (elevenlabs.io)
Also verify these common misconfigurations that cause immediate disconnects:
- Widget auth: the embed widget requires a public agent with authentication disabled; if auth is on or an allowlist blocks your domain, the connection will be rejected. Either disable auth for the widget or add your site to the allowlist. (elevenlabs.io)
- Connection mode: if you intend to use WebRTC but start a WebSocket session (or vice‑versa), the client may connect and then drop. Ensure connectionType is set correctly in the React SDK. For private/authenticated flows, obtain a conversation token first. (elevenlabs.io)
- Aggressive flow controls: an overly short turn timeout or an automatic End Call tool can end the session right after the first line. Review Turn Timeout and any End Call tool prompts. (elevenlabs.io)
- Region mismatch: if you force a serverLocation that doesn’t match your deployment, reconnects/disconnects can occur. Use the correct serverLocation or omit it to use the default. (elevenlabs.io)
If you share whether you’re using the raw WebSocket, the React SDK, or the embed widget, I can point you to the exact snippet to apply. (elevenlabs.io)
Sources:
WebSocket | ElevenLabs Documentation
WebSocket | ElevenLabs Documentation
Widget customization | ElevenLabs Documentation
React SDK | ElevenLabs Documentation
Conversation flow | ElevenLabs Documentation
React SDK | ElevenLabs Documentation
WebSocket | ElevenLabs Documentation
SignalClient.ts:380 websocket closed
{room: 'room_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6501kh0dxq9xfh49g0qv2yfk1cqn', roomID: 'RM_fKrJwvjZWxxi', participant: 'user_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6501kh0dxq9xfh49g0qv2yfk1cqn', pID: 'PA_LPCdT4Fi82Q6', reason: '', …}
code
:
1006
pID
:
"PA_LPCdT4Fi82Q6"
participant
:
"user_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6501kh0dxq9xfh49g0qv2yfk1cqn"
reason
:
""
room
:
"room_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6501kh0dxq9xfh49g0qv2yfk1cqn"
roomID
:
"RM_fKrJwvjZWxxi"
state
:
3
wasClean
:
false
[[Prototype]]
:
Object
constructor
:
Ć’ Object()
hasOwnProperty
:
Ć’ hasOwnProperty()
isPrototypeOf
:
Ć’ isPrototypeOf()
propertyIsEnumerable
:
Ć’ propertyIsEnumerable()
toLocaleString
:
Ć’ toLocaleString()
toString
:
Ć’ toString()
valueOf
:
Ć’ valueOf()
defineGetter
:
Ć’ defineGetter()
defineSetter
:
Ć’ defineSetter()
lookupGetter
:
Ć’ lookupGetter()
lookupSetter
:
Ć’ lookupSetter()
proto
:
(...)
get proto
:
Ć’ proto()
set proto
:
Ć’ proto()
this is the browser console log when elevenlabs close our connection. Is there any hint from there of what is the reason that cause the closing?
Short answer: No. Close code 1006 with an empty reason does not tell you why the server closed the connection. To find the cause, use the platform’s diagnostics and verify common configuration pitfalls documented by ElevenLabs.
What to check (in order)
- Call History: Look up the failed conversation for a human‑readable failure reason (e.g., allowlist blocked host, missing required dynamic variables, daily call limit, override not allowed). This is the primary place ElevenLabs surfaces early‑disconnect causes. (help.elevenlabs.io)
- Request Log: In Developers → Request Log, filter around the failure time to see 4xx/5xx API errors that correlate with the drop. (elevenlabs.io)
- Agent access model:
- Public agents: you may connect from the client with just agentId. (elevenlabs.io)
- Private/authorized agents: for WebRTC you must obtain a conversationToken on your backend and pass it to the client; for WebSocket you must use a signed URL. If you skip these, the session can be created and then closed immediately. (elevenlabs.io)
- Domain allowlist: If enabled, your site’s host must be on the allowlist; otherwise connections are rejected. The help center shows “Host … is not allowed to connect to this agent” as a common failure. (elevenlabs.io)
- Widget usage: If you’re embedding the widget, it requires a public agent with authentication disabled; using a private agent here will fail. (elevenlabs.io)
- “End Call” tool: If configured aggressively (via prompts or conditions), the system End Call tool can terminate right after the greeting. Verify its settings in Tools. (elevenlabs.io)
Helpful implementation notes
- For React SDK, a correct public WebRTC start looks like:
conversation.startSession({ agentId: '<your-agent-id>', connectionType: 'webrtc' })
For private agents, fetch a conversationToken (WebRTC) or signedUrl (WebSocket) from your backend and pass that instead of agentId. (elevenlabs.io) - If you’re diagnosing by IDs, capture conversation_id from the conversation_initiation_metadata event and use it to find the call in history/logs. (elevenlabs.io)
- If you are using the widget, ensure your domain is allowlisted when using the Security → Allowlist feature. (elevenlabs.io)
- Check ElevenLabs Status if you suspect a platform incident affecting WebRTC. (status.elevenlabs.io)
Bottom line
The console snippet alone doesn’t reveal the cause. Check Call History and Request Log for the explicit error, then verify agent access (public vs private + token/signed URL), domain allowlist, widget/public requirement
{room: 'room_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6401kh0fkbwcfghbb7zs55ebebft', roomID: 'RM_5VVSsVeKD8Bp', participant: 'user_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6401kh0fkbwcfghbb7zs55ebebft', pID: 'PA_JURTpyjSxDrL'}
pID
:
"PA_JURTpyjSxDrL"
participant
:
"user_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6401kh0fkbwcfghbb7zs55ebebft"
room
:
"room_agent_3701kh0825knfe2vdxdj49ssk87s_conv_6401kh0fkbwcfghbb7zs55ebebft"
roomID
:
"RM_5VVSsVeKD8Bp"
[[Prototype]]
:
Object
constructor
:
Ć’ Object()
hasOwnProperty
:
Ć’ hasOwnProperty()
isPrototypeOf
:
Ć’ isPrototypeOf()
propertyIsEnumerable
:
Ć’ propertyIsEnumerable()
toLocaleString
:
Ć’ toLocaleString()
toString
:
Ć’ toString()
valueOf
:
Ć’ valueOf()
defineGetter
:
Ć’ defineGetter()
defineSetter
:
Ć’ defineSetter()
lookupGetter
:
Ć’ lookupGetter()
lookupSetter
:
Ć’ lookupSetter()
proto
:
(...)
get proto
:
Ć’ proto()
set proto
:
Ć’ proto()
can we see the disconnection reason from the disconnect from room browser log?
I found those call i made are all succeed but only for duration 0:01, all request are 200 in the request log