Sure, I can guide you through the process of using the ElevenLabs streaming endpoint to stream text-to-speech audio in TypeScript. Here's a step-by-step guide:
Step 1: Create accounts with Twilio and ngrok
You'll need to create accounts with Twilio and ngrok for this guide. You can create these accounts at the following links:
Step 2: Get the code
You can get the entire code for this guide on GitHub.
Step 3: Create the server with Express
You'll need to initialize your project, install dependencies, install dev dependencies, and create your files. Here's how you can do it:
Initialize your project:
mkdir elevenlabs-twilio
cd elevenlabs-twilio
npm init -y
Install dependencies:
npm install elevenlabs express express-ws twilio
Install dev dependencies:
npm i @types/node @types/express @types/express-ws @types/ws dotenv tsx typescript
Create your files:
import 'dotenv/config';
import express, { Response } from 'express';
import ExpressWs from 'express-ws';
import VoiceResponse from 'twilio/lib/twiml/VoiceResponse';
import { ElevenLabsClient } from 'elevenlabs';
import { type WebSocket } from 'ws';
import { type Readable } from 'stream';
const app = ExpressWs(express()).app;
const PORT: number = parseInt(process.env.PORT || '5000');
const elevenlabs = new ElevenLabsClient();
const voiceId = '21m00Tcm4TlvDq8ikWAM';
const outputFormat = 'ulaw_8000';
const text = 'This is a test. You can now hang up. Thank you.';
function startApp() {
app.post('/call/incoming', (_, res: Response) => {
const twiml = new VoiceResponse();
twiml.connect().stream({
url: `wss://${process.env.SERVER_DOMAIN}/call/connection`,
});
res.writeHead(200, { 'Content-Type': 'text/xml' });
res.end(twiml.toString());
});
app.ws('/call/connection', (ws: WebSocket) => {
ws.on('message', async (data: string) => {
const message: {
event: string;
start?: { streamSid: string; callSid: string };
} = JSON.parse(data);
if (message.event === 'start' && message.start) {
const streamSid = message.start.streamSid;
const response = await elevenlabs.textToSpeech.convert(voiceId, {
model_id: 'eleven_turbo_v2_5',
output_format: outputFormat,
text,
});
const audioArrayBuffer = await streamToArrayBuffer(response);
ws.send(
JSON.stringify({
streamSid,
event: 'media',
media: {
payload: Buffer.from(audioArrayBuffer as any).toString('base64'),
},
}),
);
}
});
ws.on('error', console.error);
});
app.listen(PORT, () => {
console.log(`Local: http://localhost:${PORT}`);
console.log(`Remote: https://${process.env.SERVER_DOMAIN}`);
});
}
function streamToArrayBuffer(readableStream: Readable) {
return new Promise((resolve, reject) => {
const chunks: Buffer[] = [];
readableStream.on('data', (chunk) => {
chunks.push(chunk);
});
readableStream.on('end', () => {
resolve(Buffer.concat(chunks).buffer);
});
readableStream.on('error', reject);
});
}
startApp();
Step 4: Update your environment variables
Update the .env file with your ngrok domain and ElevenLabs API key.
SERVER_DOMAIN=*******.ngrok.app
ELEVENLABS_API_KEY=*************************
Step 5: Start the application
Run the following command to start the app:
npm run dev
Step 6: Set up Twilio
Follow Twilio’s guides to create a new number. Once you’ve created your number, navigate to the “Configure” tab in Phone Numbers -> Manage -> Active numbers. In the “A call comes in” section, enter the full URL to your application (make sure to add the/call/incoming path).
Step 7: Make a phone call
Make a call to your
glad stratus
gray cradleBOT