#Streaming socket close & concurrent sockets

1 messages · Page 1 of 1 (latest)

vital nest
#

Hi, just getting started with this. We're building a voice call feature with an AI, and as part of it, we send audio from client side, then there's a pretty significant pause in the middle. During this pause of AI inference we use keepalive and it keeps the socket open. Afterwards, we resume sending audio when it's the user turn, but I'm finding that upon sending 4 buffers (and exactly 4) to the socket after this pause, the socket would close (gracefully) on me for no apparent reason.

Why does this happen? Does it require a continuous stream, it doesn't seem so because if it did there wouldn't need to be the keepalive functionality. I wonder if it's something wrong with its timestamping, or if it's having trouble making sense of the silence or something like that? I'm trying to check if it's a frontend error but highly doubt it, and we never call close connection in the backend during this whole flow.

Also, for concurent requests - does this mean 100 concurrent streaming socket connections as well?

Full event attached.

📎 message.txt
last nexus
#

Yes for concurrency! That is exactly why we opt for that type of limit, because we handle more than just requests-per-minute traffic. I bet you'll get away with more than 100 before it tells you off though (although, please don't load test our live API)

My guess is that one of those byte arrays has a length of 0 bytes. A legacy feature of our API is to close the stream on a zero-byte request, and sometimes it can be sent accidentally.

Our next SDK versions (still in development) will gracefully prevent a zero-byte request, but users directly integrating with our API could fall victim to this until we do a major version increase on the API (not something planned for the near future).

For demos I have built, I have built in a synchronous function to force a reconnection before forwarding the next blob of data. I would strongly recommend always handling a reconnection until you explicitly close it.

vital nest
#

Hmm, I do check for 0 bytes like:

socket.on("audio_data", (data: ArrayBuffer) => {
if (transcriptState.processUserInput) {
audioBuffer.push(Buffer.from(data));
if (deepgramLive.getReadyState() == 1) {
const concatenatedBuffer = Buffer.concat(audioBuffer);
if (concatenatedBuffer.length > 0) {
deepgramLive.send(concatenatedBuffer);
clearInterval(keepAliveInterval);
keepAliveInterval = setInterval(sendKeepAlive, 7000);
audioBuffer = [];
}
}
}
});

We will handle the reconnection but it may result in some lost interim data (perhaps), and I feel like with the websockets we should be able to hold the long connection.

last nexus
vital nest
#

Hmm, I'm confused then, it seems like the checks are quite similar, and we never would get to sending a 0 byte packet, not sure why it's closing then, will have to look more closely

vital nest
#

Nevermind, I traced to the root. Deepgram requires all packets to be sent and there seems to be very little tolerance to anything in the middle missing if only 1 or 2, and it closes connection gracefully if that happens. We had a server flag that said "we should stop sending audio to Deepgram as it's the AI's turn and any straggling audio bits doesn't matter/shouldn't be processed" and it made a few packets processed by the microphone not reach Deepgram, and on resume Deepgram would force close the connection due to that. Removing the server flag made it work.

Wasn't obvious, and wondering if this is intentional? I think the part that threw me off was the graceful close. If it didn't tolerate any dropped or inconsistent middle buffers, maybe it should give an error instead?

last nexus
#

Wow that's really interesting. Let me ask the team on this one!

vital nest
#

Also - friended and dm'd you on a separate issue, just got off a call w @azure comet on concurrency. Thanks!