streaming example in safari

[lethal wolf]

I am building an app in next js that uses realtime transciption, with the ability to start, pause, contine, and stop a mic. This works fine, but in safari/mobile (using safari) the connection keeps closing. Related: https://github.com/orgs/deepgram/discussions/557

[bright nacelle]

Thanks for asking your question. Please be sure to reply with as much detail as possible so we can assist you efficiently. Such as:

• Provide the request_id if you've a question about a transcription response.
• The options you used or the api.deepgram.com URL you sent your request to, including parameters.
• Any code snippets you can include.
• Any audio you can include, or if you can't share it here please email it to us at [email protected] and provide a link to this thread.

[vital girder]

can you share some of your code vdz? or is it an unedited version of the repo?

[lethal wolf]

Yes of course, can I share in a DM?

[vital girder]

I'd rather a code snippet here, or a link to some code

[lethal wolf]

const toggleMicrophone = useCallback(async () => {
if (isUploading) return
if (microphone && userMedia) {
if (microphone.state === "paused") {
microphone.resume()
} else {
await new Promise(resolve => setTimeout(resolve, 1000));
console.log(isUploading, isLoading, isRecording, caption.length)
microphone.pause();
}
} else {
const userMedia = await navigator.mediaDevices.getUserMedia({
audio: true,
});

        const microphone = new MediaRecorder(userMedia);
        microphone.start(500);

        microphone.onstart = () => { setMicOpen(true); };

        microphone.onpause = () => { setMicOpen(false) }

        microphone.onstop = () => {
            setMicOpen(false);
            userMedia.getTracks().forEach(track => track.stop());
        };

        microphone.ondataavailable = (e) => {
            add(e.data);
        };

        setUserMedia(userMedia);
        setMicrophone(microphone);
    }
    setIsRecording((prv) => !prv)
}, [add, microphone, userMedia, isUploading]);

useEffect(() => {
if (apiKey && "key" in apiKey) {
let keepAlive;

        console.log("connecting to deepgram");
        const deepgram = createClient(apiKey?.key ?? "");
        const connection = deepgram.listen.live({
            model: "nova-2",
            language: "en-US",
            interim_results: true,
            smart_format: true,
            utterance_end_ms: 1500,
        });

        if (keepAlive) clearInterval(keepAlive);
        keepAlive = setInterval(() => {
            console.log("KeepAlive sent.");
            connection.keepAlive();
        }, 3000); // Sending KeepAlive messages every 3 seconds

        connection.on(LiveTranscriptionEvents.Open, () => {
            console.log("connection established");
            setListening(true);
        });

        connection.on(LiveTranscriptionEvents.Close, () => {
            console.log("connection closed");
            connection.finish();
            setListening(false);
            setApiKey(null);
            setConnection(null);
            clearInterval(keepAlive);
        });

        connection.on(LiveTranscriptionEvents.Transcript, (data) => {
            console.log(data.is_final, data.speech_final);
            if (data.is_final) {
                const newCaption = data.channel.alternatives[0].words.map(word => word.punctuated_word ?? word.word).join(" ");
                if (newCaption) {
                    setCaption(prevCaption => `${prevCaption} ${newCaption}`.trim());
                    setPendingTranscription(false);
                }
            } else {
                setPendingTranscription(true);
            }
        });


        setConnection(connection);
        setLoading(false);
    }
}, [apiKey]);

useEffect(() => {
const processQueue = async () => {
if (size > 0 && !isProcessing) {
setProcessing(true);

            if (isListening) {
                const blob = first;
                connection?.send(blob);
                remove();
            }

            const waiting = setTimeout(() => {
                clearTimeout(waiting);
                setProcessing(false);
            }, 250);
        }
    };

    processQueue();
}, [connection, queue, remove, first, size, isProcessing, isListening]);

[lethal wolf]

@vital girder any updates here?

[cunning pike]

Having this same issue @vital girder - using the example repo with small adjustments to the App.tsx https://github.com/deepgram-starters/nextjs-live-transcription/

I get two objects like this via the webhook and then it dies, works perfectly fine on browser:

sometimes it picks up the first few words I speak but then it dies instantly after, very strange

logging the connectionState shows this, 3 being closed, so it is randomly closing

https://github.com/orgs/deepgram/discussions/557 related discussion

The connection is being closed via the conn listener in DeepgramProvider.tsx:

  /**
   * Connects to the Deepgram speech recognition service and sets up a live transcription session.
   *
   * @param options - The configuration options for the live transcription session.
   * @param endpoint - The optional endpoint URL for the Deepgram service.
   * @returns A Promise that resolves when the connection is established.
   */
  const connectToDeepgram = async (options: LiveSchema, endpoint?: string) => {
    // const key = await getApiKey()
    const keyData = await fetchKey()
    if (!keyData) return
    console.log(keyData.key)
    const deepgram = createClient(keyData.key)

    const conn = deepgram.listen.live(options, endpoint)

    conn.addListener(LiveTranscriptionEvents.Open, () => {
      setConnectionState(LiveConnectionState.OPEN)
    })

    conn.addListener(LiveTranscriptionEvents.Close, () => {
      console.log("Connection closed via addListener")
      setConnectionState(LiveConnectionState.CLOSED)
    })

    setConnection(conn)
  }