Service works first time, then times out on subsequent runs, fails for a minute, then works again.

[true bane]

I have a serverless streaming service I've written that is invoking an LLM service running on my computer. I'm using a subdomain redirect -> nginx -> ollama service to run this. It will work the first time, but then if I run the service again right after, it always times out. Then works again after a minute. Rinse/repeat.

It works 100% of the time on my local machine. The service is fast and returns responses quickly. How can I diagnose the error and fix? Really could use some help figuring this out, as the Vercel logs only say "[POST] /api/generate reason=EDGE_FUNCTION_INVOCATION_TIMEOUT, status=504, user_error=true"

[torpid locust]

Post created!

🔎 This post has been indexed in our web forum and will be seen by search engines so other users can find it outside Discord

🕵️ Your user profile is private by default and won't be visible to users outside Discord, if you want to be visible in the web forum you can add the "Public Profile" role in id:customize

✅ You can mark a message as the answer for your post with Right click -> Apps -> Mark Solution
(if you don't see the option, try refreshing Discord with Ctrl + R)

[true bane]

Service works first time, then times out on subsequent runs, fails for a minute, then works again.

[twilit citrus]

What database are you using?

[true bane]

I'm using Firebase

[twilit citrus]

and can you send contents of /api/generate/page.tsx

and if there is any line number given, specially look around that line

[true bane]

I'll just put all the code here

ollama.ts (part 1)

import { readStream } from "@/app/utilities/readStream";

export async function stream(model: string, prompt: string, system: string): Promise<ReadableStream<any>> {
  return await buildStream(model, prompt, system);
}

function parseSequentialJSON(jsonString: string): { response: string }[] {
  // Add a comma between the }{ to transform it into a valid JSON array
  const validJsonString = `[${jsonString.replace(/}\s*{/g, '},{')}]`;

  try {
    // Parse the valid JSON string
    const jsonArray = JSON.parse(validJsonString);
    return jsonArray;
  } catch (error) {
    throw new Error("Failed to parse.")
  }
}

part 2:

async function buildStream(model: string, prompt: string, system: string): Promise<ReadableStream<any>> {
  const encoder = new TextEncoder();
  return new ReadableStream({
    async start(controller) {
      try {
        const { OLLAMA_HOST } = process.env;
        console.log(OLLAMA_HOST);
        const endpoint = '/api/generate'; // API endpoint

        const url = new URL(endpoint, OLLAMA_HOST).toString();

        console.log(url);

        const requestBody = {
          prompt,
          system,
          model,
          stream: true,
        };

        const response = await fetch(url, {
          method: 'POST',
          body: JSON.stringify(requestBody),
          headers: {
            'Content-Type': 'application/json'
          }
        });

        console.log(`Response ok: ${response.ok}`);

        if (!response.ok) {
          throw new Error(`HTTP error! Status: ${response.status}`);
        }

        console.log("Response is OK");

        // Handle the stream
        const reader = response.body;
        if (reader == null) {
          throw new Error("Reader is null.");
        }
        let buffer = '';
        await readStream(
          reader,
          value => {
            try {
              buffer += value;
              const next = parseSequentialJSON(buffer).map(element => element.response).join("");
              controller.enqueue(encoder.encode(next));
              buffer = '';
            } catch { }
          }
        );

        controller.close();
      } catch (e) {
        console.log('An error occurred while communicating with ollama.');
        throw e;
      }
    },
  });
}

That's where /api/generate resolves in my applicatin. Then it sends a requests to ollama's /api/generate, which does the LLM completion

I don't think this is particularly easy to diagnose either. It runs quite fast (ollama startup is less than a second).

It does work sometimes. But about 20% of the time.

[twilit citrus]

try building and starting the app locally

[true bane]

Done. I tested it locally and it works no problem, although that may not mean much since the local execution does not have a timeout like production

[twilit citrus]

you'll have to add console.log breakpoints, and see in logs where exactly the error occurs

[true bane]

Fair, I'll add more logs. Something weird is happening at some point and I need to diagnose it further.

Notably, it's not getting to the point where it hits my ollama service as evidenced by the lack of log outputs when running sudo tcpdump -i any port 11434. Thanks for the suggestion... hopefully the logs give more insight

Interesting. I think the previous stream is not closing properly, despite controller.close() being invoked. I'll try to confirm that. Could that affect the subsequent call to that service? Perhaps, but I'm not sure.

OK this is interesting.

The [POST] /api/generate status=200 log isn't generated after the controller is closed. That log is generated if I invoke the service again after waiting about 20+ seconds, and then it works again. But if I invoke the service before those 20 or so seconds, the next call is a timeout.

How do I ensure the controller is closed & the request is completed other than calling controller.close()?

I think I'm mistaken actually. That 200 is shown FIRST. My apologies for misunderstanding.

I'll work on this tomorrow. Really need to figure this out. I don't think I'm far from the solution...

[true bane]

Still having this issue unfortunately. It looks like the request goes through and closes successfully. Yet subsequent invocations that happen in under 20-30 seconds will always fail. They always succeed if the wait time is after 20-30 seconds. I know this is a weird and difficult to fix bug, but the logs look good on Vercel now, and the network tab doesn't show anything relevant to the error.

I need some ideas on how to diagnose this issue further.

[true bane]

It still fails sometimes but its working more or less better than before.