Frequent 524 errors with anthropic claude 3models | OpenRouter | Page 1

night pecan Apr 15, 2024, 11:44 PM

#

I'm seeing pretty frequent 524 errors when using anthropic/claude-3-haiku:beta and anthropic/claude-3-opus:beta with openrouter. This happens pretty consistently with ~10% of our API calls.

I'm testing our integration as we prep to ship and wasn't seeing these errors a few weeks ago when I initially implemented the openrouter integration.

The error responses contain a pretty generic HTML Cloudflare error page (attached).

Unfortunately, this is a dealbreaker for our usage of openrouter if we can't figure out what's going on soon, so any help would be greatly appreciated.

📎 message.txt

#

All of the errors (and successful api calls) are coming with the following unique headers:

  headers: {
    // Optional, for including your app on openrouter.ai rankings
    'HTTP-Referer': 'https://gptlint.dev',
    // Optional, shows in rankings on openrouter.ai
    'X-Title': 'gptlint'
  }

jade ridge Apr 16, 2024, 12:18 AM

#

ooh love this idea!

#

looking

#

oof, we don't have any of these in our logs

#

checking cloudflare directly

#

not in cloudflare either. @night pecan are you sure that's your referer? maybe the site is overriding it?

night pecan Apr 16, 2024, 12:32 AM

#

jade ridge not in cloudflare either. <@715287837200875543> are you sure that's your referer...

let me double check it; this is a CLI tool and I'm invoking the HTTP requests directly using ky so unless I'm messing something up locally, it should have these headers. double checking now

#

@jade ridge this is what ky is printing out for the headers as expected:

{
  headers: { 'HTTP-Referer': 'https://gptlint.dev', 'X-Title': 'gptlint' }
}

#

so either these headers aren't being sent via ky correctly or they're not being recorded properly on CF / openrouter's side

#

is there a field on the openrouter's response I can paste from a successful api call that you can use to trace?

#

here's an example successful api call:

{
  "id": "gen-lo5FMWaMrycdn3OTkOtKtgGqoaWR",
  "model": "anthropic/claude-3-haiku:beta",
  "object": "chat.completion",
  "created": 1713228021,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "# EXPLANATION\n\nThe `utils.ts` file does not appear to contain any code related to the \"no-hardcoded-secrets\" rule. The file contains utility functions such as `omit`, `pick`, `logEvalStats`, `createPromiseWithResolvers`, and `resolveGlobFilePatterns`, which do not seem to involve any sensitive secrets or API keys. The file also imports some third-party libraries, but there are no instances of hardcoded secrets or API keys in the code.\n\n# VIOLATIONS\n\nThere are no code snippets in the `utils.ts` file that violate the \"no-hardcoded-secrets\" rule."
      },
      "finish_reason": "end_turn"
    }
  ],
  "usage": {
    "prompt_tokens": 1997,
    "completion_tokens": 155,
    "total_tokens": 2152,
    "total_cost": 0.000693
  }
}

#

it looks like ky is automatically using the lowercase version of these headers since I'm guessing the spec says they should be case-insensitive:

https://openrouter.ai/api/v1/chat/completions {
  accept: 'application/json',
  authorization: '<redacted>',
  'content-type': 'application/json',
  'http-referer': 'https://gptlint.dev',
  'user-agent': 'openai-fetch',
  'x-title': 'gptlint'
}

jade ridge Apr 16, 2024, 12:48 AM

#

think i found it using that user agent, which is unusual

#

cloudflare logs "None" for the referrer, weirdly

#

this is what I see

night pecan Apr 16, 2024, 12:49 AM

#

maybe because http-referer can be spoofed, it disregards non-browser http-referer requests?

jade ridge Apr 16, 2024, 12:49 AM

#

right

#

that's my guess too

night pecan Apr 16, 2024, 12:50 AM

#

hmmm that looks approximately right

jade ridge Apr 16, 2024, 12:50 AM

#

so less than 10% right? looking for correlations on the 524s right now

night pecan Apr 16, 2024, 12:50 AM

#

the 524s started happening yesterday and continued til today and most of the 499s are probably hung 524s

#

most of the 200 OKs were from my previous testing

jade ridge Apr 16, 2024, 12:50 AM

#

ah kk

night pecan Apr 16, 2024, 12:51 AM

#

the 524s hang for so long that they bog everything down so I'd like to at least understand where the problem lies (in my code, in openrouter, in CF, or in anthropic)

#

before launching with the recommendation to use openrouter in https://gptlint.dev/guide/llm-providers

GPTLint

LLM Providers - GPTLint

Use LLMs to enforce best practices across your codebase.

#

(hoping to open source this project later this week)

#

I'm currently trying to run evals for launch across diff LLMs but I keep having to ctrl+C the anthropic/openrouter ones when they hit their first 524

jade ridge Apr 16, 2024, 1:00 AM

#

I'm guessing you're not streaming your request, right?

#

how large is the prompt?

#

possible that it's over 4mb?

#

(also, can you stream it?)

night pecan Apr 16, 2024, 1:30 AM

#

no, not streaming; using it programmatically where streaming is actually slower.

prompts are generally ~2k-3k tokens

#

it looks like the largest JSON body is ~40kb which corresponds to ~10k input tokens

#

and is much smaller than 4mb

#

I could be wrong about the JSON body size; this is how I'm computing it: console.log(Buffer.from(JSON.stringify(messages)).byteLength)

#

max response tokens is ~500 so the responses are pretty small

#

btw I'm no longer seeing 524 errors and didn't change anything on my side. did you change something that might've affected this?

will def let you know if I start seeing them again. either way, I'd love to understand what's going on so I can decide on what advice I want to give people for the OSS launch.

jade ridge Apr 16, 2024, 2:01 AM

#

great! as far as I can tell, this was a blip in our cloud provider... let me know if you notice it again, especially if you can reproduce it

night pecan Apr 16, 2024, 2:02 AM

#

thanks, alex 💪

#

btw any advice for fixing the http-referer issue w/ CF stripping it (or a workaround like supporting user-agent)? there's no way I'm the first customer to have this happen to them using openrouter outside of the browser

jade ridge Apr 16, 2024, 2:05 AM

#

night pecan btw any advice for fixing the `http-referer` issue w/ CF stripping it (or a work...

I'm pretty sure OpenRouter rankings pages will still pick it up. this is a known issue that others have had too, but afaik it only affects our ability to log your requests in cloudflare and datadog

night pecan Apr 16, 2024, 2:05 AM

#

ok; perfect

#

thanks again

night pecan Apr 16, 2024, 5:30 AM

#

hmmm, now I'm seeing occasional 520 errors with the same setup: openrouter.ai | 520: Web server is returning an unknown error

night pecan Apr 16, 2024, 5:47 AM

#

is there anywhere I can view statistics for openrouter overall and/or specific models to understand what their expected error rate is?

I'm used to testing against the openai api where I almost never encounter 5XX errors

jade ridge Apr 16, 2024, 4:21 PM

#

About 7 hours ago, high latencies started with a few models. this might be related. we're still investigating. see #announcements

jade ridge Apr 18, 2024, 10:48 PM

#

@night pecan we think we may have gotten to the bottom of the high latencies this week. let me know if you notice any issues!

#Frequent 524 errors with anthropic claude 3models