Aide(2B tokens/day): consistent 500s with `Provider returned error`

[astral drum]

Hey all!

One of the creators of Aide here, we love OpenRouter and want to keep using it for our LLM Provider needs. The 500s are degrading the performance of our agents as we are now scaling quite a bit and onboarding new users.

Is there anything we can do from our side to help with this?

[dusty isle]

cc @midnight nest

[dusty isle]

Thanks for flagging - any more detail you can provide?

[astral drum]

the only error I see is: provider returned error

[dusty isle]

copy, thanks

Are there specific models you are using more than others? or specific providers you are pinning / routing to by default? or is this a generic degradation in service across the board?

[midnight nest]

Hey there -- we do return the full error in the metadata of the error response. Would it be possible for you to share those with us? Also which model and provider as Toven mentioned above

[astral drum]

model: anthropic/claude-3.5-sonnet:beta

provider is left unconfigured cause we are not hardcoding anything

I also got a DM from alex who recommended adding a fallback for sonnet

[midnight nest]

Ah

unfortunately the sonnet:beta model has only 1 provider

[astral drum]

^^^ let me look into the metadata part .. cause its happening after we start polling from the stream response

[midnight nest]

so there's no other provider for us to fallback to

[astral drum]

    // now create the request for the open router one
    const streamResponse = await openRouterClient.chat.completions.create({
      model: finalModel,
      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      body: {
        'models': [finalModel, 'anthropic/claude-3.5-sonnet'],
      },
    });

does this look like a good enough request to use for the fallback

I am using the OpenAI client

[midnight nest]

that should work -- tho I'd remove the model parameter in the first block

[astral drum]

thats part of the api from openai

so the types will complain

[midnight nest]

    const streamResponse = await openRouterClient.chat.completions.create({

      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      body: {
        'models': [finalModel, 'anthropic/claude-3.5-sonnet'],
      },
    });

Yeah but there's a ts-ignore above it so I thought it would be fine

It already complained right :p

or would it throw at runtime completely???

If it does..........

[astral drum]

hmmm it does not complain .....

what even is ts but just type sugar on js lol

[midnight nest]

you got that right

[dusty isle]

[midnight nest]

But yeah anthropic upstream is still showing big red bar

[astral drum]

testing it out locally will report back

$60B in funding aint worth shit

doesnt work

[
  { role: 'system', content: [ [Object] ] },
  { role: 'user', content: [ [Object] ] },
  { role: 'user', content: [ [Object] ] }
]
Error in Claude API: BadRequestError: 400 Input required: specify "prompt" or "messages"
    at APIError.generate (/Users/skcd/scratch/anton/codestory_subscription/node_modules/openai/error.js:45:20)
    at OpenAI.makeStatusError (/Users/skcd/scratch/anton/codestory_subscription/node_modules/openai/core.js:293:33)
    at OpenAI.makeRequest (/Users/skcd/scratch/anton/codestory_subscription/node_modules/openai/core.js:337:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/skcd/scratch/anton/codestory_subscription/dist/index.js:335:32 {
  status: 400,
  headers: {
    'access-control-allow-origin': '*',
    'cf-ray': '9021951e3e2dd1f7-LHR',
    connection: 'keep-alive',
    'content-length': '130',
    'content-type': 'application/json; charset=UTF-8',
    date: 'Tue, 14 Jan 2025 23:59:54 GMT',
    server: 'cloudflare',
    vary: 'Accept-Encoding',
    'x-clerk-auth-message': 'Invalid JWT form. A JWT consists of three parts separated by dots. (reason=token-invalid, token-carrier=header)',
    'x-clerk-auth-reason': 'token-invalid',
    'x-clerk-auth-status': 'signed-out'
  },
  request_id: undefined,
  error: {
    message: 'Input required: specify "prompt" or "messages"',
    code: 400
  },
  code: 400,
  param: undefined,
  type: undefined
}

I removed the model from the api call

yeah that models parameter messes things up

maybe I should for now move to just anthropic/claude-3.5-sonnet instead of beta for now?

[dusty isle]

yeah you’d have amazon bedrock as fallback there

also vertex but they’re yellow on our status so less likely to be routed there

[astral drum]

are these 2 models equivalent?

I am .. not so sure

not from the output I am seeing on my end

oh please ignore this message ^ bug on my side

its working .. so same model.. the latency is real but thats okay

[dusty isle]

yeah should be same model just that beta tagged model is self moderated

first is that beta model with no fallback, second is the one you’re on now, so should be noticeably better uptime with maybe more variable latency

[astral drum]

last question:
why am I getting a response back from o1-preview? 😮
https://openrouter.ai/api/v1/generation?id=gen-1736900302-yZudZRBRbVCgfIeoATDk

I am pretty sure my models are sonnet:beta and sonnet

[dusty isle]

that’s a @midnight nest question haha

[astral drum]

I don't really mind.. but would be good to know

if I had to guess its cause of the models fallback logic

[dusty isle]

if you’re specifying two models and not using our auto-router i don’t know that that is intended behavior

but let’s see what lab says

[astral drum]

how do I configure the auto-router ?

    if (finalModel === null) {
      finalModel = "anthropic/claude-3.5-sonnet:beta";
    }

    console.log(finalModel);

    // now create the request for the open router one
    const streamResponse = await openRouterClient.chat.completions.create({
      // model: finalModel,
      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      'models': [finalModel, 'anthropic/claude-3.5-sonnet'],
    });

the finalModel here is: anthropic/claude-3.5-sonnet:beta

[echo girder]

slightly unserious

[dusty isle]

https://openrouter.ai/docs/model-routing

https://openrouter.ai/openrouter/auto

wait that’s the same link lol — ok fixed

[astral drum]

I am pretty sure I am not sending the model or anything over there

so does it get set to auto ?

which would be my guess ...

[dusty isle]

yeah no i don’t believe this is what’s happening

according to your code

[astral drum]

all right I will wait for you all to come back to me about this

    const streamResponse = await openRouterClient.chat.completions.create({
      model: finalModel,
      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      'models': [finalModel, 'anthropic/claude-3.5-sonnet'],
    });

this never fallsback to o1-preview

when I am setting the model over here .. but if I remove it .. it falls back to o1-preview

[dusty isle]

sorry remove what, the anthropic/claude-3.5-sonnet?

[astral drum]

woops sorry.. when I use the openRouterClient.chat.completions.create and set the model: finalModel then there is no fallback to o1-preview

but when I remove that field from the API .. it fallbacks to o1-preview

[dusty isle]

sorry just to be specific, removing model: finalModel and while still having the models array?

[astral drum]

yes correct

[dusty isle]

that’s Definitely a @midnight nest question 😅

thanks for the details

[astral drum]

no worries, for now I will just migrate to anthropic/claude-3.5-sonnet so we don't have accidental o1-preview fallbacks.

I will wait for you all to tell me how to set 2 models on the api

[midnight nest]

Can you check if you have any default model fallback in https://openrouter.ai/settings/preferences

Potentially you were also routed to our default fallback, which route you to o1

[midnight nest]

I'd do something like this:

    const streamResponse = await openRouterClient.chat.completions.create({
      model: finalModel,
      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      'models': [finalModel, 'anthropic/claude-3.5-sonnet'],
      "provider": {
        "allow_fallbacks": false
      }
    });

[astral drum]

oh right missed the provider, allow_fallbacks

so I keep the model: finalModel as well in the api call?

[midnight nest]

I'd remove it too

[astral drum]

removing the model: finalModel leads to o1-preview fallback while keeping the rest of parameters like models and provider

[midnight nest]

oh wat

so removing it will act like it ignores the models and provider?

Oh actually mb -- you can actually do this:

await openRouterClient.chat.completions.create({
      model: finalModel,
      // @ts-ignore
      messages: finalMessages,
      temperature,
      max_tokens: maxTokens ?? 8192,
      stream: true,
    }, {
      // add the models parameter to the request as a fallback
      'models': ['anthropic/claude-3.5-sonnet'],
      "provider": {
        "allow_fallbacks": false
      }
    });