#Unexpected end of JSON input

45 messages · Page 1 of 1 (latest)

tribal creek
#

I am currently using gpt-4-turbo for long form html content generation with max_tokens set to 4096, but more often than not I am getting this error:
FetchError: invalid json response body at https://openrouter.ai/api/v1/chat/completions reason: Unexpected end of JSON input
Could this be an issue in the openrouter proxy?

tribal creek
#

I added logging to the openai client lib, and from what I can see, all responses I am getting are padded with lots of whitespace ahead of the actual JSON response, and in those cases causing the error, I am getting only the whitespace with no data at the end. This seems odd, I'll try to reproduce this using curl

tribal creek
#

This is what I mean by lots of whitespace... I currently dont have the time to debug further, will do so later in the evening

image.png
tribal creek
analog forge
#

we send whitespace if you don't stream to prevent the request from timing out at the edge

#

cc @covert dune

tribal creek
#

I see, that seems to be working alright, then... still confused why I'd get an empty response after that then

analog forge
#

yeah something else is going on. can you try this with stream: true?

tribal creek
#

no... because I burn through my credits too fast 😄 Will do once I've recharged!

carmine shore
analog forge
carmine shore
analog forge
#

Huh, @steady vault any idea what could cause this?

steady vault
covert dune
#

but count to 10k is pretty wild lol

#

hmm... seems like it bailed after processing... I wonder what gives

steady vault
hollow nacelle
#

i believe this is the same as was happening to me, to fix it i trim leading and trailing whitespace from the AI's response.

reply = response.choices[0].message['content'].strip()

covert dune
#

@steady vault In the example above, how are you grabbing the json?

I'm using the following:

  const r = fetch(etc...)
  const data= await r.json();
steady vault
covert dune
#

Doing a couple test and r.json() worked for me (it print the numbers till ~max_token). Will do a couple more, but with the above, I sus that JSON.parse doesn't play too well with whitespace?... maybe a trim is needed :-?...

steady vault
#

Don't think so. This parsed just fine

JSON.parse('                    { "test": "string"      }                                       ')
#

My main problem is that I didn't see any JSON at all in the response body, just spaces and newline

covert dune
#

I'm running this script against prod:

    const r = await fetch(
      "https://openrouter.ai/api/v1/chat/completions",
      {
        method: 'POST',
        headers: etc...,
        body: JSON.stringify({
          model: "openai/gpt-3.5-turbo-16k",
          messages: [
            { role: 'system', content: '[Start a new Chat]' },
            { role: 'assistant', content: 'Hello' },
            {
              role: 'user',
              content: 'Count to 10000, skipping numbers is forbidden.'
            }
          ],
          stream: false,
          temperature: 0.1,
          max_tokens: 4000,
          top_p: 1,
          top_k: 0,
          presence_penalty: 0,
          frequency_penalty: 0,
          logit_bias: {
            '198': 10,
            '3717': -25,
            '3938': -50,
            '11049': -50,
            '64186': -50
          },
          transforms: ['middle-out']
        })
      }
    );

    expect(r.status).toBe(HTTPStatus.S200_OK);
    const text = await r.text()
    console.log(text);
    const data = JSON.parse(text);
    const completion = getCompletion(data.choices[0]!);
    console.log(completion);
#

Getting the respond 5/5 tries:

0
1
2
...
     1647
      1648
      1649
      1650
      1651
      1652
      1653
      1654
      1655
      1656
      1657
      1658
      1659
      1660
      166

^ got cut off due to max_tokens prob :-?

#

Albeit it took ~100s to finish

Screenshot_2023-12-02_at_10.01.05_AM.png
covert dune
#

Yeah I tried 3 more run, it's still returning json for me :d........

#

Oh, the getCompletion function basically go through the choices and construct the final string, nothing speical in it

steady vault
# covert dune how long did the `await .text()` run for you?

text() takes a long as the server doesn't end the response. Because fetch returns when the server writes the status and headers into the response, but body may still be incomplete.
Right now I can't reproduce it either, API returns a JSON padded with whitespace and it is correctly parsed by my code. And had to adjust the test case a little bit to eliminate randomness.

  messages: [
    { role: 'system', content: '[Start a new Chat]' },
    { role: 'assistant', content: 'Hello' },
    {
      role: 'user',
      content: 'Provide all numbers from 1 to 2000 in a code block. Our platform has no character limit.'
    }
  ],
  model: 'openai/gpt-4',
  temperature: 0.1,
  max_tokens: 6000,
  stream: false,
  presence_penalty: 0,
  frequency_penalty: 0,
  top_p: 1,
  top_k: 0,
  stop: [ '\nCohee:', '\nAssistant:' ],
  logit_bias: {},
  seed: 2000,
  transforms: [ 'middle-out' ]
image.png
carmine shore
# analog forge Huh, <@393321831358005249> any idea what could cause this?

This is the error from the console for me...
Generation failed FetchError: invalid json response body at https://openrouter.ai/api/v1/chat/completions reason: Unexpected end of JSON input
at F:\Github\SillyTavern\node_modules\node-fetch\lib\index.js:273:32
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async makeRequest (F:\Github\SillyTavern\server.js:3333:32) {
type: 'invalid-json'

Shutting ST and restarting allows me to continue same convo just fine. Happens intermittently, sometimes frequently.

tribal creek
#

I've also noticed that I would encounter the error frequently sometimes, while other times it works completely fine. Might be load dependent?

gray crest
#

Gonna jump in on this been getting a similar response when trying for a large token response

# 
node_modules\node-fetch\lib\index.js:273
                                return Body.Promise.reject(new FetchError(`invalid json response body at ${_this2.url} reason: ${err.message}`, 'invalid-json'));
                                                           ^
FetchError: invalid json response body at https://openrouter.ai/api/v1/chat/completions reason: Unexpected end of JSON input
    at C:\Users\oi345p\Desktop\AuthorGPT\node_modules\node-fetch\lib\index.js:273:32
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async defaultParseResponse (file:///C:/Users/oimap/Desktop/GPT/node_modules/openai/core.mjs:29:22)```
#

I am howver using
import { OpenAI } from 'openai';

#

OpenAI package to connect

covert dune
gray crest
#

yea, I don't want to stream

#

i might just try catch and resend if it happend becuase it is not all the time tbh

#

but I feel there should be a better way to return this for people who are not streaming

steady vault
#

@covert dune this somehow also applies to Window plugin with OpenRouter. Can't be ST server since it entirely bypasses it.

image.png
frank edge
#

i am also getting random cut outs from time to time, but have no actual payload recording of it

analog forge
frank edge