#Failed to set max_tokens

1 messages · Page 1 of 1 (latest)

frail apex
#

I added a custom endpoint and added the parameter max_tokens, but it didn't seem to take effect.

20240127_135117_086_copy.png 20240127_135136_799_copy.png
#

Failed to set max_tokens

oak imp
#

unfortunately, that's not how it works. most of the tokens handling is done before anything is sent to the API. I can address this specific issue though

frail apex
#

Yes, I did not see the request in the background logs of the API. In fact, it was intercepted in LibreChat.

oak imp
#

yeah max_tokens will only specify the max tokens the LLM should use for the response, but it will stay within its context window

#

if the app determines the context window is shorter than your prompt, than it will throw the error

frail apex
#

Can this issue be resolved in the current version?

#

I did not see the relevant configuration parameters.

oak imp
#

im pushing a "bandaid" fix in a moment but I will do a more robust, configurable solution in the near term

frail apex
#

Remember to include logs about the librechat.yaml file is loaded, not having logs when manually adding endpoints is a pain because sometimes you just don't know what went wrong!

oak imp
frail apex
#

I don't see anything related to the librechat.yaml file in the dockers logs, and sometimes something goes wrong with it and I have no idea what the problem is!

oak imp
#

I'm adding this:

2024-01-27 08:19:47 info: Loaded custom config file:
2024-01-27 08:19:47 info: {
  "version": "1.0.1",
  "cache": true,
  "endpoints": {
    "custom": [
      {
        "name": "Mistral",
        "apiKey": "user_provided",
        "baseURL": "https://api.mistral.ai/v1",
        "models": {
          "default": [
            "mistral-tiny",
            "mistral-small",
            "mistral-medium"
          ]
        },
        "titleConvo": true,
        "titleMethod": "completion",
        "titleModel": "mistral-tiny",
        "summarize": false,
        "summaryModel": "mistral-tiny",
        "forcePrompt": false,
        "modelDisplayLabel": "Mistral",
        "addParams": {
          "safe_prompt": true
        },
        "dropParams": [
          "stop",
          "user",
          "frequency_penalty",
          "presence_penalty"
        ]
      },
      {
        "name": "OpenRouter",
        "apiKey": "${OPENROUTER_KEY}",
        "baseURL": "https://openrouter.ai/api/v1",
        "models": {
          "default": [
            "gpt-3.5-turbo"
          ],
          "fetch": true
        },
        "titleConvo": true,
        "titleModel": "gpt-3.5-turbo",
        "summarize": false,
        "summaryModel": "gpt-3.5-turbo",
        "forcePrompt": false,
        "modelDisplayLabel": "OpenRouter"
      }
    ]
  }
}