openrouter/quasar-alpha tokenizer is wrong

[terse finch]

The maxium token input is around limit 750000 token

[oblique dawn]

which tokenizer are you using?

[terse finch]

I have a long context which is over 2million tokens. But I can only send around 75_0000, when I send longer than this. error will returned.

[terse finch]

data: {"error":{"message":"This endpoint's maximum context length is 1000000 tokens. However, you requested about 1124905 tokens (1124905 of text input). Please reduce the length of either one, or use the "middle-out" transform to compress your prompt automatically.","code":400,"metadata":{"provider_name":"Stealth"}}}

looks like you are using charactor based token counting

[oblique dawn]

yeah it depends on how you're tokenizing your input. we're not using character based token counting though

I can't reveal the exact tokenizer we use unfortunately, but all will be revealed in time!

[terse finch]

Can you fix this issue ? openrouter/quasar-alpha clearly use gpt-4o tokenizer ( cl100k_base ). Can you switch to that ?

[terse finch]

@oblique dawn

[oblique dawn]

sorry about the delay