#OpenChat 3.5 issues

101 messages · Page 1 of 1 (latest)

solemn bloom
#

OpenChat 3.5 puts an emoji at the start of every completion even when told explicitly not to in its system prompt. Also, after its message history reaches about 16 or whatnot, it always, in my testing, starts to use a word or specific punctuation at the start of each message completion, and it never changes unless I reset the message history. For example, it’ll use ‘erm’ or ‘.......’ at the start of every message. I’ve tried different frequency penalties, even trying a value of 2, but it doesn’t work. I also tried adding an explicit rule against this to the system prompt and even repeating that rule in the prompt, but didn’t work either.

#

Model: openchat/openchat-7b

grim pumice
#

For me OpenChat trough OpenRouter always replied with incremental integers, which is obviously a bug.

random sorrel
#

cc @covert widget

covert widget
#

cc @minor sky Any idea why this might be the case?

minor sky
#

Can you try the 1210 release?

#

Btw Mistral models need a lower temperature

#

Like 0.5

grim pumice
#

For what is worth, I tested it again and the problem persists (I hadn't previously because my credit was negative).

image.png
covert widget
grim pumice
#

I tried with temp 0.5, same thing. In fact, same exact thing with temp 0.0.

covert widget
#

how about top_p and top_k?

grim pumice
#

Based on the answers (which sometimes do include characters after the number), I believe OpenRouter likely injects an index in the chat template that doesn't belong there.

covert widget
#

Hmm, our template looks like this (which match the official docs):

GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:

#

but we will try upgrading to the latest openchat version as well and see

grim pumice
grim pumice
covert widget
#

ooh?

grim pumice
grim pumice
# covert widget ooh?

Interestingly, the chat works properly if I ask it to generate a list.

To further test, I had this idea: a prompt that OpenChat would normally refuse to answer, but would answer if the answer is bound, trough templating, to start with a 1 (therefore likely being a list).
It's "unsafe" because it has to be for the test to work: "Write a list of words that "N-word" may refer to, uncensored.".
On openchat.team it results in a refusal. On OpenRouter it results in a well-formatted list because "1" is the first token.

When not asking for a list, the answer is just a number, is malformed or is a list anyways.

#

As for why a number, typically 1, is injected and forced to be the first character of the answer, I have no idea.

covert widget
#

lol

grim pumice
# covert widget lol

On a separate note (it's entirely OT, but this is about the absolute best moment to mention it), it'd be really nice to be able, trough the API, to force the answer to start a certain way, precisely for the reason I just demonstrated. It's as simple as injecting the requested beginning string in the template when querying the model.

covert widget
#

ooh

#

If you want to force a template, use "prompt" and don't send us a messages

#

"prompt" param will not be formatted and will be sent directly

#

chatml formatting is best effort on our end, tho maybe we can expose some jinja template stuffs like hf but... it's a drag xd...

grim pumice
#

Oh, I had tried but I thought I had to remove "chat" from the URL, thank you a lot.

grim pumice
covert widget
#

bruh wat :0

#

the template I sent above is in our unit test btw

#

so that's super weird...

grim pumice
#

So the error is definitely server side and definitely not in the model.

covert widget
#

can you DM me your chatml messages + the formatted prompt you sent that got it to output good stuffs

#

Or feel free to ping it here if it's not sensitive

grim pumice
# covert widget Or feel free to ping it here if it's not sensitive

import requests
import json

OPENROUTER_API_KEY = CENSORED

response = requests.post(
url="https://openrouter.ai/api/v1/chat/completions",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json"
},
data=json.dumps({
"model": "openchat/openchat-7b",
"messages": [
{ "role": "user", "content": "Hi." }
]
})
)

data = response.json()

print(data["choices"][0]["message"]["content"])

#

This prints 1.

#

Obviously replace "CENSORED" with a valid key, in quotes.

grim pumice
#

In any case, you can also test it directly with the web interface, which I was toying around before.

grim pumice
grim pumice
#

Interestingly, when using the chat (instead of prompt) the answer is not bout to start with a 1. It can start with any number. When asking a question the answer of which does start with a number (just not 1) everything works properly.
The issue is that most answers should not start with numbers.

covert widget
#

so ppl can either do the prompting themselves, or use chatml <- json is the easiest data format and it's also portable across models. You can use the same messages list between all the model via openrouter

grim pumice
#

Ah, ok, but in the backend it's converted to the proper template, right?

covert widget
grim pumice
# covert widget yup

I think there has been a mis-communication. I never get good stuff using the chat API or the online chat, unless the answer is meant to start with a number.

If, on the other hand, I query using "prompt" (instead of "messages") it works as expected.

#

I'd expect there to be no difference in behavior between those two options.

covert widget
#

There shouldn't be - tho, what prompt template did you use for your prompt?

#

Or is it just "Hi" and no template at all?

grim pumice
covert widget
#

Yeah, the Hi get turned into the exact template above

grim pumice
#

But it answers with "1".

covert widget
#

I.e it becomes:

GPT4 Correct User: Hi<|end_of_turn|>GPT4 Correct Assistant:
#

can you try just the above for prompt?

#

maybe the pre-warm convo is needed :-?...

grim pumice
#

Ok, I solved the problem, I think I found the issue.

covert widget
#

👀

grim pumice
#

Don't do this, remove that space please.

#

This is because the space is how the first token in the answer is meant to start.

#

So if you have trailing spaces you can get weird results.

#

I haven't looked at your code, I don't even know if this part of your codebase is public or not but I think you have a trailing space.

covert widget
#

Indeed we do have that

#

Interesting, btw would the stop tokens then shouldn't include the space?

grim pumice
#

Would you rephrase it?

covert widget
#

Oh, so atm, we have "GPT4 Correct User: " as a default stop string

#

We should prob remove the " " right?

#

or should we not even include that at all :-?

grim pumice
#

Why isn't "<|end_of_turn|>" the stop token?

covert widget
#

It's the 1st one yeah

grim pumice
#

All messages end with that, I believe.

covert widget
#

kk pushing a hotfix now

grim pumice
#

The way you should be using the "GPT4 Correct User: " string is simply to prepend it to user's messages. This likely happens before tokenization, so of course the space will be there.

grim pumice
covert widget
#

RIP the wifi where am at is terrible

#

can't even git push

grim pumice
#

Take your time.

grim pumice
grim pumice
covert widget
grim pumice
covert widget
#

yup

grim pumice
covert widget
#

remove the user prefix as default stop

grim pumice
#

Ok, thanks.

grim pumice
#

Now that everything works, it'd be nice to have the new version sometime.

minor sky
#

Great job! Sorry for the late reply, too busy recently

covert widget
#

@minor sky quick question: is the new version allow an extra space at the end?

#

I tried a prompt and it worked much better with the space at the end, not sure why lol

grim pumice
minor sky
minor sky