#Question about prompt formatting when using chat history/for bot purposes...

15 messages · Page 1 of 1 (latest)

maiden star
#

So I'm trying to figure out the proper way to format a query via API for the purposes of a chat bot.

Right now, I'm using the https://huggingface.co/TheBloke/WizardLM-13B-V1-0-Uncensored-SuperHOT-8K-GPTQ model.

According to the model card here, the prompt format is:

USER: prompt
ASSISTANT:

Right now I have my prompt formatted like this:


USER: hey there
Greta Thunbot: hello there, i know almost everything about almost everything. how may i help you?
USER: what do you think about America?
Greta Thunbot: i think its a nation of many possibilities though not without its problems.

Then, I append the current conversation to the end of it, such as:

USER: whats going on
Greta Thunbot: not a whole lot, just managing the world
USER: that sounds fun

And then add the following for response purposes:

So, in full, the query would look like:


USER: hey there
Greta Thunbot: hello there, i know almost everything about almost everything. how may i help you?
USER: what do you think about America?
Greta Thunbot: i think its a nation of many possibilities though not without its problems.
USER: whats going on
Greta Thunbot: not a whole lot, just managing the world
USER: that sounds fun
Greta Thunbot:

Given the prompt template, I guess I have a few questions...

1.) Do I NEED to use "USER" and "ASSISTANT" or is "USER" (or something like, xydreen, or other people) and "Greta Thunbot" fine for the formatting?

2.) What exactly is the proper way to specify an example dialogue (ie the 4 lines of chat that is attached to every prompt to show the LLM how to respond)?

3.) What exactly is the proper way to specify the current conversation/the one the LLM needs to respond to?

It's working pretty well at the moment though there are some differences between how the webui chat responds vs how my query via API responds. However, I'm wondering if there's a more proper way to format things here - mainly the example dialogue and the current chat.

It doesn't look like the verbose flag on textgen shows the webui chat history in the prompt so I can't tell exactly what webui is sending to the LLM for processing.

Thanks!

eye.png
TheBloke/WizardLM-13B-V1-0-Uncensored-SuperHOT-8K-GPTQ · Hugging Face
slim patrol
#

1: a lot of chat-tuned models are meant to have exactly the USER: and ASSISTANT: prefixes, though not all require it.
WizardLM I believe requires it

#

2: yeah just shove it at the top

#

3: yeah just shove it at the bottom with a blank space where it's gonna reply lol

#

whatever shows up in --verbose is actually what gets sent, so if things aren't showing up that means they're not being processed and you should look into why

maiden star
slim patrol
#

it does in ui, idk about API

maiden star
#

oh what the shizzle

#

i must have been smoking crack earlier because i didnt see the chat from the ui show up at the end of my prompt in verbose

#

its there though, well okay thats good

#

well that makes it a bit easier

maiden star
slim patrol
#

yeah multi-person conversations are hard on chat-tuned models

maiden star
#

ideally i want to be able to replace USER: with an @ discord id, so if the bot ever wanted to specifically reference a person, it would do it like that and itd also trigger a mention on discord

#

i guess a way to get around it partially would be to reference the person by name in the top prompt area above the current convo, like You are talking to BLAH. then have it show the messages from USER to ASSISTANT, but oh well, will cross that bridge if it comes to it