#Response / Analysis LLM Discussion

1 messages · Page 1 of 1 (latest)

eternal turret
#

Thanks @limpid sky

A couple follow up questions if you don't mind so I can better help you.

  • Which model are you using?
  • What temperature?
  • If you don't mind sharing an example of a prompt with me that would be awesome
#

If you want to keep it private though I understand and I can just point you in the direction of some easy fixes.

#

I'm pulling up an example of my own attempt at this with my bot in Playground real quick.

limpid sky
#

I'm using 3.5 currently and I understand that switching to 4 will most definitely improve the quality, but for now I'm trying to make do with that.
Temperature is 0.3, the prompt I used was "Every time you respond, add a "\t" after you're done speaking. Then analyze the tone of USER's last message and append it to your response. Answer "YES" if you understand."

#

It works flawlessly at first, but at some point it seems to occasionally start forgetting stuff or incorrectly presenting it.

#

One thought I had was to remind the model constantly to follow this instruction (something to do with context length? Still learning the terminology) but that seems inefficient.

#

In general, if I understand correctly, you're supposed to add a bunch of context to the user's message under the hood, correct? If the user inputs "Hi", I would append/prepend a bunch of instructions as "system". Or is it fine to just keep it in history?

eternal turret
#

Getting back to this now, was just answering a quick question about embeddings.

limpid sky
#

No rush

eternal turret
#

A few quick remarks from what I can see initially:

  • The prompts get easily confused if you have multiple "answers". For example if I take out Answer YES if you understand then suddenly the model starts behaving.

  • The way you ask for a special character also affects the model. Instead of asking it to put it after the response, try putting it before the analysis.

Here's what I get on 0.3 temperature with GPT 3.5 Turbo using this updated prompt

Every time you respond, speak directly to the user. Then analyze the tone of USER's last message and append it to your response.  Add a "\t" before analyzing the tone.
image.png
limpid sky
eternal turret
# limpid sky Yep, looks very similar to what I get. The prompt is also a lot more understanda...

That's a really good question. If you can come up with a good answer then you'd have the admiration of the entire OpenAI community. 😁

My method to prevent it was to break this style of prompt up into separate prompts.

For example, my bot asks the following questions when it receives a new message:

  1. How can I help the user and what is their tone?
  2. Do I have enough information to respond or do I need more context?
  3. Are my memories enough context or should I ask the user for more information?
  4. Finally, I append all of the answers above to a single prompt and ask it to use all of the information provided to best respond to the user.
#

The reason for breaking it up to prevent deterioration is that you may suddenly get the wrong behavior from a single part of that "chain", and it's easier to fix that one simple part than to fix the whole prompt. I've spent years chasing "Single-Attempt / Single-Shot" prompts around and they always deteriorate over time unfortunately.

#

By keeping each prompt itself to one, or at most two focused questions, you can keep that reliability high.

limpid sky
#

So every time I send a call, I'm should be appending several prompts to the history anyway?

#

Or do you mean sending several calls, waiting for each of them, then appending to a single reponse?

eternal turret
#

If you have multiple passes of analysis you want to do then yes definitely, make a series of prompts with no history and then combine them together.

Even a simple two-shot prompt in your case would increase reliability, e.g. you could use GPT-3.5 to get the tone of the user. Then pass its response to GPT-4 and make a "human sounding response"

eternal turret
limpid sky
eternal turret
limpid sky
eternal turret
#

I try to run as many questions in parallel as possible, and I try to make requests in-between messages to "think ahead" to save time, but yeah there's no getting around the speed issue.

You'd need to run a local GPT model if you want very fast responses.

limpid sky
#

So getting back to my initial idea, would fine-tuning help to remedy this issue?

#

Or is it better suited for cofining the model to a particular "style", not hard limitations?

eternal turret
#

I would follow OpenAI's advice here right from their docs:

Fine-tuning OpenAI text generation models can make them better for specific applications, but it requires a careful investment of time and effort. We recommend first attempting to get good results with prompt engineering, prompt chaining (breaking complex tasks into multiple prompts), and function calling, with the key reasons being:

  • There are many tasks at which our models may not initially appear to perform well, but results can be improved with the right prompts - thus fine-tuning may not be necessary
  • In cases where fine-tuning is still necessary, initial prompt engineering work is not wasted - we typically see best results when using a good prompt in the fine-tuning data (or combining prompt chaining / tool use with fine-tuning)
#

Your prompt seems safe enough that you should be able to get far with it just by tweaking it. If it's just analysis + response then GPT is well suited to it in my opinion.

#

Ultimately though the work won't be wasted. If prompt tweaking can't get you the result you want then you'll need the prompt for the fine-tuning anyway.

limpid sky
eternal turret
#

Never hurts to try it out if you're feeling up to fine tuning. :)

#

Good luck with the chaining. Feel free to poke me or ask around the channels any time if you need help with it.

limpid sky
eternal turret