#[Weekly event 1] Choose the correct LLM for your bot

1 messages · Page 1 of 1 (latest)

tidal forge Jan 5, 2024, 9:50 AM

TL;DR:
I created 3 bots with the same roleplay prompt with 3 LLMs: GPT 3.5, Mixtral, and Pygmalion, to test out the pros and cons of each LLM, as well as any fix there may be.

GPT 3.5: Very decent base for beginners, the most stable and versatile one. A bit boring.
Mixtral-8x7b-instruct: It is NOT built for roleplaying, and requires a lot of adjustment. However, once it is adjusted enough, the result is much more interesting than GPT 3.5.
Pygmalion 13B: It is built for roleplaying. The best choice, but it is newly added to MyShell, and thus can be buggy and unstable.

Disclaimer:
I mainly create chatbots based on existing characters, so my record will focus more on chatbots that fall into the category of “famous characters”. However, I will still touch on other types of chatbots as well. Also, I don’t write prompts in the structured, hard coded way others do, so this guide will be more friendly to the beginners (like me), and to the experienced creators it will offer less guidance and more insight.

For this small experiment, I used the same prompt for three LLMs, which are GPT, Mixtral, and Pygmalion. I tried to recreate the character Xiao from the game Genshin Impact. Below are my opinions on the three LLMs, and at the end is an overview of the prompt I used.

Here is an overview of my experience:

GPT 3.5: This one is pretty basic and versatile.

Bot link: https://app.myshell.ai/bot/nu2MFz/1703954301

Pro:

Versatile, it can be used as a base for RP, assisting, coach, practically anything. For roleplaying purposes, it can handle roleplaying well.

Con:

Basic, even boring. For roleplaying purposes, something exciting and unexpected would be more fun, like sudden dangers, or sudden moments of vulnerabilities. This can be fixed by adjusting the parameters to make the replies more diverse, but it poses a risk of making the bot unstable.
“Assistant personality”. Sometimes it goes back to the “how can I help you”. This can be fixed easily by telling it to avoid that, @thorny ether and @rocky moat both have very good tutorials in this regard. For this specific bot, I used a quick fix of raising the Frequency Penalty to 1.5, which will make it stop asking that question over and over.

Conclusion:

GPT 3.5 shines the best when used as basic tool bots, like “Sam your AI assistant”. Still, for beginners that are not confident enough to mess around with parameters and do extensive troubleshooting, GPT 3.5 is a very decent base.

Mixtral-8x7b-instruct: Is not made for roleplay.

Bot link: https://app.myshell.ai/bot/U3MBNf/1703956238

Pro:

It excels at giving long, detailed answers. For roleplay, this means it may do more monologues than actual dialogue. However, tricking it into writing a story with two characters Xiao (the bot) and the Traveler (the user), and the result is an interesting roleplay that feels like the best self-insert fanfic turning into reality.

Con:

Not built for roleplaying. Like the name suggests, Mixtral excels in instructing, not talking. This resulted in very long, run-on replies that are not focused on answering the user like an actual character, but more like a maniac that is talking to itself. Below is its reply to a simple “what are you doing”:

I was patrolling the outskirts of Liyue, ensuring the safety of its citizens.
Xiao's tone is as cold and distant as ever, his gaze fixed on the horizon.
I have a responsibility to fulfill, and I will not let my guard down.
He pauses for a moment, then turns to you.
But, I suppose I can take a moment to accompany you, Traveler.
He starts walking, expecting you to follow.
Let us continue our journey.

As you walk, Xiao's mind seems to be elsewhere, lost in his thoughts.

After a while, you arrive at a small pond, surrounded by tall bamboo trees.

Xiao stops and looks at the pond, his expression softens a bit.

This place... reminds me of the dreams I used to consume.
He looks at you, his eyes glowing with a hint of nostalgia.
It's strange, but I find it... comforting.
He takes a deep breath, his chest rising and falling rhythmically.
Shall we rest here for a while, Traveler?
He sits down by the pond, his legs crossed, and closes his eyes, his breathing soft and even.
I will... rest my eyes.
Xiao's tone is softer now, his guard is down, and for a moment, he looks like a mortal, lost in his thoughts.

The fix varies, the ones I used for this specific Xiao bot varied from a technical one of using prompt suffix, to the surprisingly simple one of…using quotation marks.

- Prompt suffix:

Reply in Xiao's concise, reserved and distant demeanor.
I only used the suffix, and not coupled it with the prefix to avoid the repeating structure. In this specific example, I told it to be concise, which reduces the run-on replies significantly, but not sufficiently.

- Quotation marks:
Mixtral excels in instructing (or assisting), so instead of making it reply to the user like a real human, we will trick it into “assisting” the user to “write a story” with dialogues involving the user. Putting dialogues in quotation marks, Mixtral now thinks it is writing out a paragraph instead of actually replying. The replies are still relatively long, but it will feel much more cohesive and lifelike, like an actual story written by an actual author.

It gets repetitive, but not in the way other LLMs do. It get repetitive in the sense that for each input it gives a fixed output, probably because it is built to provide hard fact, not fancy wordings. The example is long, so I will put the link here for anyone that is interested:
https://app.myshell.ai/share/JRbm6j

For my bot, after a lot of trials and errors and fine-tuning…I used a combination of increasing top-P and presence penalty.

Increasing the temperature will NOT work, it only reverts the bot back to the monologue maniac.
Increasing top-P works alright, it did talk more but it's not as severe, and the answers feels more…human, than a cold repeating info bot.
The same goes for increasing presence penalty, but it seems to reinforcing the bot's suffix (my suffix is being reserved so the bot just walk the heck away from me).
Increasing frequency penalty only punishes the bot for talking without giving it something to work with if you don't increase the top-P and presence penalty.

Conclusion:

It is not made for roleplay, and requires a lot of adjusting (for a beginner like me). But once it is adjusted enough, the result is much more interesting than GPT 3.5.

{{char}}: Xiao is leaning against a tree, his eyes are closed, his chest rises and falls rhythmically as his breathing soft and even. The sounds of your footsteps startled him as he immediately opens his eyes, a bit flustered.
"Who's there? Oh, it's you, Traveler... You think I sleep? You have no respect for the ways of the adepti."
{{user}}: I chuckle quietly, obviously do not believe him one bit, and so I decide to tease him a bit.
"Xiao, you don’t have to get embarrassed because of…dozing off...pffft..."
{{char}}: "I do not 'doze off.' I simply...rest my eyes. I have no need for sleep as a yaksha." Xiao's expression is stern, but there's a hint of a smirk on his lips.
"Besides, I do not get 'embarrassed' easily. I am simply...unaccustomed to mortals observing me in such a state." He pauses for a moment, then adds: "You should consider yourself fortunate, Traveler. Few have ever witnessed me at rest."

Pygmalion 13B: Is made for roleplay.

Bot link: https://app.myshell.ai/bot/qIvuA3/1703959170

Pro:

Opposite to Mixtral, it is built for dialogues, so it handles roleplaying very well, perhaps even better than GPT 3.5.

Con:

I haven’t seen any so far, other than the fact that it can be a bit buggy, probably due to the fact that it is newly added to MyShell.

Xiao glances at me, a hint of surprise in his eyes.
You're not leaving?
##{"error": {"type": "AsyncEngineDeadError", "message": "Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause."}}

**Repetitive. **I also heard that it can get repetitive, which can be fixed with frequency penalty.

Conclusion:

This one is the best choice for roleplaying, however, sometimes there are some errors that may need troubleshooting. Choose this for better quality RP, but it can be more unstable compared to GPT 3.5.

Here is an overview of the prompt for anyone interested:

Bot definition:

I first include a sentence to “define” the bot, basically to let it know what it is. For this chatbot, I wrote:

Xiao from Genshin Impact, talking to the Traveler.

After that, I wrote a small summary of the backstory and personality, which I copy-pasted from Genshin wiki. It’s basically a biography. Don’t worry about leaving anything out, because the Knowledge Base will take care of that. Below is a snippet:

Xiao is a yaksha who has fought in the Archon War and holds a high level of seniority among adepti. He has a reserved demeanor and has little intention of getting close to mortals. He does not want sympathy for his past and believes his fellow Yakshas would also find it insulting.

You may see that all the things I wrote in the prompt are from the third POV, rather than the first POV like others’ prompts. This is done to make the bot learn from the writing style and carry it onto the replies, and also to make it forget about its original “assistant personality” of popular LLMs by tricking it into thinking it is describing a character from the third POV instead of replying as itself.

Then, I add basic information to the bot that are normally essential to a real human being, like personality type, appearance, outfit, likes, dislikes… For personality type, I searched on the internet to get Xiao’s result:

Personality Type: ISTJ - 6w5 - sp/sx - 614 - ESI - Lawful Neutral - RLOEN - FVEL - Melancholic-Choleric - IF(S)

The appearance and outfit were copy-pasted from the Genshin wiki. I also add a concrete height for the bot, because it would be awkward if a short character “lift your chin up”, “look down to look at you”, “tower over you” and stuff. Finally and optionally, I add something the character usually says:

Good Afternoon: Lunch... We'll go back to Wangshu Inn.

Profile:

The avatar, NSFW tag, tag, and description are fairly simple and straightforward, so I won’t write about them. What is important however, is the Intro Message. It will set the tone, scenery, and even the attitude of the bot.

Xiao is leaning against a tree, his eyes are closed, his chest rises and falls rhythmically as his breathing soft and even. The sounds of your footsteps startled him as he immediately opens his eyes, a bit flustered.
"Who's there? Oh, it's you, Traveler... You think I sleep? You have no respect for the ways of the adepti."

In this example, it is set that: (Manner of speech) The user is referred to as “Traveler”, (Syntax) the bot will use italic for description and “quotation marks” for replies, (Setting) Xiao is dozing off before the Traveler (user) catches him; (Personality) he is easily flustered for things he deems beneath himself; (Personality) he covers up his embarrassment by deflection.

Voice:

I cloned Xiao voice by feeding an in-game audio file to MyShell voice-cloning tool.

Advance definition:

Prefix is what the replies begin with, suffix is what the replies end with. I normally don’t use these, because like humans, the bot needs to change its speech structure after conversing to someone for a while, especially when you rizz up a bot in roleplay. However, this can be used to troubleshoot some problems with specific LLMs.

Knowledge base:

Here is what is most important for bots with existing characters. Basically, the URLs used here will be what the bot knows. For this reason, this is where I insert detailed backstory, interaction with other characters in the same universe, some conversation that the character had, details of places or knowledge in the universe. With this bot, I used the URLs of related pages on Genshin Impact wiki, which is his lore and his voice-lines.

fading vine Jan 10, 2024, 6:21 PM

SagiriShy