Tf TPU Transformer Chatbot project

[atomic perch]

Hi everyone, I’m just now going to get more involved in this server, now that I’m less busy. I’m working on a tf tpu transformer Chatbot project and would like some help understanding some code that is necessary for it. The project is to create a Chatbot that will pass the Turing test, and also be part of the AGI project. I’m more fluent with python, tensorflow, predictions, and this topic is newer to me.

https://github.com/bryanlimy/tf2-transformer-chatbot

This is the code we chose to start from, but maybe someone here knows of a better starting point?

[viscid pasture]

Whats the dataset you're using to train this chatbot? The cornell movie dialog dataset is a nice start pack (ideal for a rough beginner project) but as time has gone on, researchers have found a few ways to do better. First, with the GPT models, they found that training a transformer on a massive corpus of text data would get it fluent enough to generate higher quality coherent text. Then, they would follow that up by finetuning that pre-trained model on conversation samples (ie like the movie dialog dataset) to train it how to converse. With ChatGPT, it sounds like OpenAI took that one step further and used reinforcement learning to train the model on what outputs would seem more "acceptable" or "relevant"

When it comes to actual architecture, you're best starting with the GPT transformer decoder stack and going from there.

[atomic perch]

thanks so much, I just used that dataset (movie dialog), I'm super new to this but I'm familiar with TF, ANN's etc.