Alright, so just use this thread for discussing this particular issue, so we don't lose track of it
#ChatGPT alternative for age-rating-agnostic worldbuilding
opaque flame
half crest
Yeah, most of em will have 4k - 8k tokens context. I personally run 13b models with 4k context as i don’t mind the short memory (i heard if you really need a workaround you can use summaries or something called vector storage to keep context for longer), but if you’re looking for something really really good, i’ve heard many good things about NovelAI models and Goliath-120b, NovelAI’s a subscription service and Goliath, unless you have a monster PC its available on platforms like OpenRouter but it’s paid too. Sometimes they offer the models in extended context to 16k but idk how that works
opaque flame
So, you're tossing around a lot of words I'm not familiar with since I'm new to the whole AI scene, so let me just list my requirements:
- I need something that's going to have as much context as possible. I don't care at all about reply speed.
- I have 32 gigs of ram on a gaming laptop with a decent CPU and a GTX 1650
- Paid stuff isn't an option
I'd prefer something online if possible
But I absolutely don't have a dollar to my name, Christmas wiped me out
half crest
Hmm alright
opaque flame
Keep in mind, I'm not trying to generate straight-up smut. The idea is for it to be age-rating-agnostic so I can move towards more adult-oriented fantasy content without actually going full-blown yuck.
But OpenOrca was really giving me crap with trying to remember a simple detail and recall it. It kept repeating itself, too
And ChatGPT threw me an orange-text warning just because one of the characters I was describing leaned too much into the "niche fetish someone's into" category
half crest
OpenOrca is a 7b model?
opaque flame
I don't even know what that means
half crest
Alright lemme try, first of all running models locally, unless u got a really strong PC you’ll be running maybe 20b models at most. The “b” means billions of parameters, which basically is just how advanced the model is (more = better), but also makes it more intense to run. The problem is that chatgpt (gpt 3.5 turbo more precise) has like around 140b, which is way bigger. So with local models, you’ll be looking at occasional drunk ramblings, switching up people and places, repeatings, and low context (4k tokens, maybe 8k opposed to ChatGPTs 16k). That’s something you’ll have to accept if you want to run 100% locally. The advantage of local models are they’re local, most of the time way better at writing non-stale stories than ChatGPT and nsfw, no matter how illogical the things it outputs may be at times.
Usually, local models come in the following: 7b, 13b, 20b, and then higher but that’s usually way out of most people’s PC power leagues. As a rule pf thumb, 8gb ram should run you 7b, 16gb for 13b and 20b needs slightly more (still under 32gb ram) but at that point you’ll need to have a strong gpu/cpu as well. The first step however should be to get an interface to interact with the AI model with in the first place.
half crest
But OpenOrca was really giving me crap with trying to remember a simple detail a...
If i may ask, what did you use to try out OpenOrca?
opaque flame
GPT4All
half crest
For a easy and quick way to try out local models (something i used before moving onto SillyTavern, which is like the golden standard of AI chat/roleplay interfaces, but it requires some setup and is pretty advanced), I’d recommend an app called Faraday, which is a chat-based local model interface. There you’ll be able to try out multiple models and see what runs and not. I also heard good things about LMstudio
I personally like using the 13B model Psyfighter: https://huggingface.co/KoboldAI/LLaMA2-13B-Psyfighter2-GGUF
opaque flame
Here's an example of my qualms with CGPT's incessant moral whining
half crest
But to be honest, i don’t really know how to achieve long memory. I know faraday has a slider to choose between having 2k tokens to 4k tokens to 8k tokens of memory, (1 token = 3/4 words), but idk beyond that as i never particularly cared about memory. The mentioned SillyTavern has some workarounds like summarizing the conversation into a short prompt to delete and replace the summarized messages, but you’d have to look into that yourself
opaque flame
I'm finding out that even ChatGPT has concerning memory loss. I guess I just don't understand how all this works.
half crest
Yeah, AI models in general forget stuff. I guess you can think of it as the AI not reading text as-is, but more like a haze. The dumber the model is the more it will forget, especially the longer the conversation is.
Usually you just re-generate the message tho if something like that happens
It’s not perfect, and the more you dive into local models which are way less “intelligent” the more you’ll see this. In practice it just means you’ll be re-generating responses more often, sometimes nudge the AI into the correct direction
opaque flame
When I re-generate the responses, it just re-types them verbatim
Even after trying to change the definition of what I'm asking for
half crest
ChatGPT?
opaque flame
OpenOrca
opaque flame
What does the "#B" mean?
opaque flame
Thank you. I've found Hermes rather fun to use
half crest
Np, some other favorite models i personally liked (loosely ordered by how much i like em):
https://huggingface.co/KoboldAI/LLaMA2-13B-Psyfighter2-GGUF
https://huggingface.co/TheBloke/Athena-v3-GGUF
https://huggingface.co/TheBloke/Mythalion-13B-GGUF
https://huggingface.co/TheBloke/ReMM-SLERP-L2-13B-GGUF
If you'd be interested, i recommend looking into https://sillytavernai.com/ , it's a very feature dense UI for AI chat, and definitely is the best out of all of em (especially for that chatting aspect) at the expense of being almost too customizable, not being as single-click-intuitive as Faraday, and needing a separate program/source for the AI.
https://github.com/LostRuins/koboldcpp is what I use to run local models with sillytavern. You load your model and run it, and then connect to it via sillytavern. It's pretty simple and comes with a rudimentary UI as well which is more of a story-writing UI than chat, which could even be more useful for your purposes.
Lastly, if you'd ever be interested in chatting with characters by other people, https://chub.ai/ is a great place to look
Also, maybe you noticed that the .gguf models come in different variations, here's a nice chart explaining the difference. It's really just lower size = faster but slightly worse
opaque flame
Waiting on some of that to download. So with Hermes, I was trying to generate a list of what I defined as "raid bosses that are dragons, each one based on one of the seven deadly sins", and every time it would repeat envy and pride, and omit lust and gluttony specifically
It's also in the exact same order every time.
Man, trying to install SillyTavern, I don't understand all this "git" stuff
