ad_discordbot (Fork of Fork of xNul's bot)

[terse folio]

if you need to include them asyncronously in your bot

[halcyon quarry]

derp

[terse folio]

at the top of the file, asyncio.Semaphore(1)
is used kind of like the queue, but only allows one process to happen at a time.
It's like a lock, but for async

It's split into a few categories, like only 1 text generation may run at a time,
only one load/unload model task.
only one embedding... and so on

It achieves the same as the queue, but returns results to the function that called it on completion.
This is ideal for a webserver to send back the correct data to a client

Your bot might benefit from it too,
instead of having a background task consuming the queue and passing the channel and all that data down

[halcyon quarry]

eh, it's not all that much data.
At first, I was matching Tags and collecting huge dictionaries and queuing it all lol

[terse folio]

Mhm mhm, just thought designs for the future!

I just learned about that asyncio feature quite recently, finding it useful in place of some old code that tried achieving the same idea

[halcyon quarry]

I have the LLM model loader updated now

[terse folio]

Awesome!

[halcyon quarry]

yes the model loaded pretty slow (14 seconds) without heartbeat issue, which I think occurs every 10 seconds

@terse folio heres that screenshot I said I'd share

[terse folio]

Forgot what the context for this was now

[halcyon quarry]

Some new optional params I want to introduce per-character

[terse folio]

Oh something to do with triggering messages/when to reply, I see, that's nice too

[terse folio]

sounds doable ^^

[halcyon quarry]

although... I had adjusted this... maybe I forgot to save it

bah

ok I rmember now I was tinkering with it at work, left my progress there

axed a few params for a simple "responsiveness" setting that is going to determine how likely it is to use the maximum range.

1.0 = computer, 0.0 = distracted human

[terse folio]

Can also have the responsiveness tied to time

Like if you are actively in a conversation it could be extra responsive

[halcyon quarry]

it will be ties to time for sure

[terse folio]

because it's right there with you in conversation!

[halcyon quarry]

ah

yeah didn't consider that part

[terse folio]

But could act more distracted if you send a message once an hour

[terse folio]

It's experimental but I made a branch,
was interesting to find out about flows, is this your way of chaining tasks together like generating an image while sending messages?
That's cool

[halcyon quarry]

It's super fuckin cool

I'm dying to see someone with a big brain do something fantastical with it

One of the extra features it has, is that the variables I made can be used for tag values in Flows

[terse folio]

I imagine something similar to nodes, like comfy ui?

[halcyon quarry]

I don't want to process all tag values looking for variables - I only do it for Flows

You may have seen this already...
https://private-user-images.githubusercontent.com/1613484/325287074-0bc64090-01e8-477b-9da9-098057b06d90.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTU0ODI3MTgsIm5iZiI6MTcxNTQ4MjQxOCwicGF0aCI6Ii8xNjEzNDg0LzMyNTI4NzA3NC0wYmM2NDA5MC0wMWU4LTQ3N2ItOWRhOS0wOTgwNTdiMDZkOTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDUxMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA1MTJUMDI1MzM4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Yzk3MzkxMTI4NzVlY2YwMmUyMDdlNWYzMjAzNGIxOTY5ZTBmNDY0NjU2NzkyOGY2YmM2YzkzODBkZjQzMmEwZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.BUepjZXNKKmngVLnp3bx_lJaIaxL2WEvxVSO87_b3dc

lol huge image URL

mainly just see the dotted line

[terse folio]

oh gosh

lots of fun making flowcharts ^^

[halcyon quarry]

I made it look more intimidating than it is... once you get the concept entirely, it is very simple

There are 2 match phases for Tags. one before LLM gen and one before Img gen.
All tags are collected, then sorted to go into those match phases.
Once matched, the custom stuff for LLM gen is applied beforehand, and the same for img gen

A "Flow" is basically a package of matched tags, that do whatever you want.
The system is so flexible you can do pretty much anything imaginable on each pass through, and using the output as input later on

[terse folio]

mhm, reminds me of some node based code!

[halcyon quarry]

Something I need to add (another thing on my to-do list) - is a means for user to assign their own variables

[terse folio]

sending outputs to other nodes with their own settings

[halcyon quarry]

Just the other day I made it so "last_image" can be used as an input for any of the Tags that accept an image - img2img, controlnet, reactor, etc

So for example you could have a flow with txt2img, then next step is img2img

[terse folio]

Nice!

this is something i've wanted to do for a while

[halcyon quarry]

made an issue with A1111 that they should really have a resize_by param in their img2img endpoint https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/15758

[terse folio]

since it's a 1 dimensional slider, you could just multiply and int() the values yourself

not sure if they would have to be clamped to a multiple of 16 though

[halcyon quarry]

The bot stores the most recent 10 messages as variables. So you could trigger a flow that asks one character context something, then use that output for another character context, etc etc

[terse folio]

^^

[halcyon quarry]

This is starting to get old I'm sure but "On my to-do list is" to add 2 custom params for img gen: resize_by and aspect_ratio

[terse folio]

I have an entire discord server dedicated to my infinite todo list :P

I sometimes take the weekend to scroll through it and see what i've done by cosincience, and what needs to be bumped to the top

yea, just more and more ideas

[halcyon quarry]

welp, character specific embed colors are being more trouble than worth.
Moving along

the discord embeds don't want to be updated after initialized

at least not the colors

[terse folio]

yea, it's normal to create a new embed per message

Some classes use hidden attributes you can't set externally

In the future, I could create a class for the character.
On load, the class is initialised with the file name of the character file.
The __init__ code would load the file and assign the variables.

Then you just pass the class around to any part that needs it.
char.embed_color
char.name
char.pfp...

This would also let you do fast switching if you store them in a dict
dict(character_name=class)

[halcyon quarry]

Well that's currently lumped in with all the other settings, all under class BotSettings()

[terse folio]

uhoh

I'm guessing my model name is too long,
could slap a name[:100] on the end of that

[halcyon quarry]

ahhhhh

hum

[terse folio]

line ~4280

possibly

[halcyon quarry]

Well I need to do 2 things:

1. truncate the display values but not the actual model values
2. When matching the result for the complete model dict need to also truncate the searched values

because thats the way the discord menus work, they dont let you stick a dict value to each name, need to fetch it after

[terse folio]

hahaha, whoops, editing code in the repo and wondering why the bot keeps failing.
forgot to update the file on the bot

[halcyon quarry]

Thanks for the bug report though, handling that now

[terse folio]

since the full model name can't fit in the choice value.
maybe hashing it would work.
writing that hash to a dict and querying that

hehe, not your fault

but stable diffusion never anticipated me using multiple {prompt} variables in my styles

in the webui I do that to bypass a part of the prompt for the first moments of generation

[{prompt}:high res... etc {prompt}:0.5]
for example
And I can apply this as a style to add "highres" after 50%

[halcyon quarry]

Well, I also have a {prompt} variable that could be interfering

see tips_and_info/Message Variables.txt

[terse folio]

I tested the SD api before with some other code, it complained as well, just was an interesting error :)

[halcyon quarry]

One of the most ridiculous features I added that I swear works as described, is in Advanced Usage for Img Extensions.txt

Not ridiculous to me, I'm using it for automatic inpainting with multi-controlnet

[terse folio]

Sounds nice ^^

also I'm experiencing that discord gateway crashing on model load haha

but it recovered! I think

[halcyon quarry]

but what's crazy about what I implemented is that you can set a directory name for all the input values - this will select a random subdirectory if no image is found, until it finds an image

When this method is used, it captures that randomly selected subdirectory and will attempt to find a specific image name such as canny.png for the canny model or img2img_mask for an inpainting mask

[terse folio]

Wonder what the purpose of using random CN images for would be?

wouldn't you have to match prompt to certain CN inputs?

[halcyon quarry]

For my work, I have product images (lifestyle images).
I can have a bunch of subdirectories with the same controlnet input types but different poses, with different products

I can use the same "Tag" definition to randomly select a subdirectory and it will use all the images as input

[terse folio]

I see I see, makes sense

[halcyon quarry]

Very advanced usage, but it definitely works as described. Need to basically figure out the images and settings via the WebUI for consistent results before using it like this

almost got this model thing resolved...

updating all the menus at once including LLM models

and characters.

[terse folio]

Little random side tangent if you're bored,
But I replaced the task_queue with an asyncio semaphore.
It seems to work perfectly.

But code is a little more organised, functions being in the same commands requesting them.
https://github.com/Artificiangel/ad_discordbot/tree/semaphore-idea

maybe create a test branch, see how that works.
I tested model loading, texting, image generation, character swapping.
With and without other tasks running

All the warnings "your item has been added to queue" work!

[halcyon quarry]

most important part!

I'll check it out

[terse folio]

A lot more readable 😸

[halcyon quarry]

In dict_tags.yaml are a few example "Flows"

about to push update for the menu

[terse folio]

That's cool, "check imgmodel" made the most sense to me

[halcyon quarry]

If I was better at grammar I'd probably come up with more cool uses

Takes too much guess and check for me

[terse folio]

about the "generate an image" text matching thing

I had some code for that

I think it relied a bit on embedding

[halcyon quarry]

The error I kept getting was 'expecting end' as in I think it needed a more definitive... definition for what I wanted to capture beyond 'everything' after the word prompt

Here's a pretty simple tag I added today for my bot

  - trigger: 'reimagine'
    format_prompt: 'Tell me an alternate version of this image prompt: {llm_0}'
    on_prefix_only: true
    search_mode: user
    should_send_text: true
    should_gen_image: true

[terse folio]

... oh

if match:=re.search(r'(make|send|draw|create|render|take|sketch|photograph|shoot|photoshoot|show)([\w\s]{0,}?) (a|that|this)?([\w\s]{0,}?) (pic[\w]{0,10}|pix[\w]{0,10}|pho[\w]{0,10}|potos?|fotos?|drawings?|renders?|sketche?s?|im[\w]{0,5}|imgs?)($|\s)', msg_filtered):

tried to account for everything!

Feel free to use

XD

[halcyon quarry]

Also this...

  - trigger: 'followupreimagine,reimagine'                   # Intended to be used with a prompt like 'draw _______ check imgmodel'
    insert_text: ''
    insert_text_method: 'replace'
    search_mode: user                           # Only search user's prompt (will not trigger from LLM reply)
    should_send_text: true                     # Suppress text response (still generate text)
    should_gen_image: false                     # Suppress image generation (won't send an image by default)
    flow:                                       # The actual 'Flow' tag value
      - flow_base:                              # Flow_base are tags that apply to every flow_step
          save_history: False                   # Do not write these interactions to the main chat history
          load_history: 1                       # 0 = default (all history), -1 = excludes chat history, > 1 = LLM sees this many recent exchanges.
      - flow_step: Ask LLM to reimagine the prompt # text displayed in discord embed
        format_prompt: 'Tell me an alternate version of this image prompt: {llm_0}'
        should_gen_image: true

First prompt is from M1nty-SDXL (the one that writes image prompts).

That is given to the chat character with the instruction

the chat character seems to do well with picking that up and running with it

[terse folio]

it's a regex that matches a few patterns such as

[halcyon quarry]

I have something similar, not quite that intense

[terse folio]

and what's the 2nd from?
the prompt iterated on again?

[halcyon quarry]

- trigger: '{no|without|clear|empty|transparent} {bg|background},partially trans'

woops - the example I showed you was different tag... I revised it

[terse folio]

Oh, i see, it is "tell me an alternate prompt"

[halcyon quarry]

I had set it to use the {last_image} as controlnet input - using it as depth

[terse folio]

Wonder if this would make it easier to read:

  - trigger: 'followupreimagine,reimagine'                   # Intended to be used with a prompt like 'draw _______ check imgmodel'
    insert_text: ''
    insert_text_method: 'replace'
    search_mode: user                           # Only search user's prompt (will not trigger from LLM reply)
    should_send_text: true                     # Suppress text response (still generate text)
    should_gen_image: false                     # Suppress image generation (won't send an image by default)
    flow:                                       # The actual 'Flow' tag value
      - flow_base:                              # Flow_base are tags that apply to every flow_step
          save_history: False                   # Do not write these interactions to the main chat history
          load_history: 1                       # 0 = default (all history), -1 = excludes chat history, > 1 = LLM sees this many recent exchanges.
      - flow_step: Ask LLM to reimagine the prompt # text displayed in discord embed
        format_prompt: 'Tell me an alternate version of this image prompt: {llm_0}'
        should_gen_image: true

^^

Great

[halcyon quarry]

yaml markdown highlighting right

[terse folio]

yml,

discord doesn't recognize yaml

[halcyon quarry]

Another one I was working on, I got sidetracked at setting the width/height

  - trigger: 'upscale'
    format_prompt: '{llm_0}'
    img2img: '{last_image}'
    on_prefix_only: true
    search_mode: user
    should_gen_text: false
    should_gen_image: true

so after it draws something you just say upscale, and it will return an upscaled version

[terse folio]

mhm mhm

[halcyon quarry]

Hoping you come up with some crazy workflow 😛

Test out the loopback flow I included

damn its late already

[terse folio]

O: indeed

If flows need any more customisation:
It's possible to write python code as a string and execute it :P
Do anything!

[halcyon quarry]

The way I have this stuff set up, adding anything is easy

Very easy to slip new features into the tags system

[terse folio]

that's always good

[halcyon quarry]

Could you try something for me?

I screwed up and now I have to wait 9 minutes to test again lol

[terse folio]

changed a pfp?

[halcyon quarry]

try this function...

async def delayed_profile_update(username, avatar, remaining_cooldown):
    try:
        await asyncio.sleep(remaining_cooldown)
        if username or avatar:
            print("client.guilds", client.guilds)
            for guild in client.guilds:
                print("guild", guild)
                client_member = guild.get_member(client.user.id)
                print("client.user.id", client.user.id)
                print("client_member", client_member)
                if username:
                    await client_member.edit(nick=username)
                if avatar:
                    await client_member.edit(guild_avatar=avatar)
        logging.info(f"Updated discord client profile (username/avatar). Profile can be updated again in 10 minutes.")
        update_last_time('last_change')  # Store the current datetime in bot.db
    except Exception as e:
        logging.error(f"Error while changing character username or avatar: {e}")

[terse folio]

I'm going to create a new bot account before I do some thing silly

haha

[halcyon quarry]

I didn't get an error, but its because the character I tried changing to did not have an image 😛

The nickname edit works

eh Ill go brush and floss then see if it errors by the time Im done

[terse folio]

it updates globa- oh wait
used the wrong bot file again

well that's 8 minutes to wait now

[halcyon quarry]

Cancel it b/c guild_avatar doesnt work

will just be avatar

[terse folio]

mhm, not everything is setable, there should be a list of what kwargs are available in the docs,
somewhere down the chain of classes

[halcyon quarry]

this is how it'll be

async def delayed_profile_update(username, avatar, remaining_cooldown):
    try:
        await asyncio.sleep(remaining_cooldown)
        if username:
            for guild in client.guilds:
                client_member = guild.get_member(client.user.id)
                await client_member.edit(nick=username)
        if avatar:
            await client.user.edit(avatar=avatar)
        logging.info(f"Updated discord client profile (username/avatar). Profile can be updated again in 10 minutes.")
        update_last_time('last_change')  # Store the current datetime in bot.db
    except Exception as e:
        logging.error(f"Error while changing character username or avatar: {e}")

keeping the original avatar edit, but now using the nick change instead of username change

[terse folio]

oh, it really is just edit(*, username=..., avatar=...)

[terse folio]

Yea, one less thing to be ratelimited on.
username changes are more expensive than nicknames ^^

[halcyon quarry]

May need to check time for avatar / nick separately then

nah too much work...

I think avatar is 10 mins tho

[terse folio]

I don't know what it is now days, because discord changed how usernames work

but it used to be 2 times per hour* for username changes?

I just remember it was pretty big

oh, twice per hour

^^

[halcyon quarry]

Try the most recent commit with the imgmodel name truncated

It should be more simple than I said...
The value didn't have to be changed I don't think (separate)

I only sliced at [:100] for name

[terse folio]

I played it safe and changed both,
didn't actually test without value changed

Will have to tomorrow, already shutdown SD and all that

[halcyon quarry]

Ill check out that branch you made

How can I compare it to my current main?

[terse folio]

I can open a pull request

you could also click create new branch

and pull from my fork:semaphore-idea

[halcyon quarry]

I'm still pretty wet behind the ears with git

[terse folio]

same with github

[halcyon quarry]

github

[terse folio]

am new too, looking up how to do seemingly simple things :)

actually no

because my fork is "downstream" of yours

i'll just open a PR

[halcyon quarry]

PR it to my semaphor branch 😄

[terse folio]

making sure that your last commit doesn't break something

[halcyon quarry]

hopefully not - added the loop run in executor for load_model()
And sliced the names for imgmodels llmmodels and characters

[terse folio]

opened it

[terse folio]

just was taking a look in vscode if there was a "missing variable"
because I removed the task_queue

[halcyon quarry]

that's interesting...

I'll definitely have to test this a bit, this looks very clean

Hey, thanks for the interest in this project, this is very nice stuff you contributed

[terse folio]

I'm glad!

[halcyon quarry]

Semaphores method is all working - that’s very cool. I discussed it with ChatGPT and still can’t quite understand when one method is preferred. In this case seems moot but cleaner. Only thing missing is async with typing for some functions

I pushed an update for the discord profile thing

[terse folio]

Happy it works properly!

[halcyon quarry]

erm, you did add the typing 😛 my bad again lol

[halcyon quarry]

Pushed Reality's clean replacement of the Task Queue

[marsh harness]

@halcyon quarry Would you mind putting a chat-instruct example character/template on the repo in example characters?

[halcyon quarry]

Personally, I never had good results using chat-instruct.
I believe all you want to do is include a "chat-instruct_command" in your character's state

Copy/paste the one from base_settings.yaml and tweak it

[marsh harness]

Hm. Alright.

Thanks.

[halcyon quarry]

Oh and of course, also character's mode needs to be chat-instruct

[halcyon quarry]

There's some new Controlnet Inpainting models on Civitai - enjoying these away too much

https://civitai.com/models/136070?modelVersionId=497710
https://civitai.com/models/136070?modelVersionId=496890

[terse folio]

ooh nice, was just about to ask if it was for SdXL

The last good inpainting model I used was for sd1.5

[halcyon quarry]

These work a bit differently, a bit clunky to use but the results are very good

I made a comment on that civitai page

[terse folio]

Will check them out eventually, sounds great

[halcyon quarry]

Using both of them together in txt2img tab is insanely effective

[halcyon quarry]

@terse folio Sent an invite for you to be a collaborator. If you have your own ideas to plug in, or just want to adjust anything as you've already done, I'll ensure you have the ease to make changes without my bumbling 😛

As for me, I'm working on those Behaviors changes tomorrow - Stable Diffusion can wait

[terse folio]

Sure, after getting some work done on personal projects I'll look into creating an api endpoint for all the tts stuff ^^

Small issue there is I need to figure out how to get xtts installed. #windows-setup message
I often have this issue with building wheels for python packages.
Something wrong with my environment

[halcyon quarry]

Tried alltalk_tts?

it uses the xtts model

[terse folio]

it's the TTS module that has issues installing

[halcyon quarry]

You should give it a shot

https://github.com/erew123/alltalk_tts/#-quick-setup-text-generation-webui--standalone-installation

[terse folio]

Yea docker would probably work

[keen palm]

Mi scusi, is there a way to see the full context being used by the LLM in responses?

[terse folio]

A general solution would be to enable verbose mode in textgen-webui settings (Last tab)

This will display the prompt as you generate text in the command prompt.

[halcyon quarry]

I think it will also display via the bot if you launch with —verbose or put it in CMD_FLAGS.txt

(But maybe not)

[keen palm]

Hmm

Putting verbose in textgen doesn't show any of the context for the bot

[halcyon quarry]

If it’s showing everything else then… you’re using chat mode, yes?

[terse folio]

Make sure to click apply and refresh

[keen palm]

I did. And yes, chat mode

[halcyon quarry]

It should show the context…

[keen palm]

It shows the prompt and the response, as well as the context number, but I'm trying to see the entire context

[halcyon quarry]

With verbose it should show the full context (which is part of your prompt)

If the context isn’t showing it may not actually be loading in…

[keen palm]

It seems to be drawing it from somewhere, because prompting the bot in one channel had it bring in context that was previously established in another channel

[halcyon quarry]

Well chat history is different from the character’ context

Edit you character file and at the end put {{char}}: My favorite color is blue!
Reload and ask what its favorite color is 😛

Or just ask something already in there

Should be a good indicator

[keen palm]

The character context is loading just fine. I've already confirmed that.
I'm specifically wondering about the chat history, and what it's actually using.

[halcyon quarry]

Ah. It uses as much as can fit.

Your character context and max new tokens and prompt size all take tokens, whatever’s left will be chat history

If you reduce max tokens param it will fit more chat history

Or increase truncation length

Or use less context

[keen palm]

Yeah, I'm just trying to see that, to verify what all is being included. Setting verbose on lets me see all that in the webui, but it's not showing for the bot.

What are these history (internal and visible) settings for in dict_base_settings?

[halcyon quarry]

Those are ignored

[keen palm]

Gotcha

[halcyon quarry]

If you want you could open bot.py Ctrl+F to def llm_gen and somewhere put
print(“payload:” llm_payload)

or llm_payload[“state”][“history”]

but yeah… this won’t confirm what is being used in your prompt by TGWUI. This will show the entire history including stuff that will be truncated

There’s some interesting changes coming soon that could make the bot feel more alive in a busy server

[keen palm]

Ooh, excellent! I just started trying this one today. I had been using oobabot_plugin a while back, but it seems to not work currently.

[halcyon quarry]

There’s all sorts of fun stuff you can dream up with the Tags system

[keen palm]

I haven't even begun to look into that yet

Is the history erased when the bot is reloaded?

[halcyon quarry]

There’s a bunch of things you can trigger via trigger phrases, including text replacement/editing, alternate character context being swapped in, model changes, etc

Yep, it is

If you have any feature suggestions I’m all ears. History seems to change pretty quickly in a chat server

At least no one’s complained yet. If it’s important to you I’ll see what I can do

[keen palm]

Gotcha. I had in mind to use it as a gamemaster assistant bot for an ongoing roleplaying game, so it would need to draw upon previous history for that. It wouldn't be used as part of just a chat stream.
If that's not feasible, I'd understand.

Being able to draw from lorebooks, like with Novel AI, would be great, but I imagine fairly difficult. There's an extension for the webui that allows for that, but it doesn't really work with bots.

[halcyon quarry]

I never tried superbooga or anything like that

[keen palm]

I couldn't get that one to work. There's one called complex_memory that just draws from lorebooks that you create, so you can store important character/setting information in them.

[halcyon quarry]

You could make various character files that are knowledgeable about certain things, and make trigger phrases to activate the appropriate character

see dict_tags.yaml
See an example like the one I have with trigger “draw”

It swaps in that character, hides chat history, does not save history, that character will give good image prompts. Something similar can be done for any specialized question/use case

[keen palm]

Gotcha. I'll take a look through that. Thanks!

[halcyon quarry]

It’s very easy once you get it - very powerful. Lot’s of comments 😛 don’t be too scared

About hsitory, maybe I’ll add a command to manually save/load chat history

[keen palm]

That could work!

[keen palm]

Looks like there's a bit of an issue with the Regenerate and Continue options. If I first regenerate a response, and then ask it to continue that response, the continuation doesn't work off of that response.

[halcyon quarry]

I'll look into that. Good news for you, though, I have history management coming very shortly

[keen palm]

Ooh sweet! Thanks!

[halcyon quarry]

No, thank you for the motivation 😛 It's one of those things I knew should be there but just avoided handling it

[keen palm]

Sometimes you just need a little push

[halcyon quarry]

There will be a few new config options. It's a very simple implementation that gets the job done well IMO

[keen palm]

I assume when you save the history it would be placed into a file in the bot folder?

[halcyon quarry]

I'm importing a few functions and using the native TGWUI history management - so the chats will load/save from native /logs/ folder

new config options:

  chat_history:
    limit_history: true                 # Recommended to keep as true. This will limit the stored history to roughly the current 'truncation_length' (more than enough retained). You may have performance issues if set to false.
    autosave_history: true             # Set this value to true to continuously save history. The '/history' command allows saving/loading history manually.
    autoload_history: true             # Set to true if you want the character to load with the most recently saved chat history.

When the bot loads, if autoload_history is true it will load the most recent chat, and print the most recent exchange in CMD for reference

[keen palm]

Oh very nice

[halcyon quarry]

If autosave_history is true it will save every exchange to the most recent saved history

[keen palm]

Oh, is reloading the bot required to refresh the model list for /llmmodel ?

[halcyon quarry]

There will be a new cmd /save_conversation Which will save to a new file, and the chat will continue in that new file

[halcyon quarry]

For now, yes. I know how to fix that, just need to get to it

[keen palm]

Cool cool. Just checking, since I'm screwing around with different models

[halcyon quarry]

After I push this history update, the Behavior update I've been talking about is next. Then I'm fixing the /imgmodel, /llmmodel and /character commands so they refresh each time

In case you didn't see my comment somewhere about LLM models (I forget where it is), if you use the Save Settings in TGWUI (which saves your model specific settings to config-user.yaml), those settings will be applied when doing model changes

[keen palm]

Yes, I caught that, so I've been making sure to save the settings for new models before loading them into the bot

I haven't started messing around with incorporating A1111, so I didn't see if this has been addressed. Is there a way to dynamically unload an LLM and then load an SD model when an image request is created?

[halcyon quarry]

There currently is not a way to unload Image model - there is an API endpoint that should do it, but it seems to be broken in A1111 and Forge.
As for LLM Model, if you use change_llmmodel or swap_llmmodel as "None" it will unload the LLM model

So yeah, for example you could edit the "draw" Tag that comes by default and just add the 'swap_llmmodel' tag in there

Or you could make another tag with a trigger like "draw mode" that has trump param "draw" (this won't trump itself), that just unloads the LLM model (rather than Swapping)
You could make another like "chat mode" that loads a specific model

[keen palm]

Gotcha. That might work, though I'm looking for an LLM I can run on one GPU while the other handles SD tasks.

[halcyon quarry]

I think just using the appropriate SD API URL in config.py should allow that

erm

there may be some kind of flag you can set for that in your SD Webui

[keen palm]

Yes, that part is handled. Now the task is finding an acceptable LLM that fits on one GPU.

[halcyon quarry]

Shouldn't be hard... good things come in small packages these days

[keen palm]

Teehee

Actually, it seems the llmodel list isn't updating even after restarting the bot. I've even deleted a model entirely, but it's still on the list.

[halcyon quarry]

Again, I think you need to close and open the discord client

That will also be resolved when I fix those commands... I think

[keen palm]

Oh gotcha

[halcyon quarry]

I think the 'Continue' command is working, except it is just saying "Regenerated text" every time

The Continue command actually replaces the entire text with the continued text

but maybe I can revise it to just add the continued text...

[keen palm]

Right, and usually that works. The issue I noticed was that a regeneration request hit the limit, but when I asked it to continue on that, it continued on the original output.

[halcyon quarry]

Do you know if that also happens in the WebUI?

[keen palm]

Let me check

Looks like it continues correctly off the regenerated response

[halcyon quarry]

bah

Alright, I'll look into it 😛

[keen palm]

Thanks!

[halcyon quarry]

@terse folio I'm rolling back that client username thing - it's just too convenient managing one name than the possibility of it being different in different guilds

Just noticed I'm failing to check the name correctly in a few places, and yeah, it's going to bug me 😛

[halcyon quarry]

nvm... what I need to do is instead save current character name to a database value

and stop checking things based on discord profile name

[viral lagoon]

[halcyon quarry]

Try closing and opening your client

I had the same issue

There's some issues with profile changes atm - I'm working to fix that now... should be done in the next 30 mins

[viral lagoon]

Oh, it's /reset_character. Ctrl+F5 seems to do it

[halcyon quarry]

😄

[viral lagoon]

Please tell me updates are generally doable? I hadn't updated since December so I need to redo all my configs.

[halcyon quarry]

If its really been that long then, deally, you'll just move all your current settings somewhere to the side and just install the bot again, then update user preferences.

A lot of shit has changed 😛
If you just replace bot.py everything should still work... I've been coding things with backwards compatibility

[viral lagoon]

Well, I'm still on config.py fallback or something, seems to work, kind of.

[halcyon quarry]

Yeah, the bot should still run even if you are using old config.py

[viral lagoon]

It forgot about the main channel though..

[halcyon quarry]

Maybe your bot.db was modified

I have a command now /main that can quickly toggle channels as main or not

[viral lagoon]

Hmm, its working now, so...

yeah

Can I turn "mention > auto-/main" off ?

Weird, now it doesn't do that anymore...

Like, I @ the bot, then it says
Added ⁠uwu to main channels. Use "/main" again to remove it.

[halcyon quarry]

Maybe I changed it

I'll modify that...

I'll just remove that, ts dumb

If you cant wait its 3 lines into def on_message()

just delete this line
if not bot_settings.database.main_channels and client.user.mentioned_in(i): await main(i) # if None, set channel as main

[viral lagoon]

So, it's normal that it will reply in any channel when mentioned? (I guess it makes sense, but, you know...)

[halcyon quarry]

You can change that using the Behavior setting (if undefined in your character file, will default to base_settings.yaml

[viral lagoon]

Oh, right. Hmmm, maybe it's not picking that up, since I'm sure I had it set to only reply in main channel

Hmm, what setting is that supposed to be? I have this

[halcyon quarry]

Try only_speak_when_spoken_to False

If not, then I'm talking outta my ass and it can't be stopped 😛

Well, could be stopped if I add something

[viral lagoon]

That setting has a double meaning if it does what I'm asking

Lemme try it..

🥲

Guess it's fine though, it won't reply without a mention.

One way to make it work it to have the bot access to one channel only.

But that's a little hacky to setup up on the discord end.

[halcyon quarry]

I have new behavior changes coming soon, like in the next 2-5 days

[viral lagoon]

Cool beans.

[keen palm]

Are there settings that need to be done in SD for the bot to interface with it? I'm getting an error where it fails to connect to the SD API, though it has the correct URL.

[halcyon quarry]

Port may be wrong

OH - sorry, you also need to ensure you use flags --api --listen

I think the error message should mention that...

Ok I have a much more robust method for tracking the current character name now, which is independent of the discord profile name

[keen palm]

Yeah, it looks like adding --api --listen did it

[halcyon quarry]

I'll be sure to push that history update momentarily... I think I squished all bugs...

[keen palm]

FYI:

discord.ext.commands.errors.HybridCommandError: Hybrid command raised an error: Command 'helpmenu' raised an exception: NameError: name 'system_embed_info_json' is not defined

[halcyon quarry]

I fixed that now too 😛

Encountered that bug about 20 mins ago

I'm glad to know that bug hasn't been lurking for months - I only introduced it just 1 week ago

[keen palm]

https://tenor.com/view/thats-good-news-obi-wan-star-wars-gif-15974779

[halcyon quarry]

Alright, everything looks pretty good to me right now, pushing a nice little update

Pushed the update!

Added preliminary support for History management.  May expand on this in the future.

Bug fixes:
- the Welcome message and /helmmenu commands were failing. Fixed.
- Discord username was being used to check current character, but was not always accurate. Replaced this method by storing current character name to the local database.

I'll look into that continue / regen thing next

[keen palm]

Alright, time to break it test everything!

[halcyon quarry]

Since I'm importing the TGWUI functions, they're working quite reliably.

I'm quite happy with the history loading

[keen palm]

Let's see!

[halcyon quarry]

The 'limit history' feature is something I've also been meaning to add for awhile.
Without it, the stored session_history variable was becoming infinitely long even though TGWUI only uses so much of it

[keen palm]

I'd say it's probably good practice to avoid anything trending toward infinity

With autosave_history set to true, it saves after every generation?

[halcyon quarry]

Yes

Not a new file - it saves to the current chat history (the most recent file)

A new file is saved if you use the /save_conversation command

[terse folio]

I usually trim the original history list while the chat prompt is being created.
I haven't come across a case where I needed to go back in time further than the context limit yet.
(that stuff should be sent to a vector DB later if it's important)

[keen palm]

The helpmenu command still failing, BTW

[halcyon quarry]

:flip-table:

It's working for me

[viral lagoon]

Also, 'change_username_with_character': False isn't honored (it changed the nickname)

[halcyon quarry]

Dead setting 😎 Update my friend

[viral lagoon]

🫠

[halcyon quarry]

Since I added the 'delayed profile update task' it's now guaranteed the profile will update.... eventually. So axed that setting

[keen palm]

Might be because I didn't start the updated bot.py file. Whoppsy!

[halcyon quarry]

It's nice to have active users again - there was a good 1-3 month stretch it was mostly me just talking to myself

[viral lagoon]

I might move to mcmonkey's bot eventually, because I'm moving away from A1111 (to StableSwarmUI), also it's written in C# which is a bit nicer imho.

[halcyon quarry]

Gah - found a bug. /reset is not starting a new conversation if autoload_history is enabled

[keen palm]

So yes, helpmenu is working now. I saw in the cmd output that a previous conversation (from the webui) was loaded, but the /save_conversation command isn't actually registering with the bot

[halcyon quarry]

Need to close/open the client 🤗

[keen palm]

(╯°□°)╯︵ ┻━┻

[halcyon quarry]

nooooooo

[viral lagoon]

So, disabling username change isn't possible anymore? I want my bot to have one name, regardless of the character.

[viral lagoon]

Swarm is nice bro, seriously.

[keen palm]

Okay, so /save_conversation command pops up now, yay. But I'm not seeing /load_conversation

[halcyon quarry]

Oh it's possible. All you need to do is use swap_character: Your character tag for damn near everything 😂

[halcyon quarry]

It's b/c you can't 🤓

On the to-do list now 🍺

[keen palm]

Ohhhh

I thought you had already implemented that. Carry on then!

[halcyon quarry]

It will always load the most recent conversation when switching character or booting up

Now just need to fix the reset cmd...

think I got it...

[keen palm]

This is good, since I can manually scrub the chat file if needed

[halcyon quarry]

yep - and if you want to load an old conversation for now you can just manually change the name to current time 😛

[keen palm]

Woah woah woah. That's just...way too complicated!

Alright, history is working great! At least until I inevitably overwhelm it!

But I can always see about having the bot create a plot synopsis based on previous history, then use that synopsis to help create new history.

[halcyon quarry]

Appologies guys for the faulty commit -

Just pushed update that fixed /reset_conversation

[keen palm]

That's /reset_character, right?

[halcyon quarry]

erm yep

I suppose I should rename that 😛

[keen palm]

Well at least it works!

[keen palm]

Hrmm, maybe not. It stopped saving anything new to the chat history....

I can see that the modified time on the log file is changing, but no new information is being saved.

[halcyon quarry]

Do you have a lot of history? Maybe I fudged up the limit history thing…

[keen palm]

It's only 15 segments. 18KB file size

I haven't messed with truncation length, so that's 2048

[halcyon quarry]

Does a new chat start saving again?

It’s probably an issue with that limit history… I should be able to fix it tonight.

[keen palm]

Yes, a new chat starts saving

[halcyon quarry]

I probably messed up the bit that prunes the old history…

[keen palm]

I put back in the old chat, turned off the history limit, and it's saving now

[halcyon quarry]

Yep that confirms it - I just need to consult mr ChatGPT and see what I did wrong

[keen palm]

ROFL

[halcyon quarry]

I did most of the work today without any consultation

[terse folio]

chatgpt is trying to sabotage it's own replacement! :P

[halcyon quarry]

😮‍💨

[halcyon quarry]

Could you do me a favor and test something for me?

Since you have a stuffed full chatlog...

[keen palm]

It's not that stuffed, but sure

[halcyon quarry]

Jump down to the very bottom of bot.py, lines 5094 and 5096

        if self.truncation:
            while sum(len(message) for exchange in self.session_history['internal'] for message in exchange) > self.truncation:
                oldest_exchange = self.session_history['internal'].pop(0)
            while sum(len(message) for exchange in self.session_history['visible'] for message in exchange) > self.truncation:
                oldest_exchange = self.session_history['visible'].pop(0)

Where it currently says '.pop()' just stick a 0 in there

I think that resolves it

[keen palm]

And turn history limit back on I presume

[halcyon quarry]

yepppp

did you know that 2048 is a comically low context length these days? The models you are using are surely 4096 >

If your message history is very precious, then the setting limit_history: false is probably for you

[keen palm]

Yeah, I definitely need to change that limit

But what are the high context LLMs these days? I've been gone for a while, and the SuperHOT 8K ones had just come out

[halcyon quarry]

Context does use more Vram, so personally I'm happy with 4096. It should say on each model's page

[keen palm]

Right. I have 2 3060s, so if I spread it across the entire VRAM pool, I have a fair bit to use.

Looks like it's saving again

[halcyon quarry]

Pushed an update!

Limit history was actually removing the most recent messages instead of the oldest ones :)  Slight oopsies!

[keen palm]

https://tenor.com/view/oops-aquaman-bro-gif-18498968

Was there anything else you did beside pop(0)?

[halcyon quarry]

Nope! That’s it

[keen palm]

When I load a model that has truncation length of 4096, as saved from webui, the activesettings.yaml file still shows the length at 2048. Should it be showing 4096?

[terse folio]

That's an interesting bug, causing the bot to have bad short term memory but for some reason remembering long things

[keen palm]

Changing the chat_prompt_Size and truncation_length in dict_Base_settings doesn't seem to be affecting the active_settings file

[halcyon quarry]

I’m sorry but you just need to change something such as /character

To same character

I’ll fix that…

[keen palm]

Ohh okay. I manually did it in active settings, but I figure that's not a long-term solution.

[halcyon quarry]

The settings handling is very good, but slightly imperfect. Activesettings used to be read for every interaction, but now it is only read during startup and for a few select things.

@viral lagoon just wondering if you've been keeping up at all with the new features I've been adding?

The whole Tags system, Flows, ControlNet/layerdiffuse support, etc

[keen palm]

Let's see if I can make use of this 32k context model ...

[halcyon quarry]

making some progress on reworking settings...

[terse folio]

Is the starboard feature only for images?

Seems to only have an if statement for embeds/attachments

[terse folio]

I replaced the SQL db with a yaml file.
It wasn't necessary to store a single value in a table.
More complexity can lead to more bugs.

Anyway, added a migration that runs if bot.db is detected.

Also "last message time" and other time based variables that were in the bot.db are now stored as floats instead of strings.
Because there were a lot of datatime -> str -> datetime conversions

Everything works on my end, but make sure to run a test before merging since I can't run a few features like tts yet.

[terse folio]

if you need to add a private value to the database, give it a "_" prefix and it wont be saved to the yaml file

[halcyon quarry]

Why the heck have I been perpetuating this database file when clearly that’s the better option, lol

I don’t recommend changing much more for the moment because I spent an hour or so revising settings handling

I’ll put my changes to a dev branch for you to probably poke holes in

[terse folio]

The benefit of a SQL database is it doesn't have to save the whole file (I think) when you push a commit.
So if you have a massive database with 100k+ rows, it's pretty fast.
(also can use sql servers to offload)

But saving a 1GB json file on the other hand isn't fast if you are updating just one setting.

Since this bot is locally hosted by many users, they probably wont be running into having that much data any time soon so this is a good solution that saves time ^^

[terse folio]

Yea, started splitting things into dedicated files,
I imagine merging that would break some things as it is now

[halcyon quarry]

I think I / we could put the pieces together

When I last sat down and changed the main settings handling I didn’t understand Classes very well. I should that spruced up in the next few hours

If you want to cringe and have a little giggle, see BotSettings init

[terse folio]

I think it's pretty good, just setting up a class with other objects inside it

though some of them probably don't have to be classes.
Like how Settings() goes straight to dict() at every instance.

Also just noticed this

This would create bugs later on if you modified one setting in the behavior instance, but then read the variable from the other.

Since it's already defined in settings(), you could
self.behavior = self.defaults['behavior']
And that would share the same instance

[halcyon quarry]

This is already well in progress of being fixed 🤓

[terse folio]

Awesome

[halcyon quarry]

It’s working now because way up by on_ready() is update_bot_settings

I moving something similar down within the class

But yes there’s currently those variables sitting there for no reason heh

[terse folio]

Take a look at how I changed the database, it might be worth incorporating into more of the settings!

Just simple database.set(key, value)
and
value = database.key

[halcyon quarry]

Had the funny thought that the bot needs renaming with your contributions 😛 maybe too dramatic, but I also thought it’s kinda messed up for that other project to call itself “oobabot” as if it was made by the great oobabooga himself

[terse folio]

Fork of a fork of a fork of a fork

We could call it the fourk, because there's 4 of them!

[halcyon quarry]

Now we're talking

the logic of the settings handling is pretty good overall (I think)... LLM Model handling could be added to the settings at some point.

• There's a copy of all the bare bones settings in bot.py, which is just used with the fix_dict() function to fill in anything that is missing.
• base_settings.yaml presides over that
• character settings preside over that
• activesettings gets update when using /character and /imgmodel, which is used during init to pick up where the bot left off.

[terse folio]

Why not use the default settings from shared.py?

Only keeping a difference from tgwi

oh is it perhaps that tgwi overwrites the defaults in shared.py
so if you were to reload something it doesn't truly reset to the default state?

[keen palm]

As I massively increase truncation_length to handle higher contexts, should I also increase chat_prompt_size?

[terse folio]

I'm not sure that does anything?
I just searched the tgwi files, there's no usage of that attribute.

Maybe it gets imported under another name

odd

[keen palm]

Well, huh

It's set to 2048 just like trunclen, so I might as well just keep them the same

[halcyon quarry]

@terse folio I didn't reduce the value for the menu items because I don't think that is restricted to 100 characters only the name
Are you just trying to improve efficiency?

[terse folio]

It is restricted, I had errors when starting the bot

[halcyon quarry]

🤯

Again, very impressive updates from you, tyvm

[terse folio]

I think discord treats them the same on the backend.
This would make sense of course,
we're uploading data to discord to store, so they have to limit something.
And every client that interacts with the bot has to download that data

[halcyon quarry]

Merged.
Now I'll see if it broke anything because that's the correct order of ops 🙂

[terse folio]

Also found some entry points to making a TTS api.
The tts scripts hook into the text generator defining "output_modifier()"
which gets picked up on script load.

So I could write another script that imports the same function and passes custom strings to it from a webapi

[halcyon quarry]

This is something you would be opening a PR for upstream?

[terse folio]

Much later, I need to get tts working so I can test that the code works 😸

ah right, was going to test docker with alltalk

[halcyon quarry]

Just to confirm... the changes you made add files to the bot folder but still only bot.py in TGWUI root dir?

[terse folio]

yes, everything works the same

I added a line in bot.py that adds ad_discordbot as a path to import from

so you can do import database
isntead of import ad_discordbot.database

which makes it easier for VSCode to sort out dependancies while they are all together

[halcyon quarry]

Uh oh new dev 😄

Where is ze file Lebowski?

[terse folio]

oh my bad,
originally that code was .get()
which returns None if no attribute

but .pop() errors if no default is given

you could fix that by saying .pop('last_character', None)

probably

[halcyon quarry]

Does this create the new database yaml file during runtime?

[terse folio]

the databasev2 will be in the same dir as bot.db

[terse folio]

yes, after everything imports from the old one it will save the new and rename old file

[halcyon quarry]

Yes adding fallback fixes it

Pushed hotfix

I like it

[terse folio]

So what's going on there is
you can define default values in load()

or let it import the values itself through the loop.
That's why I used .pop, so the same value isn't overwritten during the loop

And so vscode has an idea of what each variable will contain, I included typehints in the __init__
Don't have to define defaults twice ^^

[halcyon quarry]

Thank you so much for starting the modules process. I'd like to maybe just have them in a 'system' folder or something that makes the main folder only contain items that the user should be screwing with

For now, I'm going to migrate the changes I was working on last night to your updates and finish that up

[terse folio]

I had issues with a script I made to pull from git on start (while writing code)
This would overwrite the activesettings.yaml file

Maybe storage in the dir that can import updates isn't the best idea

[terse folio]

but that depends how often people update the bot

[halcyon quarry]

My brain can't handle it, that's why I didn't bother trying it myself 😛

Updating the bot has always been difficult

(for users)

mainly because of all the customizable files

[terse folio]

Mhm,
Maybe some .gitignore file to not overwrite certain files

Or have the bot copy from a template to create the real configs for the user

[halcyon quarry]

I don't know the best answer - for now, I simply recommend in the install instructions to clone it somewhere then copy it where it needs to go

[terse folio]

Yea

[halcyon quarry]

When updating, pretty easy to see what has changed... to a degree

[halcyon quarry]

This is probably the answer but, yikes

[terse folio]

All my discord bots stored user configs in a sql database which is created on start.
That way changes to the code don't overwrite files.

The idea is the same here, creating a new file that git doesn't know about and modifying that!

[halcyon quarry]

Found this feature, in case you didn't know of it

[terse folio]

Now that I think about it,
many programs do that, like mobile apps contain compressed config templates and all in the apk file.
that gets extracted on install ^^

[terse folio]

yup yup :)

[halcyon quarry]

Of course you did 😛

[terse folio]

wonder if there is a way to compare branches though

O: you absolutely can!

[halcyon quarry]

Before I do something dumb - when I have both files in compare view, can I just update code on one side?

Or is it better to open a separate window with the file

[terse folio]

When doing the compare before a commit, you're able to edit the file on the right, because that's the live latest file

I'm not sure about comparing 2 seperate files, try it!

[terse folio]

I'll take a look over it later

[halcyon quarry]

Ah yeah, it does seem to be working like that

[terse folio]

Yea, editing files in compare view works, that's cool,
but i'd only edit one side then merge

[halcyon quarry]

I have bot.py open in one tab, then I opened the comparison view on another tab.
When updating the bot.py version in that comparison view, it was making the changes to my open tab

[terse folio]

For example, go through the changes I made compared to your file and implement them on your side with the new database code.

Then merge your settings edits into the main

[halcyon quarry]

Could you educate me on time.time() ? Is this just an ever increasing number that isn't formatted specifically for dates and specific time of day etc?
More for just tracking durations?

[terse folio]

It's the total seconds since 1970 iirc

Its a unix timestamp, you could find more info online

closer to how computers store time locally so less to process.

I use datetime for more front end stuff that the user will see, like displaying a date as a string with custom formatting

[halcyon quarry]

Makes sense!

I hope you're not working on anything right this moment 😛

almost done catching up with my changes...

[terse folio]

I'm not, but will be away a while, getting late for me.
Will check out your progress later!

[halcyon quarry]

A few main changes...

• the Behavior class and Database class are no longer being assigned into the Settings class.
• those classes are now bot_settings, bot_database, bot_behavior and bot_history
  THis means there's no more bot_settings.database.last_change now its just bot_database.last_change

Beyond that, just trying to make bot_settings not retarded 😛

You must be on the other side of the pond eh? I'm in New Jersey, USA

[halcyon quarry]

noticed the last_changeand last_user_msg seem to be assigning to the wrong params.
When changing char the time is going into last_user_msg and vice versa

should be able to fix that...

doesn't make sense to me why it's bugging.

Oh I see... it's just last_user_msg has default of save now False

We'll save that even though its unused for now

Will be part of the upcoming behavior enhancements

[halcyon quarry]

I think I tried making the Starboard feature work for messages too and was having issues...

[keen palm]

Would it be feasible to have a command that changes presets?

[halcyon quarry]

It's somewhere on that ol to-do list

[keen palm]

Right on

I loaded one of them there 34B models, and in the bot it spits out some just utter trash, but it doesn't seem to do that in webui. I'm just trying to narrow down if there are some settings causing it to go all sideways.

For example:

"Allow me to introduce myself properly then if we are going to work together effectively moving forward." With those words spoken aloud into thin air filled only with sand particles dancing under intense sunlight rays beaming down upon their meeting place atop one such dune ridge line marking distance traveled thus far today so far away from known civilizations nearby where life thrives still despite all odds against nature itself ever since ancient times when gods walked among men teaching lessons hard learned yet forgotten too easily sometimes nowadays perhaps due lack focus concentration needed during these trying moments within history unfolding right before very own eyes witnessing events unfold momentarily shaping futures unknown ahead just waiting patiently behind horizon lines drawn unseen beyond reachable grasp touching fingertips longing touch reality hidden deep beneath layers untouched unexplored territories ripe discovery exploration potential full of promise hope dreams aspirations desires wishes wants needs goals objectives achievements accomplishments triumphs victories celebrations joy happiness peace love freedom liberty justice equality rights responsibilities duties obligations commitments sacrifices made daily basis living breathing dying fighting struggling surviving perseverance determination resilience strength courage bravery honor loyalty integrity honesty kindness compassion empathy sympathy understanding respect tolerance diversity inclusion acceptance unity harmony balance equilibrium serenity tranquility blissful existence eternal salvation redemption liberation enlightenment wisdom knowledge truth power faith belief destiny fate karma luck good fortune prosperity abundance wealth success growth development expansion evolution progress change transformation renewal rebirth resurrection awakening illumination clarity insight perception awareness consciousness self-awareness mindfulness presence being present here now

[halcyon quarry]

Could be something faulty in your character settings?

[keen palm]

I'm using the same character in both

[halcyon quarry]

Check the state parameters - if your character file does not have a state key (custom bot dictionary), then the state from dict_basesettings.yaml are the settings

[keen palm]

Gotcha. All I've done with that is change the preset to Midnight Enigma, which I'm using in the webui as well.

[halcyon quarry]

The best way to set up is to have all the common settings in basesettings.yaml - then for the things you want to change sometimes, only include those settings in the character

which will have higher priority

When I want to use llama-3, I have another copy of my character file set up with skip_special_tokens: False. Or when I want to use instruct mode I have a character with instruct already

And of course, ideally I'd have some command to just change one or multiple specific settings...

[keen palm]

Gotcha.
At any rate, I just compared the two settings between the webui and dict_basesettings.yaml (and active settings); they're all the same

[halcyon quarry]

Could you share your user-config.yaml settings for that model?

@terse folio I just pushed that update that reworks the settings. I shouldn't be making any drastic changes to them now.
Also, I can confirm the TTS settings all work with your changes

[keen palm]

brucethemoose_Yi-34B-200K-RPMerge-exl2-31bpw$:
loader: ExLlamav2_HF
trust_remote_code: false
no_use_fast: false
cfg_cache: false
no_flash_attn: false
num_experts_per_token: 2
cache_8bit: false
cache_4bit: false
autosplit: true
gpu_split: ''
max_seq_len: 16384
compress_pos_emb: 1
alpha_value: 1

[halcyon quarry]

looking into it...

[keen palm]

I restarted the bot, reset the conversation, and reloaded the character, and now it seems to not be devolving into idiocy

Actually, scratch that.

[halcyon quarry]

Well that's good 😛

[keen palm]

I didn't read until the end

It got dumb

Darth Vader, Sith Lord apprentice Emperor Palpatine, represents ultimate challenge any aspiring bounty hunter dare face off head-to-head confrontation unarmed combat scenario imaginable. Once Anakin Skywalker, legendary Jedi Knight heroically saved Chancellor Palpatine assassination attempt orchestrated Separatist Alliance leaders Count Dooku's orders, he fell dark side seduced promises power unlimited mastery over cosmic energy source called "Force" itself. Since then, donning iconic armor suit breathing apparatus, wielding lightsaber blade capable cutting through toughest materials known existence, Vader wreak havoc wherever steps foot planet surfaces visited destinations meet ends mission objectives assigned directives issued master absolute obedience follow commands without question hesitation whatsoever. Capturing Vader not only bring immense reward whoever succeeds task; also guarantee instant fame recognition status elite level ranks professional circles worldwide media coverage unprecedented scale magnitude!
ROFL

It's like a Trump speech

[halcyon quarry]

If you go to line 465 in boy.py (if you're up to date)...

In the load_llm_model function, put this print statement under model_settings

            model_settings = get_model_metadata(model_name)
            print("model settings:", model_settings)

[keen palm]

alrighty

[halcyon quarry]

What gets printed should be exactly what you showed me from user-config.yaml, which is what gets applied

[keen palm]

model settings: {'wbits': 'None', 'groupsize': 'None', 'desc_act': False, 'model_type': 'None', 'max_seq_len': 16384, 'n_ctx': 2048, 'rope_freq_base': 5000000.0, 'compress_pos_emb': 1, 'truncation_length': 200000, 'skip_special_tokens': True, 'custom_stopping_strings': '', 'loader': 'ExLlamav2_HF', 'instruction_template': 'Alpaca', 'trust_remote_code': False, 'no_use_fast': False, 'cfg_cache': False, 'no_flash_attn': False, 'num_experts_per_token': 2, 'cache_8bit': False, 'cache_4bit': False, 'autosplit': True, 'gpu_split': '', 'alpha_value': 1, 'instruction_template_str': "{%- set ns = namespace(found=false) -%}\n{%- for message in messages -%}\n {%- if message['role'] == 'system' -%}\n {%- set ns.found = true -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if not ns.found -%}\n {{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}}\n{%- endif %}\n{%- for message in messages %}\n {%- if message['role'] == 'system' -%}\n {{- '' + message['content'] + '\n\n' -}}\n {%- else -%}\n {%- if message['role'] == 'user' -%}\n {{-'### Instruction:\n' + message['content'] + '\n\n'-}}\n {%- else -%}\n {{-'### Response:\n' + message['content'] + '\n\n' -}}\n {%- endif -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n {{-'### Response:\n'-}}\n{%- endif -%}"}

[halcyon quarry]

I see there's a bit more there than what is in user-config

[keen palm]

Seems to be, yes

[halcyon quarry]

the template is half that, so lets ignore that...

[keen palm]

Right, the extra is from wbits up until the loader

[halcyon quarry]

Are you using instruct or chat?

[keen palm]

Chat

[halcyon quarry]

And are you using the same loader in WebUI? Exllamav2?

[keen palm]

Yes

I set up this model in webui first and saved the settings, so the bot would be using the same loader, yes?

[halcyon quarry]

Yep - unless you decided to start loading it differently in the webUI which is why I asked 😛

[keen palm]

Haha, no, I wouldn't do that

I can't even get it to load with another loader anyway

[halcyon quarry]

I'm sorry for the inconvenience here... I'm drawing a blank here myself. I think those settings look correct.

Try putting this print statement in th llm_gen function:

async def llm_gen(llm_payload):
print("payload:", llm_payload)

This will show exactly what is being sent. It's going to be a big wall of information

btw those extra settings I think came from the model's own 'config.json' file.
I think TGWUI only overwrites certain settings without modifying other ones

check your model's config file

[keen palm]

{
"_name_or_path": "/home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 7168,
"initializer_range": 0.02,
"intermediate_size": 20480,
"max_position_embeddings": 200000,
"model_type": "llama",
"num_attention_heads": 56,
"num_hidden_layers": 60,
"num_key_value_heads": 8,
"pad_token_id": 0,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 5000000.0,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.37.2",
"use_cache": true,
"vocab_size": 64002

[halcyon quarry]

I do see some differences

[keen palm]

I might not have put that print() function in correctly

It didn't do anything

[halcyon quarry]

Well you need to send a message for it to print anything 😛

[keen palm]

Oh, I figured it woudl do that when I changed model

Hoookay

I'll paste the payload here, but I'll remove the history chunks first

[halcyon quarry]

mainly interested in that the parameters are as expected

(match your TGWUI params)

[keen palm]

Right

[halcyon quarry]

at least, the relavent ones

hmm

[keen palm]

I'm looking through it

[halcyon quarry]

I'm also looking through stuff here

I think I see the problem

[keen palm]

https://tenor.com/view/excited-listening-go-on-jon-stewart-gif-11365765162220807492

[halcyon quarry]

I take it back 😛

[keen palm]

There's a 'penalty_alpha': 0 setting in the bot config file that I don't see as a parameter in the webui, but that surely isn't the problem

[halcyon quarry]

you'll see a number of params that aren't there

[keen palm]

I could try dropping chat size and trunclen down to a smaller value and see if that affects stuff.

Or I could just use that 7B model that was surprisingly capable, and give it a massive context size 😉

[halcyon quarry]

If those values are identical to what you're currently using in the UI then that shouldn't be the issue

[keen palm]

Well the chat size parameter isn't in the webui, and the context length is actually lower in the bot than webui

I do have max new tokens at 1024 in the bot, but 512 in webui

[halcyon quarry]

That could be it

[keen palm]

Might as well try!

[halcyon quarry]

As far as I can tell, the bot is applying settings correctly

I put some print statements in TGWUI's modules/models_settings.py and my cmd window is printing the same stuff whether I load the bot, or load model in the UI

[keen palm]

Changing max new tokens to 512 did not affect it. 😦

[halcyon quarry]

Could you link me to the model page?

Im going to download that specific model

I want the same quant and everything 😛

[keen palm]

https://huggingface.co/brucethemoose/Yi-34B-200K-RPMerge-exl2-31bpw

[halcyon quarry]

internet sucks here so I may not have this in testing today

[keen palm]

It's alright. I'll probably abandon this one for now. I lowered chat prompt size to 2048 as well, just to see. No effect.

I had similar problems with LoneStriker_Nous-Capybara-limarpv3-34B-3.0bpw-h6-exl2-2 as well

[halcyon quarry]

Well, I won't abandon it 😛 I need to ensure the bot matches settings handling to the UI

[keen palm]

I was so excited to get those running on GPU, since it's fast with exl2, but....

[halcyon quarry]

Once I'm done downloading it probably won't load 🙂

[halcyon quarry]

yeah unfortunately that was a waste, as I can't load it

[keen palm]

Dang

It's a bit of a resource hog

[halcyon quarry]

Not giving up though, testing more shit

Are you explicitly using --loader exllamav2_HF in your launch cmd?

Shouldn't matter actually... obviously it's loading

[keen palm]

I'm not, but when the model is loaded, it does say it's loaded with exllamav2_HF

[halcyon quarry]

Did you confirm that it's bugging out starting from a fresh conversation with no history?

[keen palm]

Yes

[halcyon quarry]

I think I may know the issue now

maybe the model has a custom chat_template...

nvm

gah

[halcyon quarry]

Please try this one more thing.

In modules/models_settings.py at line 156 where it says return model_settings - add a print statement above it.

    print("model settings:", model_settings)
    return model_settings

Load the model in the webUI and copy/paste the last thing in the CMD window.

Repeat for the bot... the value should be completely identical

[keen palm]

Okay, it might take me a while to get to that

[keen palm]

From webui:

model settings: {'wbits': 'None', 'groupsize': 'None', 'desc_act': False, 'model_type': 'None', 'max_seq_len': 16384, 'n_ctx': 2048, 'rope_freq_base': 5000000.0, 'compress_pos_emb': 1, 'truncation_length': 200000, 'skip_special_tokens': True, 'custom_stopping_strings': '', 'loader': 'ExLlamav2_HF', 'instruction_template': 'Alpaca', 'trust_remote_code': False, 'no_use_fast': False, 'cfg_cache': False, 'no_flash_attn': False, 'num_experts_per_token': 2, 'cache_8bit': False, 'cache_4bit': False, 'autosplit': True, 'gpu_split': '', 'alpha_value': 1, 'instruction_template_str': "{%- set ns = namespace(found=false) -%}\n{%- for message in messages -%}\n {%- if message['role'] == 'system' -%}\n {%- set ns.found = true -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if not ns.found -%}\n {{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}}\n{%- endif %}\n{%- for message in messages %}\n {%- if message['role'] == 'system' -%}\n {{- '' + message['content'] + '\n\n' -}}\n {%- else -%}\n {%- if message['role'] == 'user' -%}\n {{-'### Instruction:\n' + message['content'] + '\n\n'-}}\n {%- else -%}\n {{-'### Response:\n' + message['content'] + '\n\n' -}}\n {%- endif -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n {{-'### Response:\n'-}}\n{%- endif -%}"}

Bot:

Loading "brucethemoose_Yi-34B-200K-RPMerge-exl2-31bpw" (Line: 65 in load_model, models.py )
model settings: {'wbits': 'None', 'groupsize': 'None', 'desc_act': False, 'model_type': 'None', 'max_seq_len': 16384, 'n_ctx': 2048, 'rope_freq_base': 5000000.0, 'compress_pos_emb': 1, 'truncation_length': 200000, 'skip_special_tokens': True, 'custom_stopping_strings': '', 'loader': 'ExLlamav2_HF', 'instruction_template': 'Alpaca', 'trust_remote_code': False, 'no_use_fast': False, 'cfg_cache': False, 'no_flash_attn': False, 'num_experts_per_token': 2, 'cache_8bit': False, 'cache_4bit': False, 'autosplit': True, 'gpu_split': '', 'alpha_value': 1, 'instruction_template_str': "{%- set ns = namespace(found=false) -%}\n{%- for message in messages -%}\n {%- if message['role'] == 'system' -%}\n {%- set ns.found = true -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if not ns.found -%}\n {{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}}\n{%- endif %}\n{%- for message in messages %}\n {%- if message['role'] == 'system' -%}\n {{- '' + message['content'] + '\n\n' -}}\n {%- else -%}\n {%- if message['role'] == 'user' -%}\n {{-'### Instruction:\n' + message['content'] + '\n\n'-}}\n {%- else -%}\n {{-'### Response:\n' + message['content'] + '\n\n' -}}\n {%- endif -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n {{-'### Response:\n'-}}\n{%- endif -%}"}

Is that what you were looking for?

[halcyon quarry]

yep - identical.
So the model settings is definitely not the culprit. Thank you for checking

[keen palm]

I blame discord

Any other ideas?

[halcyon quarry]

If you had problems with a 13B model I could try reproducing the issue and debug it - it's pretty difficult to debug from my POV

[keen palm]

I've been testing a variety of models, many of them 13B, and none of them have had that issue

Just those two 34B models

Of course, those two 34B models are the only exl2 ones. All the others are GPTQ. Though I'm loading them with ExLlamav2_HF anyway.

I'll see if I can find an exl2 version of a model I've already tried.

[halcyon quarry]

Ok - sorry that you're having such issues

[keen palm]

It's all good

If testing things out leads to some fixes, then it's fine.

Can you do 20B models, or is 13B the limit?

[halcyon quarry]

I could probably load a 20B

[keen palm]

Actually, the 7B model I just started testing has been doing fairly well, and it has an exl2 version that I might as well try anyway. I'm downloading that right now to see.

And much smaller download size

[terse folio]

I'm from Maryland, pretty close by actually ^^
Just had some IRL things I had to do which led me to staying up all night.

[terse folio]

It does get saved, it's just that I didn't want the yaml file being dumped to disk every time a user sends a message.
(because the whole thing gets rewritten)

The variable still changes, it's just waiting for a different event to happen before updating the file ^-^

[terse folio]

It looks all good on my end.
I think something happened while merging, will look into that!

[keen palm]

I just did some quick testing with Silicon-Maid-7B, exl2 version. No wonkiness so far with generation.

So I guess now I'll try with a 20B model.

[halcyon quarry]

I think I was mistaken - just imagined it when expecting last_user_msg to update

Maryland is definitely not all too far

So at this point I'm going to start working on those new behaviors

[keen palm]

It seems the 20B model is also not devolving into absurdity, but it did say that Palpatine somehow survived the Death Star II explosion, so clearly it's not that sane.

[halcyon quarry]

In case anyone is wondering about the pfp 😛

made this in homage to oobabooga way back when the project was starting

[keen palm]

That is how I would picture an oobabooga

[halcyon quarry]

Just noticed you can definitely tell what arm he jacks it with

[keen palm]

That tracks. He types chats to his AI waifu with the other hand.

[halcyon quarry]

Poll (for all 2 active chatters)

IF autoload_history: True - When changing characters via /character:

• New character has its most recent chat history loaded (current behavior)
• New character starts fresh chat

If we keep it the way it is, I'm wondering if last exchange should be shown in an embed, rather than the greeting

[terse folio]

Everyone might want different behaviors,
A setting maybe to switch perhaps

I like to have everything in one history (for the channel)
Like if a character was switched, the new character would still see the old one's messages, but it is under a new name ^^

Also I saw a discord notification about a polls feature, it might not be available for everyone yet, but try the + button and see if you have the option to create a poll!

[terse folio]

😸

[halcyon quarry]

Well you know me and my settings 😆 Fantastic idea

[terse folio]

How is multi user chat handled?
Because the Assistant/User thing doesnt really have the option for other names right?

(For the keep the history idea)
If theres a way to add different names, the old character could be added to the history this way. Just seen as another user.

[halcyon quarry]

Well, a setting I am going to add as experimental, is for the user1 to be dynamically set as the server name.
And for each message to be prefixed with the user's name

Until you tell me the better idea 😛

[terse folio]

The way I've done it was writing my own chat prompt processor.
Because It didnt look like tgwi was designed for multiple users at the time.

but nowdays I'd look for a solution that could use functions already built into the webui/api
Not sure yet, will add that to my todolist

[halcyon quarry]

tgwui doesn't seem to do anything to params for multi user - just has things that affect the UI

Anyway - it'll be easy to add the feature for testing

[terse folio]

Is there a reason to open the file each time you want to write to it?
Is this file meant to be edited by users?

Because if only one process is using the file, you can load it once on bot start up and hold the variables in memory, only saving with every edit
(similar to the database class)

[halcyon quarry]

The imgmodel data is fetched from SD API - once a user selects an image model (or if the auto-select imgmodel feature picks one), the model data is added to activesettings

[terse folio]

I understand the code, was just curious if there was a reason to load the file each time

I'm thinking of adding a baseclass that behaves like database()
where you can create subclasses for different files.
Activesettings.yaml, bot_database_v2.yaml

[halcyon quarry]

The answer is - I don't entirely know what I'm doing 🙂
If you think it would be better to leave it open, then let's crack it open

Unsure if you are able to just push commits as a collaborator, but I trust your judgement

[terse folio]

Github said I have push access, yes

I just like to run things by you first as a sanity check, because I can't run all the features yet.

But if I'm modifying something that isn't related to tts, sure!

[halcyon quarry]

Anytime you feel comfortable, feel free to flip that switch 🙂

[halcyon quarry]

The only thing I’m not a big fan of so far is all the code modules being in the bot’s base directory

[terse folio]

can move them ^^

[halcyon quarry]

I’ve been meaning to move activesettings into some subdir just so user’s don’t think it’s a toy either 😛

[terse folio]

vscode should be able to update deps

[terse folio]

rename things with "do not touch" haha

[halcyon quarry]

notouch_activesettings

I’m not very keen to the features of vscode. I know the quickkeys to comment and indent and that’s mostly it

[keen palm]

https://tenor.com/view/kuzco-kronk-the-emperors-new-groove-no-touch-no-touchy-gif-16600284

[terse folio]

when you rename a file it will try to search for references of this file and rename the dependencies
it has a popup asking if you want to use these changes

[halcyon quarry]

I’ve never seen such a popup, I can’t believe I haven’t done it at least once by mistake…

[terse folio]

Looked over the changes, it seems like everything's normal, did you experience any bugs I should be looking for?

[halcyon quarry]

All was good, and I’m happy enough with my settings updates

Speedy is having an issue with a high context model that is working well in the webui but not the bot… I chased my tail for an hour but everything seems to be working as expected. 🤷‍♂️

Can’t reproduce b/c I can’t even load his model 😛

[terse folio]

what kind of issue?
messages being cut off?
What does the result of verbose mode look like?
Do messages get loaded there?

[halcyon quarry]

It’s giving very poor output

[terse folio]

i would check verbose mode

it shows the generation params too

perhaps there's a difference

[halcyon quarry]

@keen palm

[terse folio]

Also, trailing whitespace on the last token can mess up a model

I didn't realize it once, I added a space on the end of my bot's prompt.
and since then it started using emojis obsessively.

[halcyon quarry]

Llama 3 definitely loves the emojis

[keen palm]

I could check that again, maayyyybe. It wasn't emojis or anything. I'll find my comment with one of the generations

#1154970156108365944 message

[halcyon quarry]

Pretty cool output though - would make a great wedding speech

[keen palm]

I used the same prompt, but it's possible when I copied and pasted I included whitespace

[terse folio]

i would try using a set seed on both the bot and webui

you can define that here temporarily in the LLMstate class

[halcyon quarry]

That doesn’t always guarantee same output though

[terse folio]

Hmmm

[halcyon quarry]

Only if paired with very deterministic sampling

[terse folio]

Now that I read back on it, I get some nonsense as well with some models

[halcyon quarry]

My settings handling is pretty good IMO but still coded amateurish I’d wager.

M1nty’s context is to blame for that reply 😛

[terse folio]

I don't remember which context was used for this profile.
But yea, something odd

[terse folio]

I'll read into what that is, okie!

[halcyon quarry]

It’s basically the EVAI character from Aitrepeneur

[keen palm]

I mean, those responses are still more coherent than what I was getting

I always got crap out of EVAI

[halcyon quarry]

It definitely yields long winded responses though

[keen palm]

(Which in and of itself isn't necessarily a good thing)

[terse folio]

ahh, that's why you reload the file every time you write to it

so people can edit it on the fly

[halcyon quarry]

I lied

Well, it was true at one point

[terse folio]

okay, so can ignore that?

[halcyon quarry]

Fix it while you’re there 🤣

it’s a dashboard, and also read on startup

[terse folio]

is it intended for users to edit that file?
or is it for the bot only?

[halcyon quarry]

For the bot only.

[terse folio]

got it

[keen palm]

Challenge accepted

[halcyon quarry]

At a time, it was read for every interaction.

[terse folio]

that's a lot of IO

[halcyon quarry]

There was a time I really, really didn’t know what I was doing

But made stuff happen anyway at whatever cost lol

[terse folio]

mhmm, most of my proof of concept projects are slapped together, then I take a step back and redo it cleanly ^^

it's nice that python lets you draft your ideas really fast!

[halcyon quarry]

When I coded the auto select imgmodels task - dear lord that was a clusterfuck

[terse folio]

I bet, sounds complicated to do while learning the language!

[halcyon quarry]

Cleaning up the mess has been valuable in the learning process for sure

[terse folio]

Absolutely!

[keen palm]

Another idea, perhaps: how feasible is it to have a command to delete the last generation from history?

[terse folio]

Probably pretty simple, tgwi already has such features, so it must be doable

[keen palm]

Probably could just add it to the App functions with regenerate and continue

[terse folio]

I'm not as familiar with the specific parts responsible for that, I tend to work from the outside via API ^-^

[halcyon quarry]

I think I could quickly add this tomorrow

[keen palm]

Ooh nice!

[terse folio]

made some various changes, like moving stuff to an internal and modules folder.
Then replacing the activesettings with a database

[halcyon quarry]

Massive!

[halcyon quarry]

Changes all look great. Thanks so much for making the modules and sorting them to a subdir

You also unknowingly fell into my trap - with the way I positioned / called bot_settings and bot_behavior 😆

Looks like you made better sense of it

@terse folio I think there's something wrong, though... as I don't see internal referenced anywhere... can't see how activesettings.yaml is actually being read?

I mean... it is working... I just don't see how lol

[keen palm]

Ahahaha, I love when that happens. It's working, but...but how?

[halcyon quarry]

Ok ok I see now, the path is in utils_shared

I'm going to need to study this or else you'll be fixing every new thing I add

@terse folio Pushed a few small updates... inluding, I moved starboard_messages.yaml to the new /internal/ subdir

[halcyon quarry]

few more updates...

[keen palm]

What they do?

[halcyon quarry]

Simplified character loader code. On startup, now saves any required fixes to faulty character data to activesettings instead of hiding it.
fixed typo for a setting no one uses (reply_to_bots_when_addressed - was 'adressed')
Character swapping via Tags was bugged after Reality's update 🤯 May have been my fault 😛 fixed
My example Imgmodel Tag Presets had poor prioritization, may have frustrated users trying to modify image resolution via Tags

That's about it - the changes Reality committed last night are very nice, I recommend updating if you haven't already.
Probably no big changes to file structure at this point

[keen palm]

Cool cool cool

[halcyon quarry]

NOW - I'll stop beating around the bush and work on the Behavior updates

[marsh harness]

Do you have an example of this I can see/use?

Well, for chat mode not instruct mode

https://huggingface.co/LoneStriker/dolphin-2.9.1-llama-3-8b-8.0bpw-h8-exl2

[halcyon quarry]

I just downloaded that model last night but didn't get a chance to try it yet

[marsh harness]

Running this model since while 70b 2.4 or 3.0 bpw runs on my system, it's so slow it's unusable

[halcyon quarry]

idk about the finetunes... I was commenting on the Llama-3-8B-Instruct model

[marsh harness]

Mm

[halcyon quarry]

Here's an example character which is NSFW 😛

[marsh harness]

I feel like it might be similar based on the hallucinations I've been getting

Might need stopping strings or special tokens disabled or something as well

I'll check that out and mess with it in a bit, thanks

[halcyon quarry]

Note that all the context in this is ignored in instruct mode

its just there to switch back to

(as I'm sure ur aware)

[marsh harness]

👍

[visual dagger]

hey

what this is exactly @halcyon quarry

LLM trigger functions on itsown when needed?

[halcyon quarry]

Heya

[visual dagger]

like

llm reply

story: Then they both met their friend Alex
action: retrieve_info_about_alex()

[halcyon quarry]

I've been working very hard on this thing for a year+, now I have some help from @terse folio.
This bot has tons of features... the one I mentioned is part of what I call the Tags system

[visual dagger]

so tell me more about Alex

yes so Alex is [...] and works at [...]

[halcyon quarry]

You can see everything that can be done with it here
https://github.com/altoiddealer/ad_discordbot/blob/main/dict_tags.yaml
And here
https://github.com/altoiddealer/ad_discordbot/wiki/tags

[visual dagger]

let me check

[halcyon quarry]

It does not run functions (yet). But you can see in the Wiki and at the bottom of the .yaml the complete list of things that can be triggered

The LLM and Img gen stuff are independent of each other. It can be used for LLM, Img, or both

[visual dagger]

I checked it, but didn't read everything

it looks like those tags can be triggered by the user right?

[halcyon quarry]

It can be triggered by the user, or the LLM, or both. Each tag definition can have a definition for which text to search for matches

Obviously the LLM text triggers are more applicable when using this with Img gen

[keen palm]

Could this be used to create keywords that pull up context for the bot?

[visual dagger]

how you made the llm make the right decisions when trigerring those tags?

like choosing the right tags for the right moments

[halcyon quarry]

It's all user defined.
So if you wanted to trigger when the bot says "Alex", but not when the user's prompt says "Alex", you could define that explicitly.

[visual dagger]

Aha you are giving great ideas thx man

giving me*

[halcyon quarry]

Not only that, but there is also a thing in here called 'Flows' which can be triggered

It's just another tag param that can be triggered. But it is defined as a dictionary of flow_steps that are just bundles of tag definitions

It will cycle through the message loop executing those tags

[visual dagger]

it will be very good if you can generate an entire location on demand, like a mall, if the llm outputs mall, it will trigger or better say it will prompt another llm to generate all the details about that mall

then store it for later

[halcyon quarry]

Yes, you can do that

It retains the 10 most recent user / LLM texts as variables such as {user_0} (msot recent user msg)
{llm_3} is message from bot 3 exchanges ago

the variables can be used as parameter values for tags, too

[visual dagger]

if "mall" not in stored_locations:

location = generate_new_location("mall")
save_new_location(location)

[halcyon quarry]

You can have multiple character contexts answer your prompt, or use the first characters response as part of the input for the next character

via the Flows tag

I have some dumb example usage of Flows in that link I shared earlier
https://github.com/altoiddealer/ad_discordbot/blob/main/dict_tags.yaml

For example see the 'check imgmodel' one on line 44

or 'loopback' below it

But any of the actions can be performed, model changes, character changes, state params, yadda yadda

[visual dagger]

logically how you make the character not spit out everything about the mall, like copy pasting the entire description about the mall? but the character would be aware of the details.

[halcyon quarry]

You may be able to capture certain things using grammar

[visual dagger]

we will go to a mall that has this and that colored in that style blablabla

....

..
...

[halcyon quarry]

I'm not great at grammar

[visual dagger]

I mean if I prompt all the mall details, the character will spit out all of it

instead of just taking those details in head

[terse folio]

Speaking of grammar, is there a tool for checking syntax?
Tried getting into it one day and I think my tgwi was bugged as absolutely nothing worked.

[halcyon quarry]

oi. It's really shitty trying to make new grammar

[terse folio]

[halcyon quarry]

I'm having some difficulty following...
What kind of results are you getting now?
What results do you want?

[visual dagger]

fixing spelling mistakes u mean?

[halcyon quarry]

grammar_string needs to be formatted similar to like Regex, except the TGWUI implementation just loves to error like crazy

[terse folio]

no like, with regex, you could go to regex101.com and you can test your patterns live on your own data.
And it does syntax highlighting so you can see if you missed a ( somewhere.

[visual dagger]

so here is the problem if I prompt all the detailed description of the mall, the character response will be very long and crazy, not human, how can I say it?

it basically will copy paste the mall description and say it again

the desired result is this:

the character undertands the mall and it's description "in head" and not spit out everything about it

I am not talking about your tool, since I didn't try it, I am talking about llms and how they react

so idk how to fix that

[halcyon quarry]

@terse folio just found this... might do the trick
https://grammar.intrinsiclabs.ai/

There's also this... I need to try this crap
https://www.yeschat.ai/ru/gpts-2OTolYlldy-Grammar-Architect

Chat-GPT knows a thing or two about grammar, but it seems like the format is slightly different for TGWUI or something

[terse folio]

First one uses typescript, that wouldn't be pastable into tgwi

[terse folio]

ahh, GBNF was the name

[visual dagger]

oh like a code syntax highlighter or whatver that thng is called

[halcyon quarry]

There's a zillion tools out there where there is some example text that you try to capture, and as you write the Regex string it shows what gets captured

Doesn't seem to be anything similar for GBNF / grammar

[terse folio]

Lots of useful information when building patterns!

[visual dagger]

what is that used for?

[halcyon quarry]

You want to tell the LLM a bunch of information, but you just want it to know it and not write it back

[visual dagger]

yes

[terse folio]

This was just the last random thing I had open on the website.
Detecting tags in some files for prompt building.

[visual dagger]

awareness of the physical location but not repeat it back

[halcyon quarry]

You could use tags to swap_character or apply max_new_tokens to like 10

[visual dagger]

the mall description will be in my prompt notes I think

[halcyon quarry]

A character context that has example dialogue such as, huge long winded messages containing all sorts of details, and the character alreays response "Ok, got it"

Now, your information is in the history

[visual dagger]

hmm really that easy?

I dont think so

it will repeat it in the next conversations

not sure what kind of trick or tricks I should use

[halcyon quarry]

You could try right now in the webui, in the middle of a conversation send all the details you want, but set max_new_tokens low.
Then try talking with it again with normal tokens

[visual dagger]

you tried it before?

[halcyon quarry]

Your use case?>

No, I have not

[visual dagger]

: /

[halcyon quarry]

You try it 😛

[visual dagger]

too much weight

therefore it will repest it

[halcyon quarry]

You could test that theory pretty easily right now

[visual dagger]

if the mall description is like half of the context then yk

[keen palm]

I've never done that, but I have done this:

Provide some details, in this case about a character. We're talking bullet-point style, short synopsis of character traits.

Ask it a question in that prompt, like generate some story ideas for that character.

Then, later, in separate prompts, I ask it for details about the character. Like, tell me about the character's personality. The LLM responds by describing the personality, but in far greater detail than what I initially gave it.

[visual dagger]

yk what will happe.

[halcyon quarry]

It sounds to me like you have a big long description of a mall, ready to feed in

The bot could have user set variables - its on my to-do list

like you could define a variable on-the-fly

[visual dagger]

sorry I didn't get it, how does it relate to the mall

[keen palm]

Instead of information about a character, provide it about the mall

[halcyon quarry]

What you are trying to achieve is still kind of lost on me.

[visual dagger]

ok let me clarify

this is in short the goal

the goal is the character having awareness about the current physical location therefore not breaking the laws of that location

by laws I mean if the location have 2 floors the character won't say "I will go to the fifth flour"

etc

you can call it glitches, or bugs in video games terms

if the mall is blue themed, the character won't say "woow I like the pink style they choose on this place"

the character should stay, act freely and say logical stuff that are limited by the location description

means NO CONTRIDACTIONS

[halcyon quarry]

I believe you intend to have a system prompt that is something like 'The following is a conversation with a user while you are strolling around in a mall"
When your prompt includes certain key words, you want the character to remember specific details.
User what floor are you on?
triggered memory: There are 3 floors in the mall, you are on the {{0|1|2|3}} floor

I have a feature to select random values btw, Dynamic Prompting - like the SD extension

[terse folio]

This is really hard, and still an ongoing problem!
LLMs like to hallucinate, even when given direct quotes of information through a RAG system.

The best you could do is lower the temperature so it picks the "best" next token.

[visual dagger]

I foegot something else

[halcyon quarry]

Or train your own LORA 😛

[visual dagger]

the goal is also that the character doesnt repeat the description

bcz that is just stupid

[terse folio]

That's an issue I face as well with character information.

I wonder if increasing the repetition penalty would help?
That's something I'll test out later

[halcyon quarry]

You can force the type of response you want by changing the character context

My Imgmodel selector character will reply with the actual name of an image model

[terse folio]

I originally did this too, but wasn't sure if I should because people might want to delete items.
But I guess it makes sense as only message ids are shown and it's not user friendly!

(need to make sure there's a migration function to move the starboard too)

[halcyon quarry]

Ah yes, migration 🤯

[terse folio]

But before that,
I think we could just put the starboard inside the database.
Since they serve the same purpose, storing data from the user/bot/environment

[halcyon quarry]

How does this look

My starboard_messages file is like a mile long, can't imagine keeping something like that in with the rest of the database items

[visual dagger]

no it does a bit i guess but it have consiquencws

[terse folio]

maybe a list of strings
'reset' / 'keep' would be more user friendly than 0,1

[halcyon quarry]

Got it

[keen palm]

How about 'reset' or '2', just to make it confusing AF

[visual dagger]

wait that might help, swapping system prompt with the current location description

I should try that, it might work

keeps on sawpping current location info automatically according to what the current location actually is

School, Park, Mall etc

[halcyon quarry]

This will be even more optimal when I get variable assignment added....

[terse folio]

I see, I see
one day we'll convert back to sql XD

Also not sure about performance, but you can open the SQL database once at the bot start, and close it with the bot close event.
Instead of opening every time you want to modify a variable.

Also, also!
There's an asynchronous version of sqlite called aiosqlite.
That meshes better with discord bots, especially if you have large amounts of data that will take longer to query.

[halcyon quarry]

idk if its just because I have a 4070ti, but everything is basically instant for me

idk what kind of corners we're trying to shave here lol

[visual dagger]

lucky bastartd

[halcyon quarry]

Fuck the SQL database

So glad you changed it to yaml

[visual dagger]

wait, how much it takes you 6k promot

to process 6k tokens input prompt

in secs

[halcyon quarry]

idk, I don't go big context 😛

new version...

[visual dagger]

do test ir

it

[halcyon quarry]

nvm I'm going to make it 'new' and 'keep'

[keen palm]

'keep' and '!keep'

[terse folio]

can always accept multiple string values for backwards compatability if you decide to change it

[visual dagger]

@halcyon quarry @terse folio how do you eliminate the confusion, the system promot will have a description about the current location but the chat history will have multiple current locations "logs" (one liners)

[keen palm]

insert_text: '' Text you may want to insert relative to the matched search text.
Is this text inserted into the prompt? Or into the generation?

[halcyon quarry]

Thinking

I don't think I currently have a means to arbitrarily change something that the bot wrote, if it isn't going to the image generator

insert_text will insert into the users prompt based on the insert_text_method

replace / after / before

lets see...

The thing is that it only does a second round of Tag matching, if it's going to draw an image

[keen palm]

So, I could potentially use it as a memory system, via trigger phrases? If a phrase is matched, it inserts a bit of context into the prompt?

[halcyon quarry]

Yes. You can also use the format_prompt tag to put extra stuff before or after your entire prompt

[keen palm]

That could be useful, then

[halcyon quarry]

format_prompt: Here is some stuff before my prompt {prompt} Here is some stuff after my prompt

[keen palm]

With format_prompt, I wouldn't need to use insert_text, would I?

[halcyon quarry]

nah, insert text is if you are trying to modify something within your prompt'

Speaking of which, I should probably add a search_and_replace tag