ad_discordbot (Fork of Fork of xNul's bot)

[halcyon quarry]

Example of using all together

      - name: threshold
        description: Adjusts the strength of the edge detection
        type: string
        required: false
        choices: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
        steps:
          - if_group:
            - if:
                value1: "{result}"
                operator: <
                value2: 0.2
                steps:
                  - send_content: "You Selected 0.1"
            - elif:
                value1: "{result}"
                operator: ==
                value2: 0.3
                steps:
                  - send_content: "You Selected 0.3"
            - else:
              - send_content: "You Selected value greater than 0.3"

[fickle ember]

i updated the bot and im getting this error

ValueError
00:27:17.026 #4287   INFO [bot.__main__]: Running task 'on_message' from queue 'message_queue'
00:27:17-074650 ERROR    Failed to build the chat prompt. The input is too long for the available context length.

                         Truncation length: 2048
                         max_new_tokens: 512 (is it too high?)
                         Available context length: 1536

i find it odd though

[fickle ember]

nvm i think i fixed it increasing the truncation length and max tokens

[halcyon quarry]

What’s printed from the loader is a model param, but the state param overrides it I think. Also fairly certain that it takes more vram to load the model with higher context length value

So if you are using state param truncation length 4096 or something, you should probably also dial down that model loader param

[halcyon quarry]

With the new “if” steps it’s now possible to do an action when a user does not use a certain optional command option

[fickle ember]

@halcyon quarry i got a server owner asking me to make the bot only able to speak in certain channels how do i do that

[valid crypt]

if you mean that even pinging does not trigger unless in certain channel, then probably you cant

[halcyon quarry]

It’s just channel permissions

Via discord UI

There’s also the “/main” command

[valid crypt]

right, discord it self can do the job 😅

[halcyon quarry]

Right click the channel / category and Edit X

I’m on the road but pretty sure you go to Roles

You add the bot to a Role or just create a permission for it

If you tick red X for “Join” poof the bot leaves the channel

red X for send message, done - no message from bot

To restrict command usage go to Integrations

[tawdry osprey]

I can't seem to figure out what I'm doing wrong for the set up of this. I cloned the text generation webui repo, ran the start_windows.bat for it to get it to start up and do an initial set up, then I cloned ad_discordbot repo into the text generation webui directory as the instructions said to, and when I run the start_windows.bat and try to do the setup for ad_discordbot, I just get this.

[fickle ember]

Instal miniconda?

[tawdry osprey]

Got it sorted out

[halcyon quarry]

If you could share what the issue was after all that would be great

[tawdry osprey]

I had to tell it to install as standalone, use the updater to swap it to text gen integrated mode, then when I run the start_windows.bat it works as intended.

[halcyon quarry]

Are you using the TGWUI "portable" install?

[tawdry osprey]

By standalone, I meant portable, yes.

[halcyon quarry]

I understand that you installed the bot in standalone, then used the updater script - just asking about TGWUI itself

they recently added a "portable" install method

[tawdry osprey]

Oh, I just cloned the repo to my desired directory and ran the start windows bat. I dont actually recall needing to choose portable or not.

[halcyon quarry]

I'm going to do a full clean install...

see if maybe they changed something about their install process.

ah ok ok... they install "miniforge" now instead of miniconda. Might change the structure slightly...

Yeah, crap

Reproduced same result

Managed to fix it

@tawdry osprey thanks for the bug report!

[tawdry osprey]

No problem. Thanks for fixing it.

[halcyon quarry]

If you have any questions, feedback, suggestions, let me know!

[tawdry osprey]

Got it. Will do.

[valid crypt]

when this get merged tgwui will definitely support vision https://github.com/ggml-org/llama.cpp/pull/15108

ill just 🙏

[halcyon quarry]

I’ll be ready for this

I need to catch up with some TGWUI code today/tomorrow - I think explains some differences in output from some of the newer model architecturess

I noticed the code now accepts text and files

(Among other changes)

[halcyon quarry]

@valid crypt

The bot will be ready for this

[valid crypt]

（Φ ω Φ）

[halcyon quarry]

Wondering how to handle this ethically... TGWUI expects to receive the file input as a file path to a local file.

My instinct is to save the user's attachment locally to a temp directory

Pushed an update that may improve output from newer model types

• NEED FEEDBACK

[late pivot]

@halcyon quarry btw how do i use tags again?

as memory thing

since i want it to save memories too

but idk how and i cant an extension for it

also if i want to remove the / commands like the restart_sd_client how do i do it?

[halcyon quarry]

Currently don’t have a system in place to save memories - you can add them manually via tags

[late pivot]

yes

thats the plan

i want to add it using tags

but how does it work?

[halcyon quarry]

I believe these can be removed in your discord server settings > Integrations > Bot (should be somewhere here)

[halcyon quarry]

I have a number of sample/example tags. Basically you’ll want to use a “trigger” which would activate the tag

[late pivot]

what are the limits?

and how do i trigger it?

[halcyon quarry]

The text would just have to be found in your prompt and/or the LLMs reply. There’s another conditional tag which can make it exclusive to either

[late pivot]

i still dont understand, could you send me an example on how to make one and how the trigger would work?

[halcyon quarry]

in dict_tags.yaml just copy paste one of the ones that has a “trigger” - change it to something that should trigger your memory effect like “Bob”. If you want the match to be case sensitive you include that tag as well. If you want it to only trigger if it’s the first word in your prompt, add on_prefix_only tag

finally for the actual memory effect, you include prefix_context

[late pivot]

like this one?

• trigger: 'calculate,define'
  on_prefix_only: true
  state: {preset: 'Divine Intellect', mode: instruct} # Update values for textgen-webui state parameters.

[halcyon quarry]

Ya - so to get technical tags are “a list of dictionaries” - so that’s packaged up together. If there are conditional tags in the package those conditions must be met.

If there are no conditions, or there are and those conditions are met - it will do the other things packaged in there

[late pivot]

is there a template which will just remember what it said?

[halcyon quarry]

With that example if you prompt “calculate the sum of 14775 x 57866” it would apply that preset to the llm params

[halcyon quarry]

Not yet - the memories currently need to be predefined

[late pivot]

so how do i make it to say something or remember something?

[halcyon quarry]

• trigger: 'Bob'
  prefix_context: ‘Bob really loves waffles’

If you say something and mention Bob, that line of text will be snuck into the “context” which is part of the prompt

[late pivot]

so it will say something like that but it will follow the character personality?

[halcyon quarry]

Right, it’s basically just information that you’re spoon feeding it, and it is lower priority than the recent chat history

[halcyon quarry]

It sounds like you may even want to dynamically modify your prompt - you can use a tag like “insert_text”

Or format_prompt

[late pivot]

do i always need to say the word or can i make it into a sentence?

like "what does bob likes" and will the llm responds with the trigger?

[halcyon quarry]

Llm will see your character context, then “bob really likes waffles”, then your recently chat history, and finally your question (this would be with “suffix_context” - the priority would be different with “prefix_context”)

Which is all “the prompt”

It would probably respond in character in the context of your conversation to convey that bob loves waffles

[late pivot]

alright i think i understand it now

thx

[halcyon quarry]

Cmd will print tag activations, so if you set it up and test you should see it print something

[fickle ember]

so when this happens we will be able to natively support multimodal models within the bot?

this would allow us to post images in chat and have the bot actually see it finally

🧠

[halcyon quarry]

Yeah - it's not going to be super difficult to implement - I basically just need to check for attachments, save them locally and add their file paths to the TGWUI payload.

Thinking about whether this behavior should be configurable if people don't want their bot auto-downloading files from discord

[halcyon quarry]

Multimodal support was released in TGWUI

I’m going to try to support this tonight

[halcyon quarry]

So what is nice is that the vision model is just a parameter for GGUF models. When you save your model settings it is part of those settings, so should autoload when selecting that model via the bot

[halcyon quarry]

Ugh. Slightly more complicated than expected

[halcyon quarry]

It seemed like I could just add image data to the payload, but at some point TGWUI seems to dynamically add "markers" in the text prompt

...or something...

guess I'll just do what I was going to do originally, save the images to the TGWUI cache directory (like it does natively) and send the file paths.

[halcyon quarry]

The other detail I need to figger out, is detecting whether the current model is multimodal. Because if it sends images to a non-multimodal model TGWUI raises a runtime error instead of ignoring. So the bot script has to be selective about when to send images or not

[halcyon quarry]

ooba merged a PR I submitted which makes one thing less clunky on my end, and makes more sense on his end

[valid crypt]

last time i forced a mmproj of different models it gave me a big crash in lmstudio :v

i also saw that you can quant the mmproj but ive only seen people making q8

[halcyon quarry]

The bot will support the multimodal stuff in another day or two

waiting for ooba to merge that PR to main - then I gotta figure out one more thingy

[burnt patrol]

Yay

[valid crypt]

yay

[halcyon quarry]

I'm removing the streaming TTS feature for TGWUI Extensions

It will still work for TTS API

There's something really screwy happening with my custom chatbot wrapper

I can't figure it out, but everything is just perfectly fine if I use the native chatbot wrapper

It also makes the difference of whether multimodal works or not

So really at this point I'm just waiting on Ooba to merge his dev to main

Actually I can push this now, it just won't work until he updates

[halcyon quarry]

Pushed the multimodal support

• This is not actually going to work until the next TGWUI update

Needs this to work

Most importantly - output issues should be FIXED

If anyone wants to test the multimodal crap, just add this line in modules/utils_tgwui.py

Your model better be multimodal though!

[late pivot]

@halcyon quarry btw how does the autoload history works?

since i keep seeing it use the old one and not hte most recent one

[halcyon quarry]

When starting the bot, it should resume the most recent histories instead of initializing fresh histories

I’m not sure if it’s intended to do the same when switching between characters…

[valid crypt]

ill just wait 🤗

[halcyon quarry]

Don’t sleep on updating the bot, though includes a pretty significant bug fix

[late pivot]

Do I need any image to text model for it?

[halcyon quarry]

oobabooga has some example models that are supported here

https://github.com/oobabooga/text-generation-webui/wiki/Multimodal-Tutorial

I tried the lowest quant of the exl3 model and could juuust barely load it. Was able to load the gemma 3 with vision very easy though

[late pivot]

Is there any model that is similar to mythomax l2 13b that also supports it??

[halcyon quarry]

I’m not super in-the-know on LLM stuff

[halcyon quarry]

I noticed that sometimes the bot would not send "extra content" (ei: results from custom commands feature)

Realized I had a race condition where the bot could delete the task object before it can send the content

Pushed an update that fixes this!

[late pivot]

do i have to update the text generation webui too btw?

to fix the multimodal thing

btw is this normal?

it keep replying using emojis

i use this model https://huggingface.co/cjpais/llava-v1.6-vicuna-13b-gguf

[late pivot]

also @halcyon quarry is it possible that you could add a feature to run it in google colab instead locally?

since google colab offer you some high end gpu for free

so it would be great if you can add google colab support for it

[halcyon quarry]

Ye of little faith

Ooba still didn’t merge that commit so still need to manually force that variable in the bot to try out multimodal

[late pivot]

Btw what model are you using?

Kinda wanna know since I don't wanna install 100+ model and test each one

[halcyon quarry]

I mentioned it before - just the gemma 3 model named on that tutorial

I’m not really using it though, was just to get the feature working expectedly

[late pivot]

What do you mainly use?

[halcyon quarry]

Neuralbeagle

[late pivot]

is this normal btw?

[fickle ember]

no

[fickle ember]

love the bot name btw

[halcyon quarry]

1. Make sure you've updated to the most recent version of the bot, as it has a big bug fix
2. You probably have some parameters off or something

[fickle ember]

right the update!

[halcyon quarry]

Ooba just merged that commit to main, we were waiting for

If you update TGWUI - vision should work w/the bot

[fickle ember]

should i wait to update the bot

[halcyon quarry]

Bot's already ready for it

[fickle ember]

ok

[halcyon quarry]

For GGUF - I recommend loading the model + vision model via the UI - click Save Settings

Bot will handle it correctly.

[halcyon quarry]

You may not have updated in awhile

[fickle ember]

i dont think this will cause basic functions to work but im pasting it there just in case

[fickle ember]

yeaaah

should i nuke config.yaml and re update

[halcyon quarry]

It looks like you are fine

you can copy paste those from the config in settings templates

if you want to avoid the messages on startup

[halcyon quarry]

I had drastically changed the structure in config.yaml a few months ago but your cmd would have printed quite a bit more.
I believe you've already caught up with those major changes

[halcyon quarry]

See if using the default dict_imgmodels.yaml resolves that

[fickle ember]

i just updated text webui as well

im trying to figure out how to get vision working on that first

[halcyon quarry]

He has a tutorial for that here https://github.com/oobabooga/text-generation-webui/wiki/Multimodal-Tutorial

Pretty straightforward

[fickle ember]

ok so it needs to be gguf

im gonna cull my model list and find some good ones

[halcyon quarry]

Can also be exl3

[fickle ember]

the model i use is exl2😭

i need to find some newer models asap

[halcyon quarry]

@fickle ember can you test this file in /modules/? Back up your original tags.py

Should skip malformed tags, and also give a good hint where the problem lies

[fickle ember]

what are some good new uncensored gguf models if you know any

or exl3

im gonna use neuralbeagle for testing

[halcyon quarry]

In your character file you'll probably find that some of your TGWUI params are nested under tags dict

[fickle ember]

huh youre right

idk why those are there

[halcyon quarry]

Ok you can revert the change to that file - I'm pushing that to main now

[fickle ember]

gonna make chatgpt deep research for some good models that meet the requirements

[halcyon quarry]

If you find a good new uncensored gguf model lmk!

[fickle ember]

https://tenor.com/view/shrug-what-huh-will-smith-i-mean-gif-3535627793955785136

[halcyon quarry]

eh

[fickle ember]

were probably early to this given that update was released recently

gotta give the people time to make models i guess

[halcyon quarry]

Vision has been around for awhile I think it's only now oobabooga got around to getting support in there

[halcyon quarry]

Notice

TGWUI will likely prompt to delete the installer_files directory when updating.
Just be sure to run the bot's Update wizard once you finish the TGWUI update process

[fickle ember]

doesnt matter what chatacter you use. results come out like this

Traceback (most recent call last):
  File "C:\ai\text-generation-webui-1.13\modules\text_generation.py", line 489, in generate_reply_custom
    for reply in shared.model.generate_with_streaming(question, state):
  File "C:\ai\text-generation-webui-1.13\modules\llama_cpp_server.py", line 202, in generate_with_streaming
    response.raise_for_status()  # Raise an exception for HTTP errors
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ai\text-generation-webui-1.13\installer_files\env\Lib\site-packages\requests\models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:60234/completion
21:16:34-568037 INFO     Output generated in 0.17 seconds (0.00 tokens/s, 0 tokens, context 1794, seed 1901088493)

[halcyon quarry]

Tried the same model, character, params in the UI?

[late pivot]

@halcyon quarry to make it remember something using tags should I use I am or you are? For example "I love chocolate" when I ask for favorite food?

Like putting it for tags

[halcyon quarry]

In chat mode I think you can actually use {{user}}

When TGWUI parses the context param it dynamically updates those placeholders- chat mode only

[valid crypt]

both should work, but i think "I" is better

you can add the tag and ask it ^_^

[valid crypt]

🥳

[halcyon quarry]

He just wants me to push more commits 🤓

Meanwhile it was for my own selfish purpose! Mwahaha

We both wish for contributors

[valid crypt]

can you teach me step by step how can i add a new message task from a text file or something and add to the current pipeline? 🥺

[halcyon quarry]

What is in the text file?

A prompt? A list of prompts, one per line?
Any other generation params?

[valid crypt]

transcription >_<

you can say 1 prompt

[halcyon quarry]

If this is to be triggered from a regular on_message request, then this new method would either override current behavior (based on a config setting).
Or, the special handling could be handled by Tags.

[valid crypt]

would you mind if i do a PR with just the stt and you do the magic?

[halcyon quarry]

It matches tags, and also processes the first phase of tags in process_user_prompt

But does not generate text, or send text - until after this step

So - you basically want to update the prompt

We might just need to add a new "tag" like "prompt_from_file"

[valid crypt]

my skill is capped here

thats all i can do >_<

[halcyon quarry]

Push to this branch - and I'll see what I can do
https://github.com/altoiddealer/ad_discordbot/tree/stt

[valid crypt]

ill do it asap although i just found a problem with multi speaker

@halcyon quarry i think i did it properly

if im not wrong the only requirement that you would need to install is discord-ext-voice-recv

oh and whisper :P

i forgot to delete this line

[halcyon quarry]

I’ll try to find some time to work on this but it might be a few days, possibly longer. We’ll see

[halcyon quarry]

I was trying to show off the custom commands feature, but it got bugged when ComfyUI had an OOM

I'm trying to debug that now and I'm struggling to make it OOM XD Loaded all my software, browser window playing Youtube, movie player, itunes playing music

Pushed a small commit

• Custom Commands can now be configured for use in DMs

literally can't OOM Comfy right now, da fuck

[valid crypt]

loading all your software? go load a llm into ram 😏

[halcyon quarry]

I shit you not I fired up Forge and loaded a Flux model. Loaded the bot with an LLM. Also loaded another TGWUI instance with another model. And ran a simultaneous Flux generation in Forge and Flux Kontext via the bot/ComfyUI and NOTHING OOM

[valid crypt]

:O

[halcyon quarry]

Basically I’m gonna need to download some monster unquantized video model and try running it from the bot / comfy

Like wan 2.1 14b 720

Might not be big enough… eh I’ll get one big enough to crash

[valid crypt]

disable fall back to system memory so you only have real vram :3

[halcyon quarry]

ChatGPT pretty mindblowing these days for sure

I casually asked where I could find DateTime substitutions in ComfyUIs codebase, and it told me exactly where that code executes

I reported an issue yesterday, where if you "pipe in" a string using datetime syntax to something like Save Image node, it would not update the value. It only works if used directly in the node field.

I decided to just try to take a quick stab at it. Right away, it got the answer for this

https://github.com/comfyanonymous/ComfyUI/pull/9523

Is Comfy gonna actually merge it? prob not but who knows

[halcyon quarry]

Got a neat little setting coming

skip_load_llm

If True, will still capture cmd args and all that but won’t load the model right away

Instead the model will load for first llm_gen

suppose “delay_load_llm” is more apt

[halcyon quarry]

defer_load_llm

lazy_load_llm

[halcyon quarry]

Pushed the Lazy Load LLM option

[valid crypt]

¯_(ツ)_/¯

[halcyon quarry]

This option makes the bot fire up super fast and delays the first LLM model loading.
Particularly useful for anyone using the Non-LLM features of the bot, but still occassionally using them.

[halcyon quarry]

Comfy member gives my PR a "Good PR" tag 🤩

[valid crypt]

I found the fix for this problem and did a pr 😉 (with some enhancement)

no idea how did this happen ._ .

[halcyon quarry]

I'll be looking into this code soon!

[halcyon quarry]

## Added a new Step 'load_llmmodel'

• Can load a new LLM Model
• Can temporarily unload current LLM Model
• Personally, I created this to temporarily free up VRAM for High VRAM step processing (ComfyUI generation triggered / etc)

Unloaded model will be lazy-loaded when next LLM gen is triggered

@valid crypt Is the STT configuration expected to utilize a second bot?

[halcyon quarry]

Make sure to pull changes I just made on the STT branch

[valid crypt]

just stole my own thing without cleaning

:P

works with the bot, but only perfoming stt with no actual input :v

you need to add the voice_recv import to the main bot.py

you can add conditions to these two

[halcyon quarry]

What does that do?

Does that have any bearing on the bot’s ability to play TTS on the voice channel?

Does TTS playback and STT need to be 2 separate voice channels?

If you can list the few other things needed to make this work, I’d appreciate that

[valid crypt]

no

[valid crypt]

no

[valid crypt]

run bot, run the command, good to go

[halcyon quarry]

Does that extra class assignment thing just make it an enhanced connection supporting the STT listener code?

[valid crypt]

¯_(ツ)_/¯

[halcyon quarry]

Ok seems like that

Thanks

Alright so what I’m thinking is that, when the bot sends the transcribed message to the channel, it adds the message ID(s) to a “transcribed_messages” list. In on_message() it will evaluate transcribed messages as user messages (not from the bot) - pop from the list as it finds them

Then process it as a user message

Sound about right?

[valid crypt]

im fine with everything :V

[halcyon quarry]

Just asking if it makes sense Still didn’t take a deep look into that STT code

The main problem was that the bot treated transcribed messages as if it were replying to itself, yes?

[valid crypt]

the stt code right now is literally doing stt nothing more

well it prints

[halcyon quarry]

Ok

Got it

[valid crypt]

when i was trying to make it work i tried let bot send it, but after all, you can implement it how you want

and this little file, (i used a second bot to read it and send it)

[halcyon quarry]

Thanks for taking the initiative to push that code to get the ball rolling on this feature

I suppose you were wanting a feature to prompt from file, so that you could adjust the text if it mistranscribed?

Perhaps instead of that- I’ll have an option (like review_stt_before_send) to not trigger on_message to acknowledge the message until a discord button is pressed (which would be sent with each transcribed msg)

If you need changes youd just copy paste the text block and fix it before sending it yourself

[valid crypt]

just asking for help so it can trigger a response in someway that is simple for me to make

write the content in a file is something i can do so... 👍

[halcyon quarry]

I like the idea of logging the transcriptions to file - but I’m not sold on reading from file to send

Well at least in terms of the STT. Prompting from file makes sense as a standalone feature

I think it would be more ideal to edit STT results via discord UI

[valid crypt]

writing the transcription to a file is just my skill issue

totally fine to be removed/modified

[halcyon quarry]

Was planning to 🤗 Alright so this logic I spelled out seems right to me

Need to look into this but maybe the Message object can be shallow copied and the User related variables can be substituted with the original user instead of the bot

So throughout the pipeline the Message is ensured to be handled as a user message without having to add special handling wherever it evaluates message.user

[valid crypt]

there's a voice channel id too, but i think that it is better to add it to a text channel (history and etc...)

[valid crypt]

what do think about the way it manages the transcription, right now the bot manages users independently, when someone finishes talking the message is printed
instead on my https://github.com/marcos33998/asr_discordbot/tree/v3.2-discord.py
i made the wrapped one as the only one

basically when everyone stops talking

instead of:
{A: hello}
{B: potato}

to:
{A: hello
b: potato}

[valid crypt]

also are we adding the pause tts when someone talks? in my repo that works as a second bot, it uses tags

[halcyon quarry]

That's already user configurable via Tags then?

It would be more flexible if it split the text per user IMO

The bot could concatenate the messages when I actually implement that feature...

[valid crypt]

that is in the case of the independent bot that manages stt

if im not wrong your bot cant trigger tags from its messages

[valid crypt]

ill leave both then

i forgot to add the blacklist command, in case there's a music bot or somethign

[halcyon quarry]

There's some bug in my script that's driving me nuts - every once in a blue moon the bot does not send content to channel from the Custom Commands feature. But then I add print statements everywhere and suddenly can't ever get it to happen again.

[valid crypt]

schrodinger:

[halcyon quarry]

alright got it to fail - just need to see the cmd log when I get home

seems to have some kind of correlation with activity timeout....

[halcyon quarry]

Ok the issue is that at some point I decided to store a discord object in one of my main settings attributes.
The Post Active Settings feature deepcopies the dictionary - which silently breaks when it hits that object

And it chokes up the background task queue

Which is where "send extra content" task goes

[halcyon quarry]

Resolved that issue finally - it's been bugging me for weeks

It would happen once the Auto-Change Imgmodels task executed - triggering Post Active Settings - which would then clog up the queue.

@valid crypt am I missing something?

It seems like all of these in config_stt.py are unused?

ASR_ENGINE = "whisper"  # "whisper", "omnisense" or "faster_whisper"

# Whisper configuration
WHISPER_MODEL_NAME = "small" # base, large, small,...
WHISPER_DEVICE = "cpu"
WHISPER_CUDA_VISIBLE_DEVICES = "0"  # GPU id to use

# OmniSenseVoice configuration
OMNISENSE_LANGUAGE = "auto"  # auto, zh, en, yue, ja, ko 
OMNISENSE_TEXTNORM = "withitn"  # withitn or woitn
OMNISENSE_DEVICE_ID = 0  # -1 for CPU, 0+ for GPU
OMNISENSE_QUANTIZE = True  # Use quantized model
OMNISENSE_MODEL_DIR = "SenseVoiceSmall"  # None for default

# Faster-Whisper configuration
FASTER_WHISPER_MODEL_SIZE = "small"  # Should be same as for Whisper, check faster_whisper documentation
FASTER_WHISPER_DEVICE = "cuda"  # I'll let you use cpu this time,"cuda" or "cpu"
FASTER_WHISPER_COMPUTE_TYPE = "int8_float16"  # Compute type: "float16" (GPU), "int8_float16" (GPU), "int8" (CPU)

[valid crypt]

they are used here https://github.com/altoiddealer/ad_discordbot/blob/stt/utils/asr.py

[halcyon quarry]

got it

[halcyon quarry]

I'm limiting it to just Whisper

I didn't quite finish cleaning up what I wanted to clean up, so I recommend holding off on any more commits for the moment

[halcyon quarry]

Basically the STT config options are just going into the main config file - didn’t finish this.

I defined the STT Blacklist file with Reality’s file management system

[valid crypt]

not the best option as stt, but the safest, easiest and stable option ._ .

faster whisper requiring some library and omnisense that got killed by its dependencies 😔

[late pivot]

@halcyon quarry any new updates?

[halcyon quarry]

Should update - especially if you use the Post Active Settings feature

STT is in progress thanks to Marcos effort / motivating me 🤓

[halcyon quarry]

Dear lord this code is so confusing

I'm going to assume it works and hope I never have to maintain it

[halcyon quarry]

Alright so I pushed a commit that resolves the config handling

[halcyon quarry]

For now, the target text channel will be server setting.
Could be updated for a per-user command where each user targets their destination

But that seems a bit slippery

@valid crypt at this point I need your help 🙂

As far as I can tell, the things I reworked are all still behaving the same.
Just changed some logic for loading / reading the config, and managing the connection state

So it seems like it creates the transcriptionsink and has it running, etc. The bot joins the VC.
It doesn't seem to be transcribing my voice though

to file or anything

[valid crypt]

lemme think

dont i have a lot of log spams?

[halcyon quarry]

Was it transcribing for you?

[valid crypt]

lemme pull your version

hmmm

[halcyon quarry]

J/W - had you tested the STT functionality on any of the versions you had pushed? Or was it just working in theory?

[valid crypt]

my last pull req is working

[halcyon quarry]

current commit?

Ah yours

Ok so I did bungle something

I've changed branches to your last push...

Well your blacklist command was broken 🙂 Cant user uppercase

[valid crypt]

this thing should be the first print from the moment that you start talking

the sink started without any problem and it is receiving audio packets

[halcyon quarry]

Yeah I'm not getting that on your latest commit version either

(after fixing the blacklist uppercase)

[valid crypt]

btw what do you mean about user uppercase?

[halcyon quarry]

command names need to be all lowercase

[valid crypt]

oops

i changed that without actually looking :p

[halcyon quarry]

obviously 🙂

tell me a commit version you know for sure was working

with actual testing

[valid crypt]

this one must be working https://github.com/altoiddealer/ad_discordbot/commit/c710cf321496e73459dc5b13ec292adb16003d2e

[halcyon quarry]

Or just let me know if you didn't actually have it working ever in this fork

[valid crypt]

if i have to be 10000% sure my first pr is working

[halcyon quarry]

The only print statements I get

that first error there is something I need to fix

but is unrelated

guess I'll just slap debug print statements everywhere and see where the processing is falling off

It could be that you have something extra installed that I'm missing? Or bad config for my setup?

Would be great if you could show a screenshot of your CMD window of it actually working...

[valid crypt]

got it

this one is working https://github.com/altoiddealer/ad_discordbot/tree/c710cf321496e73459dc5b13ec292adb16003d2e

[halcyon quarry]

testing...

[valid crypt]

i have a interesting guess

[halcyon quarry]

I'm thinking it may have something to do with my char not joining VC on startup -

I'm using my context command to join it

[valid crypt]

could be, mine joins on startup, but the lastes pull didnt work for me

[halcyon quarry]

hmmmmm

[valid crypt]

that is the problem 99%

[halcyon quarry]

still trying to debug this other issue atm with the bot owner check 😛

Ok resolved that one

Yeah, it's transcribing now

Ok I'm on the right track now I think

[valid crypt]

i dont think i messed up when i was copy pasting 🤔

[halcyon quarry]

The bug in my /bot_join_voice_channel command was still letting it join VC, but I think it fudged up my internal state tracking

so would not actually initialize the handler

or whatever

testing my current version again

So yeah, not working on my latest changes 🙂

At least I know I definitely screwed something up...

[valid crypt]

did you set wrap to false

i 99% screwed that thing too

[halcyon quarry]

Nah that's not it - it's defaulting to True

and user config is also True

[valid crypt]

true is working for you?

[halcyon quarry]

Might just have to repeat my steps again haha

Nah it's just not working, I broke something....

[valid crypt]

i had two bots running at the same time and so it wasnt working

[halcyon quarry]

The transcription was working for me, on that commit you confirmed

so I broke something between there and now

[valid crypt]

alr

[halcyon quarry]

I'll figger it out

[halcyon quarry]

It's not going to be pleasant but I'll figure it out 😄

Basically need to re-review everything I changed

[halcyon quarry]

painstakingly going step by step... and testing... so far making progress...

[halcyon quarry]

Ok got everything basically straightened out...

@valid crypt This is the new working branch
https://github.com/altoiddealer/ad_discordbot/tree/stt_v2

So the commands are updated, the blacklist thing, config options loading from config.py

[valid crypt]

what happened 😃 ?

[halcyon quarry]

Most of the STT stuff now all in stt module

There's something I don't understand about why AudioConfig just needs to be a class

[valid crypt]

uhhhh

[halcyon quarry]

But it does, so I just moved it directly in here and it's populated from config

ChatGPT was mystified too, I reprompted on this a few times and it couldn't pinpoint the issue

I don't know why but the whole things breaks if these self values are just set directly

[valid crypt]

😃 what?

lemme try

[halcyon quarry]

Something in the code needs these self values to be references to a class' attributes

[valid crypt]

so whole thing breaks just because no class?

completely break just because one of them has no class?

[halcyon quarry]

There's a lot of strange inheritance happening

with UserState

Anyway it just works like this so that's that

I stopped caring "why"

[valid crypt]

my independent bot works just fine

[halcyon quarry]

🤷‍♂️

[valid crypt]

also your are using an older stt code, i deleted audio normalization

[halcyon quarry]

I don't think the code was working after you deleted that

I worked from the latest commit that actually worked

You're welcome to try removing it but just test and make sure it still works 🙂

[valid crypt]

lemme open my recycle bin

this one is working

hmmm, i tried one thats before that removal too

[halcyon quarry]

That is the commit I started from

you deleted audio normalization later

At this point, I'm testing after any change is made 🙂

[valid crypt]

ill check this one, after fixing my command :v

D:

[valid crypt]

works, but the class sh*t is D:

this one i can actually set it

there is something wrong with my code

[halcyon quarry]

So what I'm doing now is passing in the server defined TTS text channel to TranscriberSink and it will simply await channel.send()

[valid crypt]

this one is on me 🥲

[halcyon quarry]

As far as I know the crisis is averted, so I'm just going to get the actual text part done now

[valid crypt]

ironically i wanst wrong, that commit contains the problem

@halcyon quarry just hardcode these 3 things 😔

[halcyon quarry]

Any clue where this is called from?

[valid crypt]

https://github.com/imayhaveborkedit/discord-ext-voice-recv

[halcyon quarry]

ok I see it's a bit limited in that it just receives the user

Gotta fetch all these discord objects whenever we want to send a message

DIY interaction

Pushed the concept of this - needs some refinement

• Going to make it an Embed instead
• going to add some sort of attribute or prefix for on_message() to handle it appropriately

[valid crypt]

i think you will need to add this to stt to stop stt

    def stop_listening(self):
        self.cleanup()
        for vc in client.voice_clients:
            if isinstance(vc, voice_recv.VoiceRecvClient) and vc.sink is self:
                vc.stop_listening()
                break

[halcyon quarry]

You don’t think my current implementation of stop_listening() works?

I certainly think it does 🤓

In toggle_stt_task()

[valid crypt]

try it 😏

[halcyon quarry]

I did earlier, seemed like it worked?

[halcyon quarry]

fixing a few bugs and such atm...

I feel like I'm losing my mind or something as the thing is not transcribing now

I'm also trying to be super quiet though as the kiddo is sleeping

@valid crypt if you get a chance try testing it

https://github.com/altoiddealer/ad_discordbot/tree/stt_v2

I added a thing to capture sent SST transcribed discord message IDs paired with the author names.
In on_message() if it matches the message ID, it tries replacing the Message object with a pseudo-cloned version where the author is swapped.

Probably needs a little more tweaking but should be on the right track...

[valid crypt]

how do i use it?

[valid crypt]

i was using the stt branch, maybe is because of that

[halcyon quarry]

You could probably just git checkout 6fe6853

I'm using Github Desktop, very easy to switch branches

[valid crypt]

right now im stuck at printing the result and nothing more

[halcyon quarry]

So it is transcribing and sending to channel still?

[valid crypt]

that's all

[halcyon quarry]

Maybe my bot instance at home doesn't have permissions to listen or something

[valid crypt]

i suppose im missing something

[halcyon quarry]

Well thanks, I can sleep easy knowing it's probably working enough that I don't have to scream

I'll be able to make more progress tomorrow... feel free to look into it

Oh -

I kind of ignored wrapped = True

[valid crypt]

isnt it supposed to do something more than just this?

[halcyon quarry]

Try wrapped = False 🙂

[valid crypt]

ah

[halcyon quarry]

I'll look into the Wrapped setting later

If anything we might want to wrap consecutive messages from same user by default

I expect some form of bug to occur when the internal Message is swapped with a psuedo STTMessage

[valid crypt]

is this how it is supposed to?

looks like stuck

[halcyon quarry]

It's halfway working!

🙌

It recognized the message as if you sent it yourself

[valid crypt]

hold your claps

the hello was from typing

[halcyon quarry]

Bah

[valid crypt]

that was what ive said

[halcyon quarry]

Ok - tomorrow's another day

gotta run!

[valid crypt]

i dont like 4am too!

before i die, stt branch

[halcyon quarry]

stt branch is dead 🪦

But yeah I understand why you thought that now 🤗

The reason it errored was because the discord.VoiceClient object gets patched with stop_listening() - it’s not in the sink object

[halcyon quarry]

Done

[halcyon quarry]

Embeds have a set_author() method that can use the member name and also avatar URL.

I also figured out why it was hanging - I was using get_member() instead of fetch_member()

Making a lot of progress here...

[halcyon quarry]

Alright so this is working very nice now

Faking the Message object is working great

I had to do a funny thing here... when it sends the message, on_message() checks it immediately before I can log it internally.

So I forcibly call it again. on_message() now only cares about Embeds if the msg ID is matched - once matched, it is popped from the meta.

So if there is some kind of delay or whatever, it will always ignore one of the calls

I pushed this update now

[valid crypt]

yay

[valid crypt]

nice

[halcyon quarry]

🤓

I love that this idea worked

The message is swiftly intercepted in on_message() and replaced with what is essentially a wrapper

The wrapper replaces message.author with the fetched member

which have very similar properties

and replaces message.content with the embed description field

[valid crypt]

are we adding interrupt tts with speech?

[halcyon quarry]

...lied?

[valid crypt]

?

stop the tts when i talk

[halcyon quarry]

I guess it could be configurable

by default people are allowed to talk over each other in voice channels

I'm removing the transcription logging completely btw

[valid crypt]

ofc

literal spam that helped me a lot along the road

[halcyon quarry]

I'm going to put some extra attribute in the history manager like from_stt or something

Another thing - I don't think it would be too difficult to make STT a "main API" function

As in, any STT API could be used for it

not limited to local whisper

and any other hardcoded local dependencies

[valid crypt]

stt api could be good
tts and llm are generated, although the promt might contain some private things
but your voice is real and sent, personally it is useful to host stt in lan, some dependencies can cause problem

like the omnisense that killed my tgwui while its dependencies also killed itself

[valid crypt]

i think that the last time when i was trying to break the toggle vc tag, i wasnt using tts streaming, here is an example

the silence paused the first audio, then i sent another message, played the new reply and played the second audio of the las reply

also with a interaction like this theres a chance that the toggle vc is not triggered by the bot it self

and this is a working example

[halcyon quarry]

I'm hardcoding wrap_messages False

[halcyon quarry]

Try to get it without punctuation next to it - that could be the issue

I don't think that's bot specific

Hey - so what other devices can be used besides cpu ?

FAILED for user xxx | Error: don't know how to restore data location of torch.storage.UntypedStorage (tagged with gpu)

[valid crypt]

cuda

and the cuda id is the gpu to use, usually the number is ordered by how close is it to the cpu

[halcyon quarry]

STT is pretty sweet

Thanks for kicking this off

[valid crypt]

task manager 😅

[valid crypt]

in normal conditions it works

[valid crypt]

i copied that one and other ones, all of them successfully paused

[halcyon quarry]

Please copy/paste your Tag definition

So I get a better understanding of what the expected behavior is / what is failing

I just pushed another update that greatly reduces the spam

[valid crypt]

you can try:
input,
tag,
trick the bot to say the tag
bot's tag ignored

also could be my fault as

[halcyon quarry]

It's probably your fault 🤗

I've got a headache thinking about it - if you want to investigate a bit

Can put some print statements in / around here

Oh... hmm...

I think you're trying to make me go crazy again with something too difficult to implement... let me see...

[valid crypt]

fixing*

[valid crypt]

this is how i encounter with that problem, might just me though

[halcyon quarry]

The toggle_vc_playback tag should be respected regardless of who sent the message

[valid crypt]

ill fix my tag problem and try it again

[halcyon quarry]

It's popped from the matched tags dict though - when it's applied - so if you are expecting it to trigger twice (once in each phase) that could be the issue

If "silence" is in both user text and bot reply

I think it's only going to apply in the first phase

unless you make 2 separate definitions, one for user and one for llm phases

[valid crypt]

you can try distil-large-v3.5, although it translates everything to english somehow

with cuda

also when using smaller models (only smaller madels have .en variant), and only speaking english, adding .en at the end of the model name is much better such as small.en or base.en

[halcyon quarry]

Added a from_stt flag in internal history

This is stable enough that I'm pushing it to Main

[valid crypt]

go try this:
make it remember a specific sentence with the keyword
make it tell a long story
make it say the specific sentence

[halcyon quarry]

The API stuff can come later

[valid crypt]

this one is quite stable, in this example, it skips the first audio

it just skipped the first audio

and kept playing the 1 and 2

[halcyon quarry]

STT Support Pushed to Main

If you want to look into that issue about the Tags, please, by all means.

I aint got time for that right now 😄

[valid crypt]

alr :v

[halcyon quarry]

I’m not sure if the match generic tags method is called for every message chunk

If it is, that would be the problem here

That tag parameter could just use yet instead of Paul, which would probably fix it

Get

Pop

[valid crypt]

i think the problem is caused by tts streaming, at least i think that this wont happen if i dont have streaming

[halcyon quarry]

Basically what I’m saying

Ok yeah it’s because it doesn’t “match tags” until it finishes all message chunks

[valid crypt]

ah

brain not braining

[halcyon quarry]

[valid crypt]

im typing them super fast, and the first one worked as expected (stopped), and the second one (keeps playing audio)

[valid crypt]

what im typing

but this is a different problem, this is audio keeps playing after pausing, as the tag is matched and triggered

[halcyon quarry]

When the tag is matched and then applied calls the pause method

That’s it it doesn’t try to enforce keeping it paused throughout the duration of the interaction

Search the code for the .play() Method and add a print statement to see when it is being told to play

Also, I am not in front of my documentation, but I think that the toggle pause tag does what it says toggles it on and off

So maybe you’re pausing it and then toggle it forget it unpause it

[valid crypt]

im only pausing and never unpausing

from what im experiencing, when it is paused the next interaction will turn it on, playing the new audio and then play others audios (the paused one is cancelled, and if split will play the second part)

[halcyon quarry]

What I’m saying is…

currently not paused:
toggle_vc_playback: pause
Pauses it

If currently paused:
toggle_vc_playback: pause
UnPauses it

[valid crypt]

that is not happening

[halcyon quarry]

Ok I think I understand

Every interaction runs .play() by default

You trigger a tag that runs pause()

Subsequent interaction may be just using .play() regardless of current pause status

… sounds about right?

The bot has an audio queue

While paused it may still be adding audio to that queue

[valid crypt]

check this

[valid crypt]

:O

[valid crypt]

ahhhh i understand it, i think

now i get why it is by chance, there is a chance that tag is matched before playing the tts, as im nearly spamming, increases the chance, and then the audio keeps playing, making it ignore the tag as is was already paused

i was just looking for a solution for interrupt tts with speech, and clearing the queue is essencial for this

[halcyon quarry]

Pushed an update that improves Voice Channel management

[halcyon quarry]

It can just be a config option, as this is something that would likely be applied universally not selectively

if config.stt.get(‘interrupt_tts’, False):

playback pause
Audio queue.clear()

I’ll see what I can do. Can pass some callback method when instantiating TranscriberSink

[valid crypt]

^_^

i had things like these, using tags with a second bot

[halcyon quarry]

I’ll be pushing a pretty nice STT update today… realized it’s super inefficient interacting with discord

Should have performance impact

[valid crypt]

😃 ?

[halcyon quarry]

guild, channel and member being fetched every interaction. But it can just store a reference to guild on init, maybe channel - at most we can just fetch channel/member once each and store reference

fetch is an API call

Me just not using 100% of my brain when writing that code

if not self.member
self.member = await fetch member

[halcyon quarry]

@valid crypt If you find a chance please test this branch here
https://github.com/altoiddealer/ad_discordbot/tree/dev

I'm not sure what the missing variable is, but the STT feature doesn't work at one of my locations but works at the other

[halcyon quarry]

This fetches member only once.
The TranscriberSink is init with the guild and channel and they’re never fetched again

[valid crypt]

i think that branch's stt is not working

[valid crypt]

dont know what do you mean by that, but mine just got stuckright here

[halcyon quarry]

Mine just stops at Starting Silence Monitor - but it was also only doing it when we definitely had this working

[halcyon quarry]

Thanks - gotta take another look at my changes… probably missed one little thingy

[halcyon quarry]

Im away but I think the only issue is this should be self.sink.guild instead of self.guild

Actually just pushed it now

should all work now - and I believe should also have a performance impact

[valid crypt]

yep

[halcyon quarry]

Let me know if you try it and it’s indeed working - in which case I can push to Main

Need to debug why exactly it doesn’t work at all on my one setup

[valid crypt]

works for me

[halcyon quarry]

Thanks! Pushed to Main

[halcyon quarry]

@valid crypt So you'll get a little chuckle...

I realized what happened

At some point I tested the blacklist command....

[halcyon quarry]

Nice TTS tools via Comfy

https://www.reddit.com/r/StableDiffusion/s/4BgNXxntFa

[valid crypt]

stablediffusion reddit 😅

[halcyon quarry]

I basically see double everything bc the ppl posting on comfy always post in both

The bot can use comfyUI as the main TTS

[valid crypt]

comfy just can do everything, although i discovered it when it was a small project, davinci resolve having similar modular thing just makes me dont want to try it :p

[halcyon quarry]

Ah man just realized that when I reworked all the STT code the second time, I did not include your actual commit history

[valid crypt]

a

also new exciting tts https://github.com/microsoft/VibeVoice feels like something that comfy will support although the small model is on the way :p

[halcyon quarry]

Yeah it’s already supported

So I was going to try and make STT also an API feature not just hardcoded - but there’s virtually no open source STT API projects

At least not for Windows

Whisper works good enough I think anyway

According to their github page the tiny/base models are almost as good as “small” despite being only like 50MB filesize models

[valid crypt]

even if you ask for paid stt services i only can say openai XD

[valid crypt]

the tiny for me is crap, the base.en is really good, but limited to english :/

and medium to large v2 are crap, making large-v3-turbo the best model with good speed and accuracy

[halcyon quarry]

I’m going to tweak the stt blacklist command so it can only blacklist ones self

Bot will be omitted from STT processing if not already

[valid crypt]

alr

[fickle ember]

If i wanted to blacklist a user from using the bot is there a way to do that

Im not very interested in a specific user trying to generate illegal content with the bot being hosted on my desktop🥲

[halcyon quarry]

I’ll look into adding a blacklist command tomorrow

Until then you can create a Tag to censor (block) if certain words/phrases are used

But that’s limited

You can also ban from server 🤓

[fickle ember]

ill see about this tomorrow

[halcyon quarry]

It will also check blacklist when checking if it should reply to messages - and simply ignore

[halcyon quarry]

I pushed the update to Main

Users can now be blacklisted / re-enabled by bot Owner

The only command which ignores the Blacklist is the /user_blacklist command - this only checks if it is being used by a bot owner

So, you can blacklist yourself for testing purposes and not have to manually fish your ID out of the internal database XD

Custom Commands feature also honors the blacklist

[fickle ember]

awesome

tried to update

(unknown location)

[halcyon quarry]

Did you use the updater script?

Bot has a few new requirements

[fickle ember]

yes

i think im fixing it

[late pivot]

@halcyon quarry latest update breaks Mythomax 13b btw

Im using ssh to do it btw

I just reverted back the version to make it work

22:14:50-917706 ERROR Error loading the model with
llama.cpp: Server process terminated
unexpectedly with exit code: -7

[halcyon quarry]

Latest version of what? TGWUI? or the bot?

[late pivot]

I updated both

Tgwui and the bot

[halcyon quarry]

I am fairly sure that is an error caused by TGWUI and nothing to do with the bot

[late pivot]

Btw do you know why the bot keep responding in emojis?

Idk why but it just responded in emoji for no reason

[halcyon quarry]

Make sure to update the bot -

If you continue to have that issue - use a different LLM model or check parameters 😛

About a month ago I pushed an important update

[late pivot]

What's the update?

[halcyon quarry]

I was monkeypatching TGWUI code chatbot_wrapper with a custom version that enabled TTS-streaming, with TGWUI TTS extensions

Upon close inspection, this had unintended side effects

I don't do that anymore - and with print statements I verified that prompts / params are now being processed identically via the bot and via the webUI

[late pivot]

@halcyon quarry the bot doesn't respond at all but it says it generated it, do you know why?

Nvm it just doesn't respond with replies

But then again I found out that it always responds with emojis

Never did that before

[halcyon quarry]

use /reset_conversation

[late pivot]

I did

Still response in emojis

I just updated the bot too

[halcyon quarry]

And if you use TGWUI - right now - this does not happen with same parameters, character, and model, and model loading settings?

[valid crypt]

i remember that i had that before, and i fixed it either deleting the history files or changing some configs

[late pivot]

It's the model

Other model works fine

Maybe latest bot update breaks it?

[halcyon quarry]

Whatever happens while using TGWUI - the webUI - this has nothing to do with the bot

so if that model sucks in TGWUI, it has nothing to do with the bot

[late pivot]

@halcyon quarry found a problem with the bot always responding with emoji

Even changed it to the examples and it still responds with rmoji

Emoji

[halcyon quarry]

Use everything same model, same character, same parameters, same model loading settings

In TGWUI

I do not think this is a problem with the bot

[late pivot]

@halcyon quarry I tried using the standalone instead but it shows this

[halcyon quarry]

That’s not what I meant 🤓

[late pivot]

I'm using ssh to host my bot so using the tgwui is impossible

[halcyon quarry]

I mean, try using the same settings and everything in Text Generation WebUI - not via the bot

[late pivot]

I'm gonna try the standalone instead

[halcyon quarry]

Ah

Standalone install is “no TGWUI”

[late pivot]

Yeah

But trying to use the standalone install doesn't work for me

How do I fix it?

File "/run/media/OneVloth/text-generation-webui/ad_discordbot/bot.py", line 49, in <module>
from modules.utils_chat import custom_load_character, load_character_data
File "/run/media/OneVloth/text-generation-webui/ad_discordbot/modules/utils_chat.py", line 7, in <module>
from modules.utils_tgwui import get_tgwui_functions
File "/run/media/OneVloth/text-generation-webui/ad_discordbot/modules/utils_tgwui.py", line 21, in <module>
import modules.extensions as tgwui_extensions_module
ModuleNotFoundError: No module named 'modules.extensions'
[OneVloth@ArchVloth ad_discordbot]$

This is the error

[halcyon quarry]

For some reason it seems to have switched the venv to the bot directory (as it should) but the internal flag is still trying to load TGWUI related code

Likely - an issue in the launcher script

I’m a Windows user and extending the launcher scripts for other OS has a lot of trust in ChatGPT

Note: Standalone version will not generate text.

[late pivot]

What does it do then?

[halcyon quarry]

Everything the bot can normally do, except for the TGWII integration

Interact with APIs - image generation, video, I mean you can generate text but it’s way more complicated to set that up with my bot

Try a more modern model, that Mythomax is super old

A TGWUI update probably broke support for it

Idk

Check Issues

https://github.com/oobabooga/text-generation-webui/issues/7222

@late pivot if you use ‘—flash-attention’ flag try removing it

[late pivot]

It's the text generation webui that's broken

[halcyon quarry]

Knew it 🙂

[calm rain]

emoji response usually means something's wrong with the chat format text

eg an old common example was having an extra space at the start of the bot reply portion of the format, for whatever reason LLMs tend to associate that messages starting with a space most likely have an emoji after the space

[late pivot]

@halcyon quarry do you still have tgwui 3.11?

Since I only have the 3.12 and the bot doesn't work with the portable version

[halcyon quarry]

Haven’t updated yet - will check it out tomorrow

[halcyon quarry]

…tomorrow 🤓

[halcyon quarry]

Ok I did not yet update TGWUI

Now I'll update TGWUI and repeat

So, seems same

[halcyon quarry]

I reviewed all the recent updated TGWUI code and made one small update - but this doesn't really affect anything

[halcyon quarry]

@late pivot

[late pivot]

That's weird ngl

For me it's just emojis

[halcyon quarry]

You are using an ancient model

[valid crypt]

clean install + llama 3.0 works

[halcyon quarry]

Update: been hooked on that game I was hooked on last time I had a 2ish month hiatus

[valid crypt]

:V

[fickle ember]

🐸

[late pivot]

@halcyon quarry is it possible to make the bots respond differently when talking to other people?

Like use different personalities when talking to person 2 but talks normally with person 1

[halcyon quarry]

It’s close, but I would need to add something else…

There is already a “tag” to filter history. So you could make it “per user histories” with that. But you would also want to trigger a swap character tag based on the user.

I could add a conditional tag like “only_for_users” which would only trigger for specific users

So you’d want to pair that with swap_character, and optionally also filter_history

I’ll see if I can add it tonight, that should be relatively easy

[halcyon quarry]

@late pivot I pushed an update that add a new tag param for_user_ids_only - which can be an integer like 12345677 (a user ID number) or a list of user IDs.

[late pivot]

Does it work with older version of tgui btw?

[halcyon quarry]

It's irrelevant to TGWUI really

[late pivot]

Great

So how does that work?

[halcyon quarry]

It's similar to using a trigger tag param

If you use this tag param, when the bot processes Tags it will skip tags that does not match the user ID

If the user ID is matched it will apply the tag

so you would include like swap_character in that tag

[late pivot]

Could you give me a template for it?

I don't really know how to use it ngl

[halcyon quarry]

It can be this simple

every time a message is received from the user, it will use an alternate character

[late pivot]

Do I include the character name extension too?

Like character.yaml?

[halcyon quarry]

nah

Just the filename without extension

I have example usage for swap_character in the default tag settings

[late pivot]

So I would need to create a different character to put that?

[halcyon quarry]

correct

[late pivot]

Alr thank you

[halcyon quarry]

That's really the only way you'll get significantly different results between users

which it seems like you are aiming for

[halcyon quarry]

I just realized that I actually did not "Push" that commit

Now I did - so that new tag parameter is now available

[fickle ember]

oh geez

im gonna need to update and figure out how to do all this when i have time

[fickle ember]

just updated

im seeing more red than im used to seeing

bot seems to work tho

i need to figure out how to use tags so i can make this happen

[halcyon quarry]

they're just warnings so you know there are new settings available, and that these defaults are being asserted

[fickle ember]

should i nuke my config.yaml

[halcyon quarry]

You can just copy/paste them in from the settings templates folder

the missing items

[fickle ember]

noted

seems to have worked thanks

[valid crypt]

this thing got a lot of updates and claims to be more realistic https://github.com/index-tts/index-tts
but still stuck at low sample rates, from examples i think it sounds like a real person with a bad mic so ill give it a 👍

with a simple review, it has no doc and no api by itself so...

¯_(ツ)_/¯

[halcyon quarry]

You can probably use it with comfyui via bot

If you have not checked it out yet, you should definitely play around with comfyui

There is probably a public workflow available to run that ttsmodel

[halcyon quarry]

https://www.reddit.com/r/comfyui/s/4yhg0np6K8

[valid crypt]

👀

[halcyon quarry]

Bot is still working fine on latest TGWUI

although I noticed I'm getting a strange reply when using swap_character tag - need to investigate what's up with that

[halcyon quarry]

alright I think I found an issue in the basesettings.yaml

nah it's fine

It may be that the Neuralbeagle model is finally just unsupported

There is an important update I just pushed

TGWUI updated their default chat_template_str value.

Apparently, it is malformatting prompts if providing the older value

@late pivot this could actually be the cause of your troubles

I was getting some very odd responses until I just tracked down this issue here

[valid crypt]

😱

[late pivot]

Oh

[halcyon quarry]

Just duplicate the chat_template_str value from basesettings.yaml from settings templates

Into your user basesettings

Unfortunately TGWUI internals require this param to be included, the way the bot uses it

[halcyon quarry]

If anyone is using "thinking" models

I invite you to test this branch (or just backup / replace bot.py and /modules/utils_tgwui.py)
https://github.com/altoiddealer/ad_discordbot/tree/thinking

This should extract the initial thinking and send it as a complete message chunk in a code syntax, followed by the actual response

[valid crypt]

What was the bot's max char per message? Thinking models can easily generate 2000+ chars also no one would listen to the whole tts right?

[halcyon quarry]

I am going to add a setting, that you can optionally discard the thought

Ignore the fight

Thought

There is no Max characters per message with the Botte

The send a long message function detects various discord, syntax and ensures that message chunks receive closing and ending syntax when splitting

Voice input by the way

[valid crypt]

I know that if it's a text file it can be expanded, idk if there's a different way to do that

[halcyon quarry]

I had thought about that, but it’s a bit too complicated for my internal handling

Mainly the settings that handle more human like responsiveness

The bot always generates text as quickly as I can. It just simulates the delay.

[fickle ember]

Whats a good thinking model?

[halcyon quarry]

gpt-oss-20b is pretty good

TBH - I'm struggling to get good output via the bot. From my experience the thinking models really suck in "chat" mode and need to be used in chat instruct or instruct

In the WebUI or bot, no matter. Very sensitive to parameters and prompting

[halcyon quarry]

Merged the thinking thing to main. Will improve it when I have time

I also fixed an issue for anyone using the recent "lazy load LLM" setting

When the model was "lazy loaded" the Instruction Template from the model would not take effect.

I have trouble getting good responses from thinking models in general

[halcyon quarry]

I have an idea - which is to extend the functionality of the latest and greatest feature (no one but myself is using) - the custom commands feature

So that custom "context commands" can also be created

such as one where you right click an image, or a text block, or a file, etc

[halcyon quarry]

alright so I just learned that the bot can have a maximum of 5 total context commands

which I already have hardcoded. So the usefulness of this is greatly reduced... even with an option to disable native context cmds

Sucks because I just got this working lol

[halcyon quarry]

Pushed update that adds a "disabled commands" config setting

[halcyon quarry]

This is a bit different from managing them via Discord as they never get registered in the first place.

[valid crypt]

do you have all nvidia libraries installed? i mean im not sure if they are included in the cuda toolkit and what if they are not installed

[halcyon quarry]

you asking me?

[valid crypt]

yes

[halcyon quarry]

I'm not sure 😄 But I recently did fresh Comfy install

it installs the correct and if not current then almost current libraries

[valid crypt]

like from code implementation whisperx is almost the same to whisper, and it seems to be better in every aspect https://github.com/m-bain/whisperX

wait, it requires cuda toolkit

idk if most of people have it installed 🤔

is it required by tgwui?

[halcyon quarry]

🤷‍♂️

[valid crypt]

from my quick investigation, some say tgwui need cuda toolkit, and pytorch installation includes runtime that is necesary for it, and whisperx seems to use pytorch so cuda toolkit is not required? maybe???

i dont have the enviroment to test it :/

[halcyon quarry]

I’m not really interested in working much more if any on the text to speech integration

[fickle ember]

Would be easier if there was a good way to collect custom commands

[halcyon quarry]

Well the system is designed to be pretty easy to set up really, just need to read about it on the Wiki, take a look over the StepExecutor page

I have a few good examples

Once you understand how it works it's extremely powerful

I have the "custom context commands" almost done, but as I mentioned it is limited to 5 total, and would require disabling native context commands in order to add these types.

[halcyon quarry]

Maybe Ill see if ChatGPT is any good with gradio coding yet

Gradio interface for settings management would be so nice

Gradio coding is quite different animal

[valid crypt]

👏 , idk why always gradio but

¯_(ツ)_/¯

[halcyon quarry]

I think bc pretty seamless with python

[halcyon quarry]

finally finishing up this custom context commands expansion

I had added a config setting to disable commands

Another thing to know is that a user can define a command that can run any number of user defined tasks

As in, the context command could be like run_task and it would just present a drop down menu of tasks to choose from - each could execute a "run workflow" step

there's a seemingly unlimited number of things this bot can do with an understanding of how the step executor system works

[halcyon quarry]

Need to test this before pushing to main but added another useful "step" called "interaction" which will allow you to get and use any value that is in the interaction which triggered the StepExecutor

ei can get the username string, guild name, etc - to use in subsequent processing steps.

[halcyon quarry]

Ok, realized this step is actually a bit redundant as it's basically my extract_keys step with a handler for attachments

I've changed up the logic so that attachments are preprocessed and passed in as context items (attachment_0, atachment_1, etc). Won't need to run any "steps" to actually get attachment data.
This will be detailed in the Wiki.

[halcyon quarry]

I've documented the Custom Context Commands feature / and updated the Slash Commands write-up

https://github.com/altoiddealer/ad_discordbot/wiki/Custom-Commands

Custom Commands feature Expanded to include Context Commands

• It's now possible to define your own context commands
• The discord object: Message (or User, depending on type) is passed into the command's processing steps.
• Attachments and text are captured in a cleaner, easier-to-use format

[fickle ember]

Is there an example of how this can be used?

[halcyon quarry]

Heya

I really need to whip up a video demonstrating it.
The config files include some examples in them

I just pushed an update which extracts the "Workflows" and "Presets" out from dict_api_settings.yaml to their own files.
These are now applicable to both API Settings as well as Custom Commands, possibly more things in the future.

I plan on making a solid example of a smart Custom Context Command which will use Workflows.

[halcyon quarry]

The best way to understand this is to just jump in and get your hands dirty. If you were to tell me a youth that you would like to do with a command, I could probably just walk you through it

use*

The main used case I had in mind when developing the whole thing is to load a payload update it send it to an API, save and send the results to channel

The difference between the slash commands, and the context commands is that the slash commands have a bunch of options you can set up. All of the inputs are selected or entered in these options before it is processed. For context commands, it passes the entire message object in as input. Anything can be picked up from it, such as the text attachments, etc..

For example I might decide to take a comfyui upscaling workflow and make it so that I only need to right click an image in the chat, and boom it will run it and send upscaled image to channel. This would be a bit more clunky as a slash command as you first need to save the image, then attach it

[fickle ember]

pretty much the only thing id be curious about making happen would be to get my bot to understand whos who in chat and act accordingly

as of right now the best i was able to do is supply that information within the character card which is highly inefficient

and ineffective

i remembered in the past you had mentioned id have to basically swap character cards entirely when given a keyword

this doesnt work very well for several reasons so having a way to do this dynamically would be nice

[halcyon quarry]

This is where I wish Reality was back in the saddle 😆

I'm looking into it - some history management needs to be done here and it's a bit tricky

[halcyon quarry]

Pushed an update which adds alpha version of "Server Mode"

If enabled:

• Intended to make the LLM feel more like it is part of a group chat, rather than just with "user"
• payloads/prompts will be slightly manipulated.
  • name1 value of the payload will be set as the server name, instead of the username. As far as LLM is concerned, each request won't be treated like a full blown conversation with that one user.
  • All prompts get prefixed with the sender's username
  • These will not be reflected in saved chat history

When using this setting, the character file should be tweaked to emphasize that the character is part of a group chat setting.
See new example character M1nty_ServerMode

I have a lot of plans to expand this.

@fickle ember this doesn’t solve your request btw - it’s a stepping stone though. I need to think some more about how to implement your idea

[halcyon quarry]

I think I'm going to end up moving this to a character behavior setting, rather than a global config setting

So this functionality is character specific

[halcyon quarry]

ok changing it over to a character behavior was actually super simple

this seems to be working pretty good so far

[fickle ember]

Well this is absolutely a great start

If it helps i can share how i sorta simulated it via a character card manipulation

[halcyon quarry]

There is already a method in place to filter history pairs for specific users (ei: current user). What I was thinking is that after every X number of exchanges, an internal prompt could request LLM to review and make a concise analysis of the user’s personality, etc.

On future analysis the internal prompt could include the current disposition of the user, for updating

And so the concise user description could just be always injected at the tail end of context (or appended to the “user_bio” param… essentially same thing really)

Need to double check if discord policy forbids profiling users though as that’s essentially what this would do, even if only to improve/personalize LLM response quality

I could make this an opt-in feature where users would have to use a command to toggle it on/off for themselves

Let me know if this sounds like what you’re looking for

[halcyon quarry]

This has inadvertently given me the idea to add an “idle_queue” for tasks which would only execute while there are no other tasks queued in the other main queues

[turbid jungle]

for some reason my model makes up user responses but only on discord, when using tgwui it's fine

[valid crypt]

That seems to be some wrong config of the bot, usually happens when you update from an very old version

[turbid jungle]

it's my first download, and yeah I figured but I'm not sure what to change to fix it

[halcyon quarry]

heya

[turbid jungle]

I tried a bunch of stuff but none of it worked

[halcyon quarry]

So you mean it replies to itself as you such as

Spitfire: What's your favorite car?

[turbid jungle]

yeah pretty much

Spitfire — Yesterday at 23:22
@goybot do not make up a response
bot APP — Yesterday at 23:23
I understand your instruction! My responses will be based solely on the text you provide from now on. Please go ahead and ask me anything.

User: What are some ways to improve my English writing skills?
AI: There are several strategies that can enhance your English writing abilities:

[halcyon quarry]

Are you using the same character card? Checked if your chat params are about the same?

[turbid jungle]

character card... is that the thing in the characters folder?

[halcyon quarry]

yea

[turbid jungle]

name: AI
greeting: How can I help you today?
context: |
The following is a conversation with an AI Large Language Model. The AI has been trained to answer questions, provide recommendations, and help with decision making. The AI follows user requests. The AI thinks outside the box. Respond only to actual user messages. Do not assume or fabricate what a user said.

pretty much the default one

but I (think I) set it to instruct

[halcyon quarry]

So I have a "getting started" wiki page that explains among other things how the settings logic works.
https://github.com/altoiddealer/ad_discordbot/wiki/getting-started

All of your baseline defaults can be set in user/basesettings.yaml. Many of these settings are only applicable for certain model types.
You can override any of these within your character card. See the example character M1nty for reference

The defaults the bot ships with are pretty close to native TGWUI settings.

If you want to use instruct, add the state config to your character, and add mode: instruct.
Or if you just always want instruct mode, just change mode in basesettings

As I'm writing this, I think I will add an option for mode when using the /character command

At any time, you can make a request using a different mode by using the /prompt command which includes some extra options

M1nty also includes a custom stopping string the bot has, which can be dynamically set as the username who is making the request

name1 will stop on the user's name. name2 will stop on the assistant's name. The bot example shows "\nname1","\nname2" meaning, it will stop if a new line begins with the username/assistant name. This would immediately resolve your issue, but there's likely something else with the settings, etc, causing the issue

If you update in the next day or two I'll likely push an update which adds "mode" to the /character command because that would be quite convenient

[turbid jungle]

trying the name1, name2 thing rn

custom_stopping_strings: '"\nname1:","\nname2:"'

trying this, but it says it's not a list

do I also have to fill stopping_strings?

[halcyon quarry]

It needs to be in this format:

stopping_strings: ["\nname1","\nname2"]