ad_discordbot (Fork of Fork of xNul's bot)

[valid crypt]

found a interesting tts

but not sure if it is for tgwui

https://github.com/legekka/GanyuTTS

it got the tag

[halcyon quarry]

I was having some trouble with the "is typing..." management I incorporated in each Task() object.
This is my new approach to managing it... this ensures that there is only one running "is typing..." task, driven by the state of the event flag

[valid crypt]

hardly digging around found this chinese? project

https://github.com/zixiiu/Digital_Life_Server

[halcyon quarry]

Off topic

[valid crypt]

absolutly not

it gives ideas

worked

so

can you add emotion recognition? ^-^

can the bot stream?

i mean if a discord bot has the ability to stream

or share screens...

i think ive never seen a bot doing that...

i found a part that could be useful for you, this guy divide text chunks by detection .!?

and send the text to tts

by my calculations, this makes tgwui only need about 4 tokens per second to speak fluently.

i exploded

[halcyon quarry]

Chunking text is planned already, won't be a problem

TTS - I don't plan on marrying one TTS API yet so it's not planned

Improved TTS at the very bottom of my list

[valid crypt]

what about this

[terse folio]

discord bots cannot send video in a voice channel

[valid crypt]

any way to make the bot play the tts on the computer too?

[terse folio]

it is possible using pyaudio library.
But no the feature is not part of the bot.

How would you want that implemented (if it's put on the todo list)

Like tied to a specific user so only some users can trigger TTS on the desktop?
It would be weird to hear replies to conversations you're not a part of.

Also joining discord VC already has that functionality.
What feature are you looking for that couldn't be done over discord?

[valid crypt]

discord has a delay

i want to plug it into a 2d model

and lip sync or something

if i do it by discord, i have a delay sending the audio to discord and receive and generate and send to discord

[terse folio]

I ran STT to textgen to TTS and the tts model was running on Google colab at the time.
I used discord for voice recieve and output.
It wasnt that bad!
Most of the latency probable came from colab and sending voice data across multiple pcs.

I think its just that streaming needs to be figured out and it would be an awesome experience!

5 seconds latency from voice input start to tts reply

[valid crypt]

imagin i have a 2d model with lip sync

it would be terrible

[terse folio]

Yea, I see, youd want the video synced to the audio

[valid crypt]

speaking but lip not moving at the beginning

[valid crypt]

that it

[terse folio]

One hacky idea is to create a virtual webcam and stream that to discord using your user account (if other people want to use it)

[valid crypt]

i just thought that if the bot can send the audio to discord why it cant just play it

[valid crypt]

smart

you got my idea

[terse folio]

Discord doesnt provide all the features of user accounts to bots

I imagine streaming video is computationally expensive for discord, so they dont want 1000s of bots spamming video feeds

Voice on the other hand is insanely small in comparison

[valid crypt]

👀

[terse folio]

An idea I once had was creating a few reaction images/gifs for an avatar and having the chatbot send those as emotional states change.
Not the same as lip sync, but a step in that direction ^^

[valid crypt]

cant we plug an user account xd

just some ideas from someone that has nmo idea

[terse folio]

Discord doesn't allow user bots, that's risky for account termination

Again, spam and people abusing the system

Using an account to stream video from a program doesnt involve automating the discord account, I think that's fine

[valid crypt]

¯_(ツ)_/¯

this is the game i talked about :v

https://helixngc7293.itch.io/yandere-ai-girlfriend-simulator

how can i plug this https://github.com/legekka/GanyuTTS into the bot with api

how do i plug something via api

[terse folio]

at the moment the bot uses tgwui to handle tts.
The bot doesnt make the tts calls itself.
it's the tgwui extension that hooks into the text generation pipeline that returns audio.

To get it working with the bot as is right now, one could make an extension for tgwui.
Or wait for api usage to be implemented

[valid crypt]

didnt you plug a tts with api

this look alright

[terse folio]

Yes it apis make things easy to use.
Its just some stuff needs to be shuffled around in the bot I believe.
So that it doesnt use tts from an extension and api at the same time

hmm, that could just be part of the setting.
Like instead of "Edgetts" or "alltalk"
it could be "api"
and another key specifies the url
Maybe another would be needed to specify the schema (Alltalk api will differ from Ganyu for example)

[valid crypt]

right not all the apis are the same

f

@terse folio do you control the bot.py?

like know how is the code

where is each thing

[halcyon quarry]

Reality has the authority to change anything in the bot, but they’re not super familiar with everything in it. Especially now that I just revised a most of the code

Reality wrote the history manager, and really helped with improving settings management

[valid crypt]

i suggest a file that divides the code in several part (tell the line of big chunks)

6000 lines is somthing that i cant ask chat gpt

27k words is nearly half of a book

[terse folio]

theres a function for sending the tts in the discord_utils.py file iirc.
you can search that function name to see where its used.

[halcyon quarry]

Yes the bot has a zillion features and so also has many lines

[valid crypt]

way too many line for a beginner (gpt user)

gpt handles around 30k characters

about 600 lines

[terse folio]

using tags to trigger tts would be an interesting solution

already using the existing "do this after that" type logic

Tags just need a tts function implemented

[valid crypt]

?

[halcyon quarry]

Will do 🫡

the “Tags system”

[valid crypt]

just something that i did not understand

what the difference between that and the /speak

brain isnt braining

sorry guys need more detail

[terse folio]

/speak is a command on discord, a way to interact with the bot.

tags are a sort of programming/scripting language where you can put various parts together.

I suggested using the tag system to implement API driven tts to leverage the already existing code instead of adding new code that decides if tts should be generated on each generation

[valid crypt]

did not understand "instead of adding new code that decides if tts should be generated on each generation"

in a conversation every reply should have a tts

[terse folio]

at the moment the bot doesn't not handle TTS at all.
The bot tells TGWUI to load the TTS extension.
That extension hooks into the text generation part of the code so when you generate text, it also outputs an audio file.

[valid crypt]

yeah

[terse folio]

To use an API for TTS, we need to start handling the TTS (which we were not)
so it's not a simple swap.

[valid crypt]

tag to handle stuffs thats outside of tgwui

[terse folio]

Thing is, the tags system already works when text is being generated.
It run other code before/after generation

So instead of reimagining how the bot generates text.
just make a little addon to the tags to do tts there

[valid crypt]

https://tenor.com/view/skb-tutorials-technology-star-gif-12272656

one more question, why the bot cant have a webui?

[terse folio]

the bot just runs TGWUI

which is the webui

you could disable the webui, there's a flag for it

[valid crypt]

so all the time i could just type 127.0.~~~ and change some settings?

[terse folio]

not bot settings

tgwui settings, yes

Oh, you were asking where there's isn't a webui for the bot

#1154970156108365944 message
this might be your answer

[valid crypt]

i just want to use tgwui with the bot

some ui to make setting more accessible is good though

[terse folio]

it should work.
The webui doesn't work for me with the bot running oddly, I don't know what the issue is

[valid crypt]

that is what im asking

[terse folio]

you'd have to ask @halcyon quarry about that

[valid crypt]

the api is working after his fixes

just not sure if it is working working

oh wait

i didnt plug the ethernet to the server ._ .

there is no such a message that tells you the url

and still can open the webui

[terse folio]

I have the same issue, I'm not sure what the issue is

[valid crypt]

nah api working but not working

[terse folio]

Will need more details than that :3

[terse folio]

something is already using port 5000

do you have another copy of tgwui running/another bot?

[valid crypt]

nah

just bcs the bot loads the extension twice

[terse folio]

interesting

worthy of a bug report!

[valid crypt]

just have the message bug and not working

the webui has no message and not working

and i still dont understand why complex memory wont work

cant ask gpt bcs the code is 6000lines long ._ .

[terse folio]

does it work with normal tgwui?

[valid crypt]

yeah

altoiddealer seems to know why but he didnt explain the details

[terse folio]

My guess is addiscordbot bypasses something internally where the complex memory extension would hook into

[valid crypt]

at the beggining extensions that have setup in their code dont work

and what altoiddealer did was ignore and instead of skipping retry

fixed edgetts but

complex memory is strange

these are his words

/v1/chat/completions i found it

the bot has now a release 😱

[halcyon quarry]

Hi

[valid crypt]

@halcyon quarry how about stop "is typing..." for a few second if the message is taking too much time

[halcyon quarry]

There’s two reasons the bot doesn’t use the API

[valid crypt]

not about that

is about why api of the bot isnt working

as you fixed the openai extension

there should be a api

and one issue, extension loading twice

[halcyon quarry]

That’s a feature

[valid crypt]

alright

[valid crypt]

answer this one then

[halcyon quarry]

I don’t like the API.
The API does not return a TTS response, so I can’t manipulate it - send it to voice channels /etc.
I monkeypatch load_extensions in order to update extension params anytime we want such as custom TTS voices etc - can’t do it while using the API.
The API is extremely strict about the payload. I don’t like it

[terse folio]

They mean that the api doesn't work with the bot running

same for me, when using my other apps after working on the bot

i have to shut it down and start tgwui normally

[halcyon quarry]

I don’t really know what I’m doing, per se

Most of the TGWUI code comes from the TGWUI server.py file

Part of that code could block additional instances / the API / etc

[terse folio]

it's possible something is going wrong that breaks some extensions

Also, I'm pretty sure the webui itself doesn't load?
(at least I remember having an issue, maybe it was that the port was different)

i'll check again later

[halcyon quarry]

It’s likely because the bot uses the shared module

[terse folio]

also you 100% can write your own extension that returns whatever you want over api.
I've done it before!

I think it was before the openai extension,
I made an endpoint to stream text per sentence, also using uploaded regex as a stopping string on server side that could match on those sentences/the full string

It was to match things like "if the next sentence doesn't start with AI: then stop"

[halcyon quarry]

🤯

Since I decided to start working on the new behaviors, I’ve been digging myself deeper and deeper into a hole I may never find myself out of

[terse folio]

Ohno!
the ever lasting recursive hole of adding more detail!

run while you still can

[valid crypt]

what did i miss

[valid crypt]

whoa

[terse folio]

work on things from a more general/broad perspective, so you don't lose vision of where the feature should be going

get yourself a nice todo list ^^

[valid crypt]

whoa

[terse folio]

Btw, I like Obsidian.md + Kanban plugin for todolists

😭 "todo 1 year ago"
procrastinated too long

can make categories and move notes around/mark them as complete ^^

[valid crypt]

im having a ton of tabs lately...

closed a lot of them already

so what gpu are you using? 👀

[terse folio]

I'm using a 2080ti

But I'm also working on a DNS like system to easily connect other pcs without bothering with networking/ips and ports

[valid crypt]

👀

[terse folio]

so a lot of these models would be distributed

[valid crypt]

?

[halcyon quarry]

I’m still stuck making the things work I added recently

[valid crypt]

what

i can test

i good at finding bug

maybe

[terse folio]

DNS is short for Domain name system.
That is how your browser can connect to "google.com" and some DNS host will resolve "google.com" is actually 142.250.65.206 and then send your packets there.

[valid crypt]

i know that

but what do you mean with connect them

[terse folio]

My idea is similar, running little programs on one's pc that registers your machine as a worker to a central point where you can search by worker type without caring about ips

and all that tunnelling gets resolved automatically

On top of this, I can write queries.
Like: Connect me to 3 PCs running stable diffusion that have controlnet and are using model Y

[valid crypt]

im using remote desktop

[terse folio]

Awesome ^^

[valid crypt]

im using sd on the desktop

and host the bot on the laptop

[terse folio]

Yea, so you know how with cloud computing you don't really have to deal with scaling yourself.
you set up an image and tell the system to start new instances when you need more.

Here, you can connect more machines any time to the system similar to a home hosted cloud

[terse folio]

Yea, I read about that, impressive the laptop runs at such little power draw!
Might have to look into that myself one day!

[valid crypt]

still not getting the idea

servers can add more machines to get more speed

but to be able to connect you need pay a lot

and not getting much

[terse folio]

I do tests on my pc,
I sometimes use my old laptop to run some models on CPU.
I can also use other PCs from the family if they're not busy.

Issue is, normally you have to change the ip/port in your app to point to those machines if you decide to change where a model is hosted.

[valid crypt]

parsec can remote control without dealing with ip

[terse folio]

I'm not talking about remote desktop

[valid crypt]

im super comfortable using it

thats true

what about vpn

if you dont need tooo real time vpn is a solid option

[terse folio]

Vpn is a means of connecting to another network securely,
Yes that's great for security, and could also be used with the system possibly.

The system acts as a list of known workers (your pc hosting a model)

As your pc runs the worker client, it becomes part of the network where it gets a load of requests to process until it disconnects any time

(very simplified example)
instead of:
config:

• SD: 10.0.2.1:7680
• tts: 10.0.2.3:10000
• textgen: 10.0.2.2:5000

you just:
config:

• SD: "jugernaut model"
• tts: "use xtts"
• textgen: "solar10b"

And the system figures out what PC to route your requests to

[valid crypt]

oohhhhh

thats why you said dns

i can remember my internal ip adress

and you can change the port to ez numbers

[terse folio]

yup, you absolutely could do it manually.
My goal is to make it so I can turn on any pc, start the program on any pc and it knows what it can do and is already processing data!

[valid crypt]

you want redirect url or a website that has everythin

g

you are goin to use like the tgwbui or something like the discord bot

[terse folio]

yes, sort of like a redirect.
you send your textgen/tts/imagegen request to the central point with some other paramaters like what model you'd prefer ... etc
And it picks the best device for the job and sends it back.

[terse folio]

I will be working on a tgwui extension so it can join home hosted versions of this network

[valid crypt]

i want a chatbot, server: gotcha, activating laptop1?

[terse folio]

wake on lan would be a cool feature.
I wonder how you'd go about security with that.

[valid crypt]

wake on lan is not that hard

[terse folio]

how do you get past the login and start up programs?
idk, never done this

[valid crypt]

if you mean turn on the computer it is easy

[terse folio]

yea, but I mean before you start running programs, you need to be logged in as a user right?

[valid crypt]

follow a simple yt tutorial

[terse folio]

will put that on my todo list

[valid crypt]

turn on the computer is easy

log in is pretty hard

[valid crypt]

you dont have to log in

windows it self has a feature

like if i turn on the pc execute this file

works in the background

i think

i still can access to tgwui with the laptop locked

[terse folio]

more like:
You have the chatbot running on your pc, but a friend comes over and offers to add some extra compute to run image generation as well.
The chatbot being subscribed to events from the "Resolver" system, sees that it now has image generation ability, and now lets you prompt images.

[terse folio]

Good to know, that would be useful!
I was thinking of traveling sometimes and wanting to get textgen loaded on my pc haha

[valid crypt]

maybe you have to leave your computer on 24/7

i think the logging after a reboot is special

[terse folio]

the gpu alone uses 33w idle

unlike your laptop at 2w total? idle

[valid crypt]

2.4

[terse folio]

I wish

[valid crypt]

:v

my 3060 uses 26w >_<

[terse folio]

Sleep/hibernate isn't the best option, it breaks a lot of programs for me

[valid crypt]

hibernate never broke something for me

[terse folio]

might have been sleep then, okay

[valid crypt]

so i never turn off my computer

hibernate saves unloads everything into the drive

so i dont think i will break anything

even someone changed the cpu during hibernate

[terse folio]

Yea, keeps a snapshot of everything

[valid crypt]

https://www.youtube.com/shorts/7LfRiBJ_EWg

video: not very useful but more you know

[terse folio]

it is cool ^^

[valid crypt]

but remote desktop is really useful, suppose that something went wrong, you can just fix it

and gonna try something...

[terse folio]

yea, each tool has their own use case

[valid crypt]

alright

programs can start in the background after a reboot withou login

program something with Windows Task Scheduler
send a magic packet and everything done

and remote desktop to make sure what is wrong and you can travell with tgwui

[terse folio]

Mhmm, mhmm ^^

[valid crypt]

just connected to my tgwui with mobile network using vpn

another good thing about laptops is that the system wont use vram if you have a igpu

well desktops could

but i prefer cpu with f

:v

[terse folio]

yea, my pc was the first time I built a pc without an igpu, was confused why the monitor wasn't working haha.
was many years ago

didn't know that was a thing back then

[valid crypt]

my 3060 is at 3/12 so i only have 9gb

but my laptop has a 4060 at 0/8

[terse folio]

running 4k monitors or something?

[valid crypt]

not a 4k

but got a lot of stuffs opened

[terse folio]

ah, makes sense

[valid crypt]

oh game running in the background 2.5gb

i feel pretty normal using 2~ gb vram

[terse folio]

I get around 1.3-1.4gb idling with no games/video editor open

typically using vscode

[valid crypt]

sus

[terse folio]

yup yup

[valid crypt]

is flask api famous

[terse folio]

flask and dijango are the 2 more well known python web server frameworks

[valid crypt]

what api do alltalk use?

[terse folio]

Personally I think flask is simpler and to the point, but the alternative is probably more feature packed

[valid crypt]

their own api?

[terse folio]

it uses fastapi, which runs on unvicorn which is a newer??? python webserver

also faster than flask

[valid crypt]

faster?

api has speed?

[terse folio]

it's probably compiled to C++? or C i forogt in the the backend.

no, the unvicorn webserver

[valid crypt]

👍

[terse folio]

Some python libraries like Numpy are compiled and can leverage C speeds that are really really fast

That's why it doesn't really matter what language we build our AI apps in

well, if you use Python

it's using compiled libraries, so you get the benefit of both worlds!

Coding speed, and execution speed

[valid crypt]

https://github.com/skshadan/TTS-RVC-API
this is fastapi?

oh coqui

[terse folio]

if you can go to ..../docs it will tell you

http://127.0.0.1:8000/docs?

for example

fastapi creates a documentation page from your code.
and also helps validate your requests to adhear to your schemas

[valid crypt]

do you have a project with fastapi support?

[terse folio]

sure, tgwui openai api extension is one of them btw

you can go to localhost:5000/docs

to get the tgwui docs

[valid crypt]

so how i plug it? :v

[terse folio]

hmm?

[valid crypt]

i ask you because you said something like you use alltalk by api...

?

[terse folio]

I checked out alltalk and found that it uses the same library as I.
But no, I wrote my own little api for TTS generation based a project from github

[valid crypt]

alltalk and coqui are pretty similar maybe

[terse folio]

coqui is part of TTS (lib on pyip)

TTS is used by alltalk

I believe coqui was an early version/is XTTS?

which is one of the engines supported by TTS (the lib)

it's the one with the frog logo iirc

[valid crypt]

😵‍💫

[terse folio]

yes, coqui created the TTS lib

and xtts

okay, that makes sense about their relationship now

[valid crypt]

gonna try easier stuffs

like this one :D

https://github.com/Arondight/vits_api_tts

[terse folio]

Alltalk looks easy imo, it has a standalone installer and can be ran as an extension for tgwui.
I tested it and it worked fine

[valid crypt]

but this is not alltalk

i just dont like alltalk

[valid crypt]

this looks good and i like vits

[terse folio]

this would be a step after tts generation.
RVC is a voice changer

[valid crypt]

one of the reasons to try first https://github.com/Arondight/vits_api_tts

[terse folio]

Sure!

[valid crypt]

it says tgwui extension

extension 🤩

[terse folio]

try it with tgwui standalone first so you can rule out any potential bugs as being from the extension

[valid crypt]

i just believed the readme and almost killed my self

[valid crypt]

i hope this is easy 🤩

backup just in case ._ .

[terse folio]

compiling things is always scary, especially on windows,
had a lot of bad experiences with things constantly failing.

turned out I had some paths wrong, it was using some wrong version of buildtools and couldn't find required libraries

[valid crypt]

ill tell you that it has a step by step instruction

looks like 100% beginner friendly

who is really friendly was chatgpt ;-;

[terse folio]

was chatgpt able to help you install it?

[valid crypt]

when projects have short codes ._ .

[terse folio]

ah, was going to say that's impressive for something it never seen before

[valid crypt]

the longest was 800 lines

i copy the log and send it to chatpgt with the code

and magic

[terse folio]

Hmmm

[valid crypt]

when you dont know about code you cant select a part of the code and send it to chatgpt

have you ever seen that kind of folder in github?

[terse folio]

I just went through the repo, that's cool

it's a link to another repo

[valid crypt]

thats the problem

one of the problem

[terse folio]

git doesn't download it properly?

[valid crypt]

no

nathing

no

thing

and that is not a valid link

[valid crypt]

the repo

the project is cool but pain

at least i ve done it :)

[terse folio]

git clone --recurse-submodules <url>

[valid crypt]

thats why it was a betrayal

[valid crypt]

if you dont know what are you doing (completly new)

[terse folio]

i searched "how to git clone links to sub git repos"

[valid crypt]

https://tenor.com/view/eric-cartman-crying-you-betrayed-me-gif-26835238

"step by step'

[terse folio]

mhm, unfortunately a lot of programs aren't that friendly.
I've spent months learning/researching something to get some things working in the past...
yea big pain

[valid crypt]

but the project is very cool :v

my 2d model idea was inspired by it

[valid crypt]

with simplified explanation and showcase

you can decode it with subtitles

[terse folio]

awesome :)

[valid crypt]

.

[terse folio]

hmm, i'd expect the "deployment package" to already have a ready made environment in it

[valid crypt]

buddy could show me the quick deployment first

definitely gonna go for the quick one

[terse folio]

ohno compiler errors

you cut off the error above

[valid crypt]

nah i choose the quick one

:)

[halcyon quarry]

So I finally have the new behavior timing working correctly. Perfectly.
It's just a bit lame at the moment, until I update it further... streaming responses, replying to multiple messages at once, etc

[valid crypt]

"replying to multiple messages at once"

https://tenor.com/view/spongebob-computer-fire-gif-21352118

https://tenor.com/view/everything-is-fine-everything-is-ok-fine-okay-everything-is-okay-gif-8096560573170242651

nah its fine

https://github.com/Artrajz/vits-simple-api got simple in its name i see a hell long readme

[halcyon quarry]

Pushed some updates to Main

This includes the thing that made the RVC model output work for edge_tts

[terse folio]

did you get my message about the mp3 file actually being a wav from rvc?

[halcyon quarry]

Thanks for the reminder 😄

Looking into that now

[valid crypt]

died

[terse folio]

kinda confused what the code screenshots are about

[terse folio]

you have no voices?

[valid crypt]

it should be using the api

https://github.com/Arondight/vits_api_tts this is the extension

[halcyon quarry]

It isn't populating any voices

[valid crypt]

which should using the api to do something with https://github.com/Artrajz/vits-simple-api

[terse folio]

yes, but the program is complaining about something missing

[valid crypt]

i cant even launch tgwui

gonna cry to chatgpt

[halcyon quarry]

I really don't know what you mean by this... are you saying it is producing a .wav and saving it in an .mp3 container?

[terse folio]

it's producing a wav, just named ".mp3"

you can rename files to any extension, doesn't mean they'll work.

but many do.
like if you rename a .gif file to .png, or jpg?, it will open as a still frame

[halcyon quarry]

Oh yes I see

you are correct

[valid crypt]

i know i know betrayal

[valid crypt]

😨

[terse folio]

I've never used the repo, I have no idea what's going on there

[halcyon quarry]

Good eye on this lol

[valid crypt]

never gonna belive for beginners

[terse folio]

that codec error is what led me to checking the file with ffprobe

[halcyon quarry]

The suspicious thing is the other outputs are < 100kb while the output from the RVC model is > 2MB

[terse folio]

yea

[halcyon quarry]

our good pal chatgpt helped add a check to identify and fix improperly saved files before processing

[terse folio]

what's the link to the rvc extension you're using that's bugged

[halcyon quarry]

It now saves that to wav before processing correctly

[terse folio]

maybe there's a setting to change the output format

[valid crypt]

our good pal chatgpt is on his second try ;-;

[halcyon quarry]

It's this extension https://github.com/Unorthodox-oddball/text-generation-webui-edge-tts

[valid crypt]

this has no forks

no way to install a diferent one

[terse folio]

it is the fork

[valid crypt]

:O

whaaaaaaaaaay

whaaaaaaaaaaaat

right

[terse folio]

the issue

[valid crypt]

now i remembered everything

[terse folio]

you could make a pull request

you could also make it save as an mp3

[halcyon quarry]

ideally, this stuff should always save as wav

[terse folio]

quality loss is minimal, and it's not like tts is super high quality to begin with

depends on the use case

like sending over network

would want some compression

[valid crypt]

you have to mention https://github.com/Unorthodox-oddball/text-generation-webui-edge-tts bcs the original one has problems

[halcyon quarry]

yes that is the link I just shared

[valid crypt]

ayo

i mean here

or somewhere

[halcyon quarry]

I'm pretty bad at updating the Wiki consistently 😛 Will do that now, thanks for the nudge

[valid crypt]

for some reason the original one is on https://github.com/oobabooga/text-generation-webui-extensions

[halcyon quarry]

The RVC stuff seems pretty interesting, I may start using edge myself 😛

[valid crypt]

buddy's webui is in chinese

im gonna kill myself

[halcyon quarry]

sounds like it's coming from a tin can but the voice model you shared definitely sounded like the source character

Edna Mole

[valid crypt]

i have more

https://tenor.com/view/420-immigration-justice-marijuana-legalize-marijuana-weed-gif-26872879

[halcyon quarry]

Finally fixed the continue task from endlessly typing (and possibly others)

[valid crypt]

i think the extension was fine, i have to start vits then tgwui but i can find models ;-;

gonna kill myself

https://tenor.com/view/zzz-zzzero-zenlesszonezero-goodnight-goodnightchat-gif-3570746195542614340

[terse folio]

said any vits model, try huggingface

[valid crypt]

i already downloaded 2 different model

[valid crypt]

1 randomly and stole one from this guy

going for the third one

still no model

i downloaded the realease

idk what is wrong

gonna try renaming

[valid crypt]

readme example

[valid crypt]

dont work 😭

cant even understand why

[vestal python]

Any thoughts on llama 3.1 8B for the discord bot. I am setting up a new mini server with one of those p102-100 10GB GPUs under PopOS (Seems their preconfigured nvidia drivers works out of the box for it).

Was thinking with its 128k context limit, if it was any good for Q5 with 20k+ context for memorizing a lot of conversations for the discord bot, and coherent enough to use.

[valid crypt]

thats more on the model than on the bot

if you are satisfied with its performance in tgwui then there shouldn't be any problems

[halcyon quarry]

For the first time, making a post on reddit for the bot.

https://www.reddit.com/r/StableDiffusion/comments/1efaecp/ad_discordbot_not_an_ad_please_read/

Appreciate any upvotes / supporting comments 🙂

[halcyon quarry]

started working on streaming text responses 😮

[burnt patrol]

Yes!!

[halcyon quarry]

Whipped up a nifty new Tag useful to those using the bot as a SD tool, last_img_payload
It can can be used in two ways - in both ways it is applied first before other payload related tags (can be updated).
last_img_payload_dict: true will use the entire last image payload as the current payload.
last_img_payload_dict: ['cfg_scale', 'seed', 'positive_prompt', etc] - As a list, will use only those specific values from the last payload.

Personally, I'll be using this with flow tag to do upscaling / regenerating txt2img / etc

[halcyon quarry]

There were quite a few hidden nuances when doing this, yeesh.

[valid crypt]

is possible to add edge setting in the setting.yaml

¯_(ツ)_/¯

[halcyon quarry]

As character specific settings, yes

#1154970156108365944 message

Those are all valid parameters you can use for the character's edge_tts settings

[valid crypt]

not that bad being 8b

[vestal python]

I'm waiting on an adapter, but I'll be trying out some setup in the next few days and give my discord not an upgrade.

New/old hardware change

[halcyon quarry]

Added Gradio interface for settings management to the to-do list. Will get streaming responses working first

[valid crypt]

do you guys use cmd or powershell?

[halcyon quarry]

cmd

Progress ❔

[valid crypt]

i see no much difference

[terse folio]

confused, what am I looking at?

she takes a deep breath

that's quite a read

[valid crypt]

it feels so good

[halcyon quarry]

First attempt at streaming response lol

[valid crypt]

to hear that again

[halcyon quarry]

Except it sent a message after every update from chatbot_wrapper

[valid crypt]

understood

[terse folio]

I'm not sure how streaming works internally, but I think in a very old version of tgwui, streaming would send the entire context back with the next token

[halcyon quarry]

Yes if a print statement is added in there it prints an ever increasing text string

[valid crypt]

change the rate can improve that but wont solve the problem :/

make it edit

[halcyon quarry]

What I plan on doing is detecting sentence completions / line breaks, and use a character setting value to roll random and see if the message should get chunked

[terse folio]

oof,
you'll want to add some note of the last token, and split the text at that point or something.

[valid crypt]

if you are not planning to make it be able to plug into a tts

edit pretty much solves it?

[halcyon quarry]

The TTS response is going to be returned as current

not in separate chunks

[valid crypt]

then cant you make the bot edit the message?

[halcyon quarry]

For simplicity I plan on logging the complete response, and collecting all the chunked message IDs as related_ids

[valid crypt]

?

[halcyon quarry]

(backend talk)

[valid crypt]

ill let you cook

[halcyon quarry]

The bot can edit its own messages yes

Which it currently does for Regenerate Replace, or Edit in History

[valid crypt]

from what i see the last line is the finished reply, if streaming responses you mean updating the message as they are generated, then you can just make the bot edit the message and it is pretty much done

[halcyon quarry]

This text streaming stuff is going to work, just need to fiddle around with it more.
Then overcome the challenges of all the other recent crap I added (possible delayed response behavior, typing speed, etc)

[valid crypt]

true

[halcyon quarry]

Just wanted to share funny outcome 😛

I had put some code in that was supposed to detect line breaks but it didn't work so it did send a discord message for every new token

[foggy cradle]

@valid crypt I was thinking about building a custom long term memory system from scratch. It seems like you've exhausted a lot of options looking for something decent, have you had any luck at all or should I prioritize that long term memory system?

[foggy cradle]

This is how that made discord feel inside

[vestal python]

Yeah.. I had a problem earlier with chat history from streaming where it was saving a new chat history in the array everytime a new word was streamed in the same sense..

[halcyon quarry]

@keen palm had chimed in to remind me that I added a pair of “tags” (in the Tags system) to mimic complex memory

Only downside is they are not shared with the actual complex memory extension, they’re managed separately

Just had an interesting idea… if complex memory is enabled, it could try finding the complex memory file for that character and import them in tags format

Fun idea but I wouldn’t want it to write the changes to file

Tags system version is more comprehensive anyway, you can have comma separated list of trigger words

[valid crypt]

same thing buddy

you can have comma separated list of trigger words

[valid crypt]

for tgwui you can use complex memory, but for the bot you better use the tag system

i plan to try more extensions but, complex memory seems the easiest and good enough but you cant use the original one but one of the forks

[halcyon quarry]

OK well the tags system method is more comprehensive than for the fact that you can use similar trigger phrases in separate memories and have one of them Trump the other

For example you could have a trigger for a memory that is Bob and then you could have a different memory with the trigger phrase Bob is sad or something like that and have a Trump the more generic casual one

Using microphone to text by the way

[keen palm]

Still waiting for the relationship type triggers, so if Bob AND Doug are triggered, that specific memory will be given to the bot

[halcyon quarry]

Multiple trigger phrase

Will add that to the to do list and high up

I think I would have to disable text insertion or replacement mechanisms for that scenario

Except of course for insert before and after context as well as

Prefix prompt or suffix

[keen palm]

Right. You would get some rather wonky outcomes if you didn't

[halcyon quarry]

Bigtime

[halcyon quarry]

The setting to control streaming

In the background, there will be more weight factored for line breaks versus sentence completions.

I'll also have to flag for code blocks / italics / other syntax so it doesn't screw it up (sigh)

[halcyon quarry]

Making good progress (I think)

[valid crypt]

for this fork of complex memory https://github.com/Imitationman/complex_memory
i would say it is pretty easy to convert it to tag format

the file is in the character folder of the extension

as the bot dont have webui yet, auto convert would be pretty good

[halcyon quarry]

I did have the thought of asking ChatGPT to make a little utility .bat file that I could include, where you could drag/drop a complex memory file onto it and have it convert it to tag format

Or vice-versa (convert tags to complex memory file)

Would just be annoying if the syntax is different from fork to fork

[valid crypt]

i have to say that others are just crap

the original one dont work

a newer fork mess around with the character card

and do a lot of *** to it

[valid crypt]

in comparison, this one has a independent file in the extension folder with this format

[halcyon quarry]

Yes, same reason I wouldn't want to write tags to character card

When you save data it removes all comments and reformats it all

[valid crypt]

you dont know what did the extension do to my character card

random order

[halcyon quarry]

For the bot, you can compare /internal/activesettings.yaml against settings_templates/base_settings.yaml

[valid crypt]

context at the begining then blah blah blah and name at the end

[halcyon quarry]

activesettings is the result of saving data

after manipulating base_settings

[valid crypt]

also it modifies the context

making it dont work with the bot

[halcyon quarry]

oof

[halcyon quarry]

@terse folio I imagine you've dealt with this scenario before, and ChatGPT (along with myself) are being bolts...

print("check_resp:", check_resp)
if check_resp.endswith('. '):
    print("Ends with Period Space")

I've printed the value of the ever increasing check_resp and also printed each character as they are being received - as far as I can tell, the string ends with ". " at some point, yet it never triggers

[terse folio]

models dont normally output tokens with trailing whitespace, check if it ends with a period

[halcyon quarry]

I'm starting to think that the space must be prefixed on the next token

[terse folio]

yes

the Llm can distinguish tokens that are parts of other words vs words on their own because there are different tokens for text chunks that start with a space

[halcyon quarry]

I'm having a bit of a struggle trying to come up with a logcal way to skip a check cycle if the string ends with X - (last token) for this syntax, and string ends with X for this syntax

Mainly, trying to add more weight to chunk the text if \n\n but it always adds one at a time

[terse folio]

I'll need an example I think, also need to sleep, will have to figure that out in a bit

[halcyon quarry]

alright so just checking for . isn't too shabby.

last_checked = ''

def check_should_chunk(partial_resp):
    nonlocal last_checked
    chance_to_chunk = bot_behavior.chance_to_stream_reply
    chunk_syntax = ['\n\n', '\n', '.']

    check_resp:str = partial_resp[len(last_checked):]
    for syntax in chunk_syntax:
        if check_resp.endswith(syntax):
            last_checked = check_resp
            # Ensure markdown syntax is not divided
            if not patterns.check_markdown_balanced(last_checked):
                return False
            # Roll probability to chunk
            if syntax == '\n\n':
                print("Double newline matched")
                chance_to_chunk = chance_to_chunk * 1.5
            elif syntax == '\n':
                print("Newline matched")
                chance_to_chunk = chance_to_chunk * 1.0
            elif syntax == '.':
                print("Period ending matched")
                chance_to_chunk = chance_to_chunk * 0.5
            print("chance to chunk:", chance_to_chunk)
            return check_probability(chance_to_chunk)

    return False

It can currently match \n or . and factor the probability.
But \n\n will never match

because after it matches and I roll probability, I make sure it doesn't check the same string again

[terse folio]

you would have to do some waiting to check if it's a new line on the next iteration,

yea that's not a fun problem

[halcyon quarry]

Alright, if I can't solve it in like the next 20 mins it's going on the indefinite TODO XD

[halcyon quarry]

yeah I'm giving up on double newline

[halcyon quarry]

Welp, I have it now successfully sending the message in chunks as it is generating, based on chance_to_stream_reply when it encounters a line break or sentence completion.

fun part is now making everything else work with this...

[halcyon quarry]

Yo Illysaviel is back in the Forge saddle for real

I tried pushing this commit months ago when he was apparently on hiatus from Forge, and it languished for over a month before I decided to cancel and push it to the dev2 branch (other code owner was maintaining)

This time, 6 hour wait

[calm rain]

wonder if he saw Flux and was like oh boy time to play with normal image gen tools again for a bit

[halcyon quarry]

Whoa, you’re also alive 😆

[calm rain]

i've been alive lol

focused on image gen stuff rather than LLM stuff for a while now

[calm rain]

(well other than last week I was MIA because i was at ICML in Vienna)

[halcyon quarry]

Certainly! And I haven’t been frequenting #general much myself

[calm rain]

(it was very funny hanging around with the BFL crew as we got a best paper award for SD3, knowing sd3 was about to get rekt by Flux)

[halcyon quarry]

SD3.1 🙏

[halcyon quarry]

@calm rain Swarm basically dead now right? Do you miss it at all?

[calm rain]

what?

Swarm is like

-the- interface to use for image gen

you'd be fuckin nuts to use anything else these days

[halcyon quarry]

🤔

[calm rain]

day-1 flux support yo

[halcyon quarry]

I’ll have to check out Flux now that you mention it, man I’m out of it apparently

@calm rain Anyway congrats on the award! I’m checking out Flux ASAP

[calm rain]

just released today

info to use it in the swarm discord announcements https://discord.gg/q2y38cqjNw

[halcyon quarry]

My code is a bit messy with this message chunking :S

difficult to organize everything

[vestal python]

I have a spare P40 24GB I'll see about getting it to work on it later and hooked in for Flux*

[halcyon quarry]

Ok I definitely have the response streaming working well now for default settings (no intentional delays, etc)

It is only creating one "Hmessage" (internal history management), which is the complete response.
It collects each sent message ID in the related_ids list and ensures that the most recent sent message is allocated to the main id attribute.

This allows all our crazy functions to manage all the separately sent messages

Such as the history reactions feature, but apparently it does not let the bot just blaze through and react to 10 messages instantly, it allows like 5 at a time then pauses a few seconds. Nothing I can do about that - users may wish to disable that for streaming responses.

I will be pushing this feature today

[halcyon quarry]

Also fixed some bugs along the way

[keen palm]

What is the history reactions feature?

[halcyon quarry]

If messages are "hidden" (not in chat history, only in the bots internal history) it will react to those messages with the 'hidden' emoji.
Will also react to messages that are continued or regenerated

So with this message streaming, if you trigger a tag that hides the interaction in history, it will go through and react to all the separate message chunks

[keen palm]

Ahhhh, yes! Excellent, excellent

[halcyon quarry]

It's really crazy how all these features work

If you use edit in history on one of the chunks, it will delete them all and just create one message

This history manager from Reality is so good

[terse folio]

Because of ratelimits,
I wonder if theres another way to indicate what messages are related without editing/reacting to them

[halcyon quarry]

All messages could be sent as embeds instead, with different embed colors 😛

would be interesting if they could have no other stylized effect besides the color beam on the side...

I'm going to skip message streaming for Regenerate Replace task (for now). There's a lot of code to overhaul to make it work correctly.

actually might have a solution...

[halcyon quarry]

yes, have that figured out

[halcyon quarry]

Now seems like only "Continue" needs some updating...

[halcyon quarry]

ugh. Realized it is sending them on a race condition...

[halcyon quarry]

ok fixed it with asyncio.event()

[terse folio]

what was the race condition?

[halcyon quarry]

chatbot_wrapper() is imported from TGWUI - synchronous function

So I'm using run in executor to prevent blocking the discord heartbeat

[terse folio]

I see, and run in executor spawns a thread, which causes the race condition?
the output perhaps?

[halcyon quarry]

meanwhile, as it is chunking the messages and sending them - the llm_gen() function was sometimes returning before the last message chunk could send - and collect its message ID

I really don't know what I'm doing, I trusted ChatGPT and it worked for the most part 😛

[terse folio]

you're getting streaming responses from chatbot_wrapper?

what's being chunked?

[halcyon quarry]

Looks something like this:

def callback(chunk, is_last=False):
    asyncio.run_coroutine_threadsafe(process_chunk(chunk, is_last), loop)()

# Offload the synchronous tasks to a separate thread
await loop.run_in_executor(None, process_responses, callback)

The responses were always streaming from chatbot_wrapper() - the process was always just waiting for the complete output before proceeding

Now, I'm actually checking for triggers to chunk the message, processing them as they are chunked

[terse folio]

does the chatbot_wrapper return an iterator?

where you can "for token in streamed_response"

[halcyon quarry]

idk 😛 I'm pretty happy with it atm it is working

[terse folio]

because I have a bit of code to run iterators in executor too

it's in the aio_utils I think

[halcyon quarry]

I'm looking forward to the day you get your hands dirty in this project again 🙂

ok the damn solution actually didn't work

asyncio.event(), with set() and wait() and clear()

[terse folio]

I tried searching but I don't think the code is commited

would like to look at what's going on

[halcyon quarry]

I'm going to comit now

Took that bit out

There's a bit of clusterf**k here and there

Pushed to streaming_replies branch

So there's 2 known bugs at the moment:

1. Sometimes, the message ID for the last chunk is not captured because llm_gen() returns and gets to send_responses() before the last message chunk actually sends
2. Its possible that queued message chunks work right now but probably not 😛 No time to test atm (If bot_behavior.responsiveness < 1.0)

Logic of chunking:

• Whenever the 'chunk reply' syntax is encountered
  • it checks to see if markdown is balanced. If not, it marks that string as ignore.
  • otherwise, it rolls probability based on chance to stream reply.
• if a message gets chunked:
  • A flag is set in the Task that chunking is happening.
  • Each msg chunk creates a temp HMessage so it can use send_long_message() to assign and capture IDs while it handles the chunk.
  • All those IDs get appended to the parent Task - so this keeps increasing as messages are chunked.
• At the send_responses() function:
  • If no chunking, it will send a response and use the IDs for HMessage creation (as normal)
  • If the responses were chunked, it fetches the chunked msg IDs list and creates the HMessage using them.

I had added some print statements and realized that sometimes the ID in send_response_chunk() (for the last msg chunk) would print after the HMessage assignments in send_responses()

Unfortunately, I'm going to keep this off Main branch for now.

But this is working quite well, if anyone wants to try out the streaming_replies branch, you'll be in for a treat

[terse folio]

ah, yes, chatbot wrapper does return an iterator

you can make this async, and update your chain of functions to also be async

[halcyon quarry]

If you can give more details, that would be great 🙂

I would love to make this async

Or, play around with it and fix it for me XD I've been at this for a long time and getting brain rot

[terse folio]

if you're not working in that area I can update it

[halcyon quarry]

I've been putting off work work for the past 2 hours, now trying to do my actual work in the next 20 mins before office closes 😛

[terse folio]

D: ohno!

alright

lets say streaming is cut off half way, should these variables be set to the current string?

need to figure this out.
I guess it depends what part of the iterator exited.

could move that closing code outside the iterator

[halcyon quarry]

I'm letting self.last_resp and self.tts_resp collect the full responses

[terse folio]

I understand that,
would it break anything to set those iteratively as the code generates the reply?

[halcyon quarry]

I think it already does?

Er I understand

Nope, no problem

So long as when it's done processing they are complete

[terse folio]

pushed it, but not tested

im confident it will work

[halcyon quarry]

Thank you so much

[halcyon quarry]

Seems like 2 days ago would have been ideal time to add Swarm support to the bot

What with the extreme hype on Flux

[valid crypt]

ui?

[halcyon quarry]

testing Reality's fix now for streaming responses...

Noooo it doesn't work 😐

Will consult the great and powerful chatgpt

[terse folio]

ohno!

are there errors?

[halcyon quarry]

Oh you're around

[terse folio]

i was in a call the last hour, im here now

[terse folio]

ahh

i was afraid of that

i'll push an update

do, that's because the asyncio loop isn't really created until you do bot.run()

but I already create a loop at the top of the file for my coro handler

so they're missmatching oops

my hope was that asyncio.get_event_loop or whatever would use the same one

[halcyon quarry]

The other loops are in on_ready()

[terse folio]

good point, I wonder if I can just hook into client.loop instead

[halcyon quarry]

I don't know much about loops, but they seem like something to limit unless necessary

[terse folio]

different kind of loop, I mean the asyncio loop that is how async code works in python

[halcyon quarry]

basically dedicates a thread?

[terse folio]

yea

because "async" runs single threaded, every async function you await, is actually pushing stuff to a loop, where it runs the next thing for a little bit until it reaches another await statement, then switching to the next thing in the asyncio loop

that's how you can get this "threaded like" behavior, out of a single thread

its nothing you need to worry about yourself, asyncio loop is just a term for where the async stuff runs

so issue is, you can't await something from a different loop inside another

[halcyon quarry]

I see so in this case it is hooking into one loop but it is async'ing for code defined in another loop

or something

[terse folio]

merged it

[terse folio]

yea

[halcyon quarry]

testin'

worked!

got the last ID and everything

[terse folio]

async functions are (probably still) secretly iterators

in a super old version of python before async became a standard feature, it was a library.

And you had to use yield from function() instead of await.

yield from is the same as writing:

for i in generator():
    yield i

[terse folio]

awesome!

[halcyon quarry]

I first learned of yield when I first asked ChatGPT about how to handle this - but that didn't seem like the right answer... it was not what you did.

Then noticed yield over in chatbot_wrapper()

[terse folio]

knowing that's how it works,
you can understand how stuff like task.cancel works.

it's just telling the async function (which is an iterator) to stop iterating at that point in the loop

[terse folio]

im curious to see how close it got.
but yes, it probably wouldn't have given you the best answer, because we had* a different problem here

converting a sync generator to an async one

[halcyon quarry]

In my test, learned that I need to add another condition to look out for when deciding to chunk

[terse folio]

eheh, yea had some issues like this in my tests of similar text chunking!

Splitting with newlines usually works well

like your double \n\n idea

[halcyon quarry]

that's definitely the easy route

[terse folio]

what's the goal?
to split sentences?

[halcyon quarry]

To make it flexible enough to behave like a variety of different types of people

Some people write one sentence at a time. Er well... hmm I suppose the character card could just handle that

Yeah, that's a good point...

I'll just split on \n\n - if ppl want their characters to send many short messages, the context needs to dictate it

[terse folio]

ahh, maybe you could add something onto the system prompt

like always write your outputs like AI: "text"
and tell it each "" is it's own message and its okay to send multiple at once

then you can use regex to select all content in those quotes to send as each message

[halcyon quarry]

Too complicated XD

[terse folio]

i can share some of my code for that

on top of that I was using grammar to enforce it. but that's not so nessecary

[halcyon quarry]

Appreciated, but not interested in that handling

[terse folio]

Oke!

[halcyon quarry]

maybe, maybe it could be an optional alternate method for chunking

I'll think about it

[terse folio]

if you want to go that route, you could define chunking as a user editable pattern somehow

(not sure how it was implemented)

[halcyon quarry]

Yep thats the idea. I already made this method adjustable

[terse folio]

interesting

[halcyon quarry]

  chance_to_stream_reply: 0.5         # Chance trigger sending partial message as it is generating. Range: 0.0 - 2.0. 0.0 = Never split reply / 1.0 = Splits very often / 2.0 = Always splits for any trigger
  stream_reply_triggers: ['\n', '.']  # If you may want to adjust triggers. default: ['\n', '.']

afk for a bit

[terse folio]

actually, this would need more than just modifying the chunking pattern.
you also need to change the chat template for tgwui to include the quotes

hmmm, on 2nd thought, the quotes might not be needed that much.
I should test that out later.

but on the other hand the quotes are useful for filtering out LLM censoring or the random stuff it says afterwards like "this has been a conversation with X about XYZ..."

[halcyon quarry]

stopping strings should handle that (mostly)

[halcyon quarry]

Testing that TTS is still working - then pushing to main

chunked messages are working with my dumb responsiveness queueing behavior but it's not synchronizing well with typing. Good enough for now.

Welp, this is unfortunate.

The responses don't output until the TTS finishes generating

1. Payload is sent
2. Alltalk or other TTS generates audio
3. Text responses begin streaming

I wonder if alltalk has a setting for that...

[terse folio]

I don't know how a tts engine would know when to return the "visible" portion of the generated text that contains the audio file

since there is only one audio file that is just streamed during generation?

if you stop generation per sentence and do it that way it would work.
but ultimately using a TTS api would solve this.

[halcyon quarry]

I think alltalk splits the output into multiple chunks

then generates the audio files for each, then merges the audio

[terse folio]

if you use different speakers

I imagine so

[halcyon quarry]

Even not

[terse folio]

interesting!

[halcyon quarry]

Yea, the result can jump a bit drastically between sentences and such

even without narrator

Well, streaming TTS responses will have to be on hold. Sad

Streaming replies is now committed to Main!

I may double back and attempt to detect double line breaks ('\n\n') in order to increase their strength, but for now '\n' will suffice for normal strength

"." has half strength. It can be omitted via the setting.
I added a check that prevents chunking after a list number like

1. Won't chunk after the period

[halcyon quarry]

@terse folio in the case of chatbot_wrapper(), when you initially asked if it returns an iterator, is it because it uses “yield” ?

Any other types of iterators that a synchronous function could return?

[terse folio]

I don't remember exactly which is which, but I think it's pretty interchangeable, will look it up later.

But when you use the yield keyword, that creates a generator/iterator
What's unique about those is if you stop it midway using the break keyword, it will stop processing the code.

for example:

# No way to cancel, returns full result at once
def test():
    output = []
    for i in range(10):
        output.append(i)
    return output

x = test()

def test():
    for i in range(10):
        yield i

x = test() # this didn't actually run the code yet
# later we can access the items by using
for i in x:
    print(i) 

# or this will iterate over the generator, and put it in a list.
x = list(test())

one other thing, with that for i in x example.
if you were to try iterating over x afterwards, it would throw a iteration ended error
so if you want to go over items multiple times, cast it to a list first

[halcyon quarry]

Although you didn’t show example with break. I imagine that would be
for i in range
if i == 6:
break
yield i

[terse folio]

outside the generator

[terse folio]

Talked about it here while demonstrating something else

[halcyon quarry]

was actually just about to ask this 😛 yeah that’s more interesting

[terse folio]

Yup!

[halcyon quarry]

So at some point I may have a listener that would break for the output, such as if the author of the request is typing or just edited their message

[terse folio]

yea, you can do that pretty easily

[halcyon quarry]

Easily is a thing of the past now that my dumbass incorporated the delayed response behaviors

[terse folio]

I think there's a version of asyncio.wait_for for receieving discord events. I forgot what its called.
but put that in a list with your text streaming function and asyncio.gather it? there's probably an option to return when one or the other finishes instead of waiting for both

[terse folio]

ohnoo

[halcyon quarry]

I think it’s close to being manageable but I’m handling something wrong

There’s a lot of moving parts to consider or ignore and it’s difficult to make it all functional without being overhanded

I set up 2 checkpoints after queuing the message, where the function is supposed to analyze the message task and adjust timing for come_online() and/or istyping() , but it's still being a bit clunky. Had refined it many times

In an effort to keep forward progress, I think I made the relationships of Task(), Message(), IsTyping(), etc a bit more complicated than necessary

Its history repeating itself though, my progress has been like that since I got into it - write a bunch of code that works, then spend double that time going back and doing it the right way

[halcyon quarry]

Back when I was coding a fighting game character for MUGEN, the triggering system was like this:

triggerall: (must evaluate to True for any other triggers to work)
trigger0: Some condition
trigger0: Another condition.  Both trigger0 AND triggerall must satisfy
trigger1: Some other condition. requires triggerall and trigger1 to be True
etc

triggerall was optional. Any number of the other triggers were allowed.

Ideally, I would mimic this for Tag triggering, but the entire format would have to be overhauled to support it.

So the best I can do "easily" will be like:

trigger (or trigger0): 'some text,another text'
trigger1: 'another trigger text'
trigger2: 'yet another'
etc

And all of the triggers would have to be found in the search text to consider the entire tag as "Matched"

Working on this now

[halcyon quarry]

Good progress on this.

I'm pretty confident that the revised match_tags() handles multiple triggers now. Just need to comb through the rest and make sure there wasn't some oddball thing regarding triggers

[halcyon quarry]

ok there were a few more sections that needed updating... trumping, expanding triggers. All good now.

[halcyon quarry]

@keen palm It turns out the logic can allow insert_text and positive_prompt tag values to be applied (insert the text where it is matched). I'm going to allow it, and print a warning that multiple triggers were matched but it will use the matched phrase from the first trigger definition.

idk if you use this or not, but just a heads up

[keen palm]

I only use suffix_context, but that's handy to know

[halcyon quarry]

it's pushed to branch multi_trigger_logic - and seems to be working great but I don't have time to thoroughly test atm

[valid crypt]

got this https://github.com/Arondight/vits_api_tts working maybe and it seems like it got streaming response, gonna be tough to plug it into the bot but first gonna test it out

someone know what is wrong?

[halcyon quarry]

Is the error in the bot?
Or when using TGWUI

[valid crypt]

tgwui

if tgwui dont work, bot is dead as h***

chatgpt says that its because im using windows and it is something wrong with the file name

omg colons

[halcyon quarry]

Haha

[terse folio]

Something you can add is a setting like "pass through: true/false" to the triggers.

Where if disabled it will not run all triggers leading up to that one.

Passthrough: false: (trigger all + trigger1) activates trigger1
passthrough: true: (trigger all + trigger1) activates them both independently

[valid crypt]

the extension is working but i only found a japanese vits model that worked

and the quality is pretty poor

[halcyon quarry]

I'm not using triggerall per se - it is simply collecting all keys that start with trigger and ensuring they all match.
I like your idea though, to add another param which dictates whether all triggers must match, or any can match.

[terse folio]

From your explanation it sounded like you were doing multiple steps of matching
"Cat + Dog" = response 1
"Cat" = response 2
"Dog" = response 3

But what if cat+dog was set to pass through backwards?
You could get:
"cat + dog" = response 1, response 2, response 3 (Or any of the triggers that also allow being used standalone)

Actually, that standalone part would already be implemented as part of tags I believe

[valid crypt]

for some reasons most vits model are not detected by the https://github.com/Artrajz/vits-simple-api

but those model that works are not very good

at least i cant find many

maybe have to train myself ;-;

[valid crypt]

https://tenor.com/view/seriously-facepalm-fml-crazy-head-bang-gif-16492152

[halcyon quarry]

After a few drinks and thinking about it, there’s no need for the pass through type tag because the trigger definitions are each essentially comma separated lists of triggers.

[halcyon quarry]

Things like this are already possible due to the ‘trumps’ tag

Except now cat + dog is possible

You can have two tag definitions that have identical triggers, but add some unique trigger to one like ‘djdhd’ and set it as the ‘trumps’ param on the other

I do this with a bunch of tags for “illustration” LORAs, paired with the “random” param.

[halcyon quarry]

Just use alltalk tts and be happy 🤗

[terse folio]

interesting, okay!

[halcyon quarry]

Am finding some minor bugs I overlooked

[halcyon quarry]

[TAGS] Multiple trigger keys now supported

@keen palm pushed it to Main

[keen palm]

Ohhh boyeee

How does that work?

[halcyon quarry]

  - trigger1: 'draw'
    trigger2: 'cow'

search text: 'draw a cow'

This tag definition will be matched

search_text: 'a cow could draw'

Also matched

The triggers don't have to be incrementing or anything, they just need to start with 'trigger'

These are valid:

- triggermybutt: 'some test,another text'
  triggeryourmom: 'mom is nice,mom sucks'

search text: 'mom sucks, but she passes some test'

...Matched!

[keen palm]

Ahh very nice

And as long as one from each of the triggers matches, then the tag will be triggered

[halcyon quarry]

- trigger: 'generic_sw,star wars'

- trigger: 'star wars'
  trigger2: 'boba fett'
  trumps: generic_sw

search_text: Star Wars movie with Boba Fett

Result: only the second one applies

[halcyon quarry]

yes, one phrase from each key

[keen palm]

Very useful. I might have to update!

[halcyon quarry]

Updating should never be a problem now really

You'd be helping me a lot by updating, because if something does break I'd love to know and fix it

[keen palm]

I won't be able to do much today, at least not for a while

[halcyon quarry]

Bah!
I just recently embraced this variable for swap_llmmodel feature

Now gotta go back to the old way (capturing and storing it)

[keen palm]

Whoopsy!

[keen palm]

Well, the bot is completely non-responsive, so that's a good start

Okay, there. It required a few restarts for some reason

I've got the trigger1 and trigger2 thing to be working somewhat, but it doesn't work whenever the triggers match other tags as well.

So let's say that you have a tag that triggers on Bob, and another one that triggers on Doug. Both of those provide the bot with context about those two characters.

But also, there's a tag with trigger1 for Bob and trigger2 for Doug, which provides additional context about their relationship.

That tag doesn't get triggered, but the individual ones do.

[halcyon quarry]

So the text says “Doug … Bob… “ and the ones with only one trigger are working, but not the one with Doug and Bob?

I had tested with two tags - one with a single trigger “draw” and another with two triggers “draw” and “cow” and they were both triggering together and applying

[halcyon quarry]

Did you use the updater script?

Not that important I suppose git pull would also do the job 😛 The script is good if we add any requirements but we haven’t recently

[keen palm]

I did use the script, yes

And yes, based on my testing, the tag with multiple triggers wasn't working. At least it doesn't seem to be injecting context appropriately

[halcyon quarry]

@keen palm Could you put a print statement here? Ctrl+F to process_prompt_formatting()

            print("format_prompt:", format_prompt)

[keen palm]

Alriiiighty

[halcyon quarry]

actually

nvm

wrong thing 😛

[keen palm]

Good cuz I didn't want to do that

[halcyon quarry]

in process_llm_payload_tags()

            print("context:", self.llm_payload['state']['context'])

Orrrrr

wanna share your tag definition? I can check it's formatted right (probably is)

[keen palm]

I'll test again

[halcyon quarry]

I'm testing too

[keen palm]

Getting the toddler up too, and she's a bit needy

[halcyon quarry]

I'll test first

Ugh yes... definitely something not right

I must have a "break" statement in the wrong spot, so when it matches a tag it is stopping checking the rest.
Will fix this tonight

yes found it

Fixed

Merged to main

[keen palm]

https://tenor.com/view/emperor-star-wars-good-goooooood-evil-gif-12447951

[halcyon quarry]

I have a "background task queue" for processes that may take some time, but which do not use VRAM / interfere with main task processing

I just set all "react to messages" functions to use this (the "history reactions" feature).

This solves the issue of it taking too long when it sends a bunch of message chunks. Now, there's no rush, it can go ahead and react to the messages while handling other tasks.

Wondering if there's any other time consuming processes I'm overlooking that should get this treatment...

[valid crypt]

add stt and blow up our mind

[keen palm]

The bot just reacted with 🙈 on a prompt that should not be hidden at all

[halcyon quarry]

If you check the log in internal/history/the channel number/the character name was the messaged logged as hidden?

also, was it just a regular message request? Or was it a regenerate / continue?
Was an image generated?

[keen palm]

It is marked as hidden, yes. I see the issue though. I triggered my IdeaBot character, which sets things as hidden.

That shouldn't happen though

I have on_prefix_only:true for that tag

[halcyon quarry]

Does it have one trigger key? Or does that have trigger1, trigger2 ?

[keen palm]

Just one

  - trigger: 'Idea'
    search_mode: user
    on_prefix_only: true    
    swap_character: IdeaBot    
    load_history: -1              
    save_history: false 

[halcyon quarry]

Checking it out now

I reproduced the issue... so now need to see why the code isn't working as I expected...

[keen palm]

I'll just comment out tag for now

[halcyon quarry]

think I figured it out...

yep - pushing it now. Had a flaw in the logic

Did some additional testing to ensure I handled it right

It checks each trigger phrase in each trigger key.... when a phrase was matched, but failed on_prefix_only I was letting it continue to try the next trigger phrase.

But I was not setting the flag that the trigger key was unmatched as a whole

Now, it checks if the matched phrase is the last one.... if it fails the prefix check, it sets that flag. Otherwise lets it continue

@keen palm Pushed

[keen palm]

I will break update a bit later

[vestal python]

I've been trying out some different models for RP on the discord bot with some general character persona, but I'm getting tired of the constant breaks... Is there some specific model that doesn't include these breaks:

Everytime I add a new end phrase, a new one pops up

[halcyon quarry]

Llama 3 based models?

Make sure you are on latest TGWUI too, they've improved support for llama 3 models

[halcyon quarry]

Welp, pretty damn easy... just added a new /prompt command with just one additional option for now which is begin_reply_with

It's working just fine and as expected

[keen palm]

Fuck yes

[halcyon quarry]

Adding a tag for it now, too.

aaaaaand that's implemented now too

Pushed to main!

[keen palm]

What's the tag?

[halcyon quarry]

Same begin_reply_with - If a tag is triggered with that, it will use the value to Continue from

Just added it to the Wiki now https://github.com/altoiddealer/ad_discordbot/wiki/Tags#text-generation-tags

I'm going to start looking into per-guild characters now...

Seems like the name can be customized per server

But not avatars... avatar must be shared

might need to add a command to change the bots avatar

yeah I think this is the way

Eh. nvm I'm pushing it off for now 😛

Let's see... on the to-do list is: discord based conditional Tags
Can't remember exactly what we had in mind for this....

Ah ok I remember now.

Like this

[halcyon quarry]

And a few more… Roles…

[keen palm]

What about roles?

Oh, there was also the idea to use the channel info as additional character context

lol, it just added a period

[terse folio]

👌 Nothing more had to be said

that's funny, I get some unexpected ends sometimes too when trying to guide responses

[keen palm]

I'm going to have to do a more serious test

I've noticed that Continue almost always fails to add anything new, as well.

[terse folio]

it's often like that in the webui as well.
if the llm has written a few messages for the character, it will keep returning end tokens because it thinks it's done.
(since that's where it stopped the first time)

what works for me is adding an extra newline and saying continue, then it usually adds 1-2 more sentences

[halcyon quarry]

I meant, like another conditional tag for something like for_role_name_only

[keen palm]

Ahh yes

[halcyon quarry]

Ah OK you can get the Role ID from the Server Settings... just no where to get it easily from the chat window

I'll make these possible to be comma separated

I'm welcome to suggestions for more

[halcyon quarry]

With the new ‘begin_reply_with’ tag param, I had a very interesting idea for a “Flows” definition. The idea I had was for something equivalent to the “alternating prompts” feature in SD WebUI, but for TGWUI

By triggering a low “max_new_tokens” setting, swapping character context, and recycling the previous responses with variable {llm_0} in ‘begin_reply_with’ - this could have drastically different characters piecing together one lengthy response

One could make a true Jeckyl and Hyde type character in this way

[terse folio]

interesting idea ^^

[terse folio]

Some voice changes mid sentence would be cool as well

[halcyon quarry]

Ya know… that may be a hack to stream TTS… instead of chunking the text, just stop generating, then feed it back with continue

Er… yeah… wouldn’t work great actually

[terse folio]

I don't think it's possible to change the prompt while its's generating with the current webui.

[halcyon quarry]

Would need to limit tokens to make TGWUI cut the gen short to trigger the TTS

which could be wonky

I’ll see if there’s some mechanism to trigger a Stop generating

[terse folio]

Do the tts engines that support mulitple voices have tags you can send them?
like "this is a test <voice B><speed 0.5> and now in a new voice!"

[halcyon quarry]

Alltalk has the separate narrator voice, which applies to certain syntax

There may be something like that with other engines though

[terse folio]

hmm hmmmm

[valid crypt]

this guy's code might help, this project has stream tts through openai api just im not sure if it is located here https://github.com/zixiiu/Digital_Life_Server/blob/master/GPT/GPTService.py

¯_(ツ)_/¯

[halcyon quarry]

I could fairly easily stream TTS if I chose a specific TTS api like alltalk TTS API, but to keep the bot flexible for different TTS clients and let TGWUI manage the extension - would need to do some hack to make it generate partial responses

If there is some kind of "stop generating" function (I'll look for it), it could be the answer

There's no parameter to make TGWUI stop at a line completion / newline, only hard token count

[valid crypt]

🤔

[terse folio]

what about stopping strings?

[halcyon quarry]

That's an interesting idea...

But no that doesn't work

[terse folio]

you can append an extra stopping string ontop of the user payload

[halcyon quarry]

With the way I had set it up, with a "chance to stream a reply" - I'd have to axe that and make it all or none

[terse folio]

Hmm, hmm, alright

[halcyon quarry]

But I like the way it is, adding some randomness to when it decides to split message

[terse folio]

yea, ultimately tts would be hooked into separately somehow.
I wonder if the extensions expose functions one could directly access.

if so, you'd also have to figure out how to unhook tts generation from the text-gen pipeline so it doesn't do it twice

[halcyon quarry]

Ahh yeahh... damn

forgot that bit

There is always monkey patching but, from my experience with monkey patching load_extensions() it was not fun at all