[WIP] Neuro's Desktop (An integration for letting neuro use a computer)

[timber basin]

anywah!

[plucky marlin]

this obviously doesnt work i just pasted random things in random places

[timber basin]

No...

that's the types object

you're supposed to use the stuff in properties

[plucky marlin]

  --url http://localhost:1337/ \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "action",
    "data": {
        "id": "screenshot",
        "name": "random_screenshot_idk",
        "region": "{}"
    }
}'```

im stuck here

what do i put in region

[timber basin]

you don't have to

that one's optional

region is basically selecting a region to screenshot

otherwise it's gonna take the fullscreen

[plucky marlin]

  --url http://localhost:1337/ \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "action",
    "data": {
        "id": "screenshot",
        "name": "random_screenshot_idk",
    }
}'```

doesnt work

oh because of ,

ok seems to work but something is wrong

so i mixed up action and id

now it repeats this

with random numbers

sorry it looks like im not helping at all we just wasted time

[timber basin]

its okay

are you using the latest commit of neuro-relay?

[plucky marlin]

i installed it all like 30-60 min ago when we were installing neuro windows

is it not the latest

[timber basin]

neuro windows rely on neuro relay

cd into neuro-relay

and try running git pull

[plucky marlin]

and it was freshly cloned from venv

ok i'll try

"already up to date"

[timber basin]

hmmm

[timber basin]

wait-

no that works

[plucky marlin]

huh

[timber basin]

it forwarded it to windows-user

[plucky marlin]

i want one screenshot not infinite screenshots

bash isnt sending anything

[timber basin]

but now the issue is, windows user somehow disconnects

[timber basin]

it's because its trying to resend the failed action to neuro windows

windows user is neuro windows

but neuro windows relies on windows api

[plucky marlin]

oh i see

[timber basin]

so either neuro windows or windows api disconnected smh

[plucky marlin]

user not connected

hm

[timber basin]

you could try turning on the tracer

and see where its coming from

I've been trying to do that for hours

the tracer just walks too fast

[plucky marlin]

yeah it will be impossible

[timber basin]

I mean, tracer does output to a file

[plucky marlin]

let me find where its coming from via common search

oh wait its in neuro relay, i searched in neuro windows

[timber basin]

I thought working with others would be faster, but it seems to be the opposite lol4

[timber basin]

yea

[plucky marlin]

right

[timber basin]

neuro relay keeps breaking down for some reason

[plucky marlin]

so its from here

[timber basin]

yeah

but something is making neuro windows disconnect

and that's the real issue

something even stranger is

[plucky marlin]

it means this doesnt work

[timber basin]

this popped up a few times after I've ran the repository

before it ommited an actual error

[plucky marlin]

does it work the same way for you?

or are you able to make screenshot

[timber basin]

where is self.clients being registered

[timber basin]

I was not able to take a screenshot with neuro windows

but before it ommited an actual error

now we just have this disconnect error / issue

[plucky marlin]

hmm

i might remove some log spam, and then add my own prints in places which i find interesting

where this client list gets filled, emptied and all

[plucky marlin]

to see why its empty here

also i should disable auto-resend feature

[timber basin]

Also, might note this here
If neuro-windows is ever used, probably ask a few integrations like vsc-neuropilot to auto connect to port 8001 if it's found active

Notes (Nakurity)

[timber basin]

it's hardcoded

[timber basin]

kay

[plucky marlin]

what do you mean

i have the code

can i not dehardcode it

[timber basin]

the retry function has no easy way to be disabled

its called in a lot of functions

with no configuration to disable it

[plucky marlin]

why do we need to retry anything anyway

[timber basin]

it's called hardcoding

[plucky marlin]

if it fails first time its a problem needed to be fixed

[timber basin]

sometimes things just works again

and plus, if we want to ignore it

we could just try and start up the server in another terminal

[plucky marlin]

i am really against ignoring anything and patching it with "just restart"

[timber basin]

and it automatically gets reforwaded again

[timber basin]

lol

Cass is, I'm not

[plucky marlin]

because it leads to exactly this

something breaks somewhere

and we have no idea where

since everything gets restarted on fail

[timber basin]

that's why miracles exist

I'll just sanitize the tracer output for its log file

[plucky marlin]

ok so its not the problem with get(), the dict is actually empty

who is supposed to be the client?

windows user?

[timber basin]

yea

[plucky marlin]

where is that one located

previouusly we run some "test" guy

and now we use this window user

where is he

[timber basin]

He is neuro windows

he lives in neuro windows

btw this is the tracer debug log

with its output sanitized for a file

40+ pages long, yes

it means this program is doing stuff for over 40+ pages

[plucky marlin]

it seems like it removes client on fail here

which is kind of weird

why would you remove client entirely if random send attempt failed

[timber basin]

I wonder who's commit was that

[plucky marlin]

let me find if it actually happens...

OH WAIT

does it send log to file?

[timber basin]

yea

[plucky marlin]

this log

[timber basin]

that doesn't...

[plucky marlin]

ok nvm then

[timber basin]

tracer does tho

I just sanitized its outputs

[plucky marlin]

AHAAAA

[timber basin]

if you are sane like me, go through tracer

[plucky marlin]

IT WASNOT EMPTY ON FIRST ATTEMPT

[timber basin]

That's crazy, tysm

[plucky marlin]

ok so look here

iterable must contain uniform types

also i dont see where the client gets removed here.. weird

[timber basin]

here

[plucky marlin]

but the print is not in the log

[timber basin]

hmmm

then it must've crashed

[plucky marlin]

look, here we have 2 attempts, one had client and the next one doesnt

this appears only once i think

right after send

[timber basin]

sometimes it silently crashed

[plucky marlin]

let me find more hold on

[timber basin]

this might means it crashed

[plucky marlin]

what is [context]

[timber basin]

the messages that tell neuro what environment she is in

its sent by the ocr backend for neuro windows

which is basically a secondary eyesight for neuro, when vision is off

[plucky marlin]

it actually crashes here for the first time

exception is "must contain uniform types"

[timber basin]

AHA

💥 ConnectionClosedError: sent 1011 (internal error) error in fragmented message; no close frame received server.py:289

tracer is useful, finally

windows-api crashed

[plucky marlin]

where can i get source for this

[timber basin]

or maybe neuro windows crashed

[timber basin]

pip install neuro_api

[timber basin]

this means I have to move on to neuro window's tracer

[plucky marlin]

hmm i have it

but vscode doesnt see it

[timber basin]

click quick fixes

and select a different interpreter

select the one that has .venv in it

[timber basin]

also, can you please just read this

its not that long

plus, it gave some very useful information

that would've taken much longer with traditional debugging

[plucky marlin]

i dont understand it

it doesnt print this. i will find where it crashes, hold on

[timber basin]

just read where it says stuff like these: ╭▶ read_from_websocket() client.py:26 [31.412s]
those are function calls tracer caught

read_from_websocket() is a function

[plucky marlin]

look here

is it valid neuroaction

[timber basin]

that's vaild

please, just try and understand the tracer log

[plucky marlin]

is it valid that it has NeuroAction()

[timber basin]

so we can move on

[timber basin]

yea, its vaild

[plucky marlin]

ok let me check that tracer

[timber basin]

you can ignore the stuff below the function calls

function calls are what is useful rn

if it ends with () its a function call tracer caught

[plucky marlin]

its just i dont know what these functions do

[timber basin]

search it in the code

tracer also provides the file for it

and the line

[plucky marlin]

but i will need to search 50 functions

i cannot remember it

i will maybe remember 3

then i need to do 40 more

to get full picture

[timber basin]

its not all functions

[plucky marlin]

while i can just do my way and see where thing gets wrong

[timber basin]

its just 4 functions

[plucky marlin]

but i'll try understand the tracer, even though i dont really know where to look

[timber basin]

╭▶ handle_action() client.py:41 [31.583s]
│ args: self=<src.dev.nakurity.client.NakurityClient object at 0x00000..., action=NeuroAction(id_='tony_action_0', name='screenshot', data=...
│ → print(f"[Nakurity Client] ========================================")
│ • locals: self=<src.dev.nakurity.client.NakurityClient object at 0x00000..., action=NeuroAction(id_='tony_action_0', name='screenshot', data=...
│ → print(f"[Nakurity Client] RECEIVED ACTION FROM TONY (Neuro Backend)")
│ • locals: self=<src.dev.nakurity.client.NakurityClient object at 0x00000..., action=NeuroAction(id_='tony_action_0', name='screenshot', data=...
│ → print(f"[Nakurity Client] action.name: {action.name}")
this stuff > | is basically stuff that the function do

you can ignore those

[plucky marlin]

but its because server killed client

its not the cause

cause is this

it fails here

[timber basin]

we need to check neuro window's tracer

to see why it failed

[plucky marlin]

action msg isnt formatted properly, error is this

[timber basin]

wait- really?

[plucky marlin]

well that what he said

[timber basin]

wait, brb

[plucky marlin]

maybe this needs to be googled, ws is socket

oh wait no its deeper

its from client

[timber basin]

client is neuro windows, right?

[plucky marlin]

or maybe because they run in parallel

i can print client

hold up

[timber basin]

thats why we need neuro window's tracer

[plucky marlin]

this is our client

also i feel like your tracer is about something different?

i really cant grasp how to read tracer

do i read it up to down or down to up

and for example here

how are these 2 parts connected

is it like

crash1, then crash2, then crash3

or bottom part is like, elaboration of top part?

and who crashed

i have server,client,intermediate,etc

first part tells "return"

who returned? web socket?

this is how i try to read the tracer and see nothing at all

also i like how it tries to resend some random block of memory after it failed first time

vs normal

[plucky marlin]

also its not this who kills the client, i removed this line and it still gets popped somewhere

i really dont like that we pass it some shit like
Neuroaction(a,b,c)

doesnt look like data at all

i want to find this function NeuroAction but i cant

[timber basin]

they are not, the arrows should've been indicators

[plucky marlin]

this looks like alien language to me, i dont see where the function that has (_id) as argument for example is

its from neuro-sdk

api

my current version is that this doesnt work at all, it formats it wrong

[timber basin]

the top: handle_action() ... stuff is the function call. And below inside the arrows are the function contents

[plucky marlin]

or doesnt format at all

[plucky marlin]

can you help me point where the end is

the line where it says "fuck this, im out" and exits

[timber basin]

youre looking at the wrong thing

[plucky marlin]

i need source of this

NeuroAction(name, id_, data)

[timber basin]

hold control and click on the neuro api.command import stuff

hold conytol and click on neuroaction

in vscode

[plucky marlin]

change interpreter, google more, or ignore x2

[timber basin]

select a different interpreter

[plucky marlin]

oook im changing it now..

[timber basin]

to the one that has .venv in it

[plucky marlin]

it worked you genius

alright so its here

but... does websocket accept it?

it seems to not like this type of class format at all for some reason

[timber basin]

thats just a type for it

typings

[plucky marlin]

it formats it as a class

[timber basin]

click on the metaclass (NamedTuple)

that should have the logic for it

[plucky marlin]

it needs to be formatted into something

i think

so websocket accepts it as valid data

[timber basin]

https://chatgpt.com/share/68fce078-95f4-8011-88e5-2bef33da42d0

this is an explaination of my tracer via chatgpt

I'm on mobile rn, so its a bit hard to type

my keyboard ran out of battery

[plucky marlin]

[timber basin]

so sorry!

[plucky marlin]

but actually its exactly what i need

[timber basin]

I'm so sorry, I didnt mean to call you that!

[plucky marlin]

its not a problem, i dont mind

do we use this tracer you got or we need a new one

is your tracer from first attempt to send, before client was killed?

if your tracer from 2nd or further attempts then its not very interesting since the problem of sending is the absence of client

[timber basin]

this is the full tracer logs

[timber basin]

That was the sanitized one

[timber basin]

this is the latest

[plucky marlin]

this one right?

[timber basin]

yea

[plucky marlin]

ohh look here it pops the client

[timber basin]

you'll have to patch neuro window's tracer with neuro relay's updated tracer

if you want to use it for neuro windows

[timber basin]

yea

[timber basin]

printing tracer logs for neuro windows outputs a very interesting output rn

[timber basin]

now we need to find where or why neuro windows sent that close error

[plucky marlin]

but

is it different problem or same problem

what about "iterable must contain uniform types"

[timber basin]

hmmm

we would need neuro window's tracer

or we'll need to go through the full tracer logs

[plucky marlin]

so this is exact place of websockets where i get my error

but how does that class neuroaction get here

it basically means that whatever websocket got to send, wasnt byteslike and wasnt str

and i can really see that, we gave it some dank class actually

[timber basin]

its not supposed to be a byte....

or atleast... i dont think so

i think tracer might tell us a lot more actually

[plucky marlin]

can you ask gpt "how to convert this neuroaction class into something that websocket library can send"

[timber basin]

it doesnt know about the neuro sdk for python

[plucky marlin]

[timber basin]

I tried

[plucky marlin]

what if you give it the class describtion

[timber basin]

it wasnt trained on it

[timber basin]

it needs the class namespaces and functions, etc. Otherwise it outouts assumed function names that are wrong

[plucky marlin]

let me try give it just string and see if it sends it from that point of program

oh it doesnt want string either

it needs to be json encoded message i think

[timber basin]

yea

[plucky marlin]

i just dont know what format it expects

i followed examples from client

created json

encoded it

IT WORKED

SUCCESSFULLY SENT

sent action to windows user

this is what i sent instead of action msg

so client is alive there

we just need to format it onto json

let me try...

ok apparently json.dumps doesnt work with _id and other json format

[timber basin]

try using one of neuro_api's provided types for that

im gonna go sleep now

its 2am again

gn!

[plucky marlin]

goood night

i'll continue for a little, maybe i can figure it out

ok so here is the answer from neuro api

because, server sends reply to neuro api! but i think when we make a screenshot, it should make a screenshot? and give neuro some kind of response "screenshot taken"

i dont quite understand the flow here.. what happens when neuro sends "screenshot command"?
i guess it should be redirected to windows user integration? and not to neuro

neuro api doesnt like whatever i give it

i dont think i can do it

its just

why does server send it to neuro api

its supposed to take screenshot and send neuro raw data

but client.send sends it to neuro api ??

then what???

[plucky marlin]

or not raw data idk

whatever

i tried sending action/result - api doesnt see it anyway

because it receives it here

and this comment says that we should implement some handling ourselves ?? what does neuro api stand for then at all

why do we send it to it anyway

[plucky marlin]

so actually whatever i send ends up here and does nothing

this thing is such a mess

ok so basically

the answer to current problem - format message properly so websocket can access it

example:

                                "command": "action/result",
                                "game": "windows",
                                "data": {
                                    "id": action_id,
                                    "success": True,
                                    "message": data
                                }
                            }
                    
                            print("[SERVER_DEBUG] ... sending action msg=", action_msg)

                            await ws.send(json.dumps(action_msg).encode())```

[plucky marlin]

websocket will successfully send it to neuro.api. which will end up here and respond with "unknown action"

i have no idea how its supposed to work

why does it end up in that dead branch anyway

what does windows integration have to do with all if this

is it even linked to neuro relay in any way now

wait, but it successfully sends my mouse coordinates

so windows integration is functional

i'll try stealing format from there

and wtf is this.. i didnt send this????

is neuro.api part of windows integration

why do i get message from [neuro.api] in my windows integration log

oh wait

is all this... only because...
im sending query from some random git bash? and randy sees this message, while never asked for screenshot, and says "hey wtf is this?"

but idk how to format it anyway.. "screenshot" is unknown

even though its in the list

it ends up in branch of neuro api which only sends "action unknown"

to anything you give it

i just need properly formatted command

and not this garbage

if only my tony worked

and i wouldnt have to format this json nested hell

[plucky marlin]

ok so i think it works but i just cannot format json properly

  --url http://localhost:1337/ \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "action",
    "data": {
        "id": "screenshot_id",
        "name": "screenshot",
    "data": "\"{\\\"name\": \\\"screenshot\\\",\\\"region\\\": [100,200,400,500]\"}"
    }
}'```

currently i have this

it results in this

the last "action" is some bug, probably because my screenshot command format is incorrect

"extra data" in second package probably leads to some leak of memory idk

nakurity says this

oh wait i see something

ok now this seems to be correct but it doesnt work

i cannot make valid json which will work for this temlate

i have no idea what windows integration expects

[plucky marlin]

this seems valid

yet it doesnt work

only command that is valid that gets send by actual randy is reregister all

which consists of 1 field

i cannot use it as example and its not even provided with template

if only i could see 1 proper command that was sent to integration by randy according to template

i just dont know what integration expects

even though template is right there

what is 'type'

do i need to include it too

i guess i only need properties

wait.. maybe i can take this sent-by-mistake command as example

[plucky marlin]

ok enough for today

results:

1. server sends wrongly-formatted data to integration, can be fixed with json dump
2. i do not have operational tony and i do not have example of format of message according to screenshot schema template should be

so i cannot send correct query with git bash

steps to fix: obtain correct neuro query to integration

then fix server format accordingly

without example i have to find exact combination of correct query from bash and correct processing in server.py

which will take like a month with my current knowledge

staring into trace for 50 hours wont help this is im sure of

its really bad strategy i picked - now knowing what integration expects, not knowing format of neuro's query, not knowing how client processes server's relayed message

it can be broken in any of these steps

it feels like i know all 3 but not exactly

i have idea but not exact symbols it wants

[timber basin]

yea. it should've told neuro after the screenshot was taken

[timber basin]

You'll need to use the class ActionResultData

Its in neuro_api.server

[timber basin]

back the action result

[timber basin]

neuro api is an sdk package for developing neuro integration, that's why its in every loop

[timber basin]

because it's an unknown command

[timber basin]

that is because its an abstract class, its for us to implement some handling for the game we're integrating. Neuro API is an SDK!

[timber basin]

BECAUSE IT IS AN SDK

[timber basin]

that's because it's an unknown action, that is not an actual action the neuro backend receives

an action is supposed to be something the neuro backend sends, not receives. the action/result thinks its supposed to be an action

[timber basin]

its an sdk...

[timber basin]

WE DEPEND ON AN SDK, IT IS A PYTHON PORT OF THE NEURO SDK

[timber basin]

yes.

[timber basin]

no. Its not even developed by me, or my friends: http://github.com/coolCat467/Neuro-API/

[timber basin]

how tf did you manage to get randy to work with action/result ?????

[timber basin]

type basically tells neuro what she's supposed to send, text (i.e. string), a number (i.e. integer), an array (i.e. list), etc

[timber basin]

I don't really feel like working on this integration today

[plucky marlin]

I will try something else later today

Got some thoughts after sleep

[timber basin]

kay!

[timber basin]

I couldn't sleep last night, I ended up sleeping at 9 in the morning

[plucky marlin]

sooo

i need to put some logs into windows integration and try to figure out what it wants first

then see what it gets depending on what i send from server

then send some valid static screenshot command from server without attaching it to actual command from my git bash

then fill the server's command with actual git bash command data

sounds like a plan

windows integration crashes on some invalid commands but i think this is pretty much manageble from the relay side

it can check it and decide what to do with it accordingly from there

wtf

i just run commands from yesterday and it just worked

this was made by neuro

i actually tweaked server.py yesterday before quitting this, but never checked because felt like it makes no sense

wtf

so this is what im sending from server to windows integration through client right now. it doesnt use any parameters from actual command at all

  --url http://localhost:1337/ \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "action",
    "data": {
        "id": "screenshot_id",
        "name": "screenshot",
    "data": "{\"name\": \"my_screenshot\",\"region\": [100, 200, 400, 500]}"
    }
}'```

and this is the command itself

even though the command works, it doesnt work the 2nd time. i have to restart windows-integration

no errors visible though...

[timber basin]

but did neuro actually receive the action result as context?

the win resp output is what should be sent to neuro btw

[plucky marlin]

i think neuro didnt receive it for some reason though

[timber basin]

it should be sent to her as context

but I assume that isnt implemented yet

[plucky marlin]

i'll get to that when i fix this part

[timber basin]

that was cass's task

but i think she left this issue for me to fix first, before she continues dev and stuff

you could try commiting it to your repo and creating a pull request!

your first pr with me, and my first outsider contributor!

I love how these two messages, contradict for some reason...

why did I write that?

past me seemed to suggest development != testing

[timber basin]

wyd rn?

@plucky marlin

[plucky marlin]

figuring out why this isnt getting sent anywhere

it appears in console but server/intermediary/whoever doesnt receive it from neuro relay side

after this console line, it just proceeds to do normal context update

[timber basin]

you'll need to use the send_context command somewhere

[plucky marlin]

so here what i know right know

server sends screenshot command to integration, waits for response

it never gets the response

[timber basin]

did randy receive anything?

[plucky marlin]

because of it, server just stops doing anything and i cant send another screenshot command for example.

no

of course

server would relay it further to randy i guess

but it didnt even reach the server yet

figuring it out rn

[timber basin]

thats because randy's supposed to simulate neuro waiting for an action result

[plucky marlin]

nono thats not what i mean

look

here server sends the action command to integration

then it awaits for recv_q

it awaits it for the rest of eternity

[timber basin]

also, it might be best to tell neuro the action's been relayed instead of letting her wait for the actual action results a second later, and send the actual result via context instead

since bc of latency issues

[plucky marlin]

i think we should make it work in clean conditins first and then handle things like "neuro sent garbage command/she didnt send anything at all/whatever"

rn it doesnt work with perfect command

[timber basin]

lol

[plucky marlin]

need to fix this first

[timber basin]

no, what I meant is, it takes too long fof the result to be sent back, that neuro will just have a stroke or smth

even with working system without relay

[plucky marlin]

it seem to send it back instantly though

[timber basin]

sooo, it'd be best to send the result as context

[plucky marlin]

it doesnt reach anything from there

[timber basin]

wait- really?

[plucky marlin]

well i get the message, and screenshot actually appears in that folder

[timber basin]

hmmmm, then where did I get my latency issue from?

[plucky marlin]

figuring it out

the response from integration goes up to here

im not good at python, maybe @abstactmethod makes it work?

because this function seems to be empty

oh i mean write_to_websocket

wrong screenshot. its empty too

nvm i found actual function

aaalright

it goes up to this, which seem to be endpoint

i assume that data from integration actually reaches server, it just doesnt see it

it certainly sees common-update commands from integration though

i'll look there to see whats wrong

tried this but it gets errored

[timber basin]

obv

[plucky marlin]

the original function seems to not work though

any ideas how to fix it?

[timber basin]

revev_q was the recev traffic queue, if ur using ws.recev it'd just bypass that queue and get the latest non existent message, since it was already passed to the queue

[timber basin]

hmmm ask chatgpt!

might help

[plucky marlin]

i refuse

[timber basin]

oh.

[plucky marlin]

i believe it will consume me

[timber basin]

noooooo

let it eat u!!!

[plucky marlin]

can it be because we are using common .send

and we need some "super async queue handler send"?

im not good with async things

maybe we need this?

wait no

i have a feeling that this asyncio.queue not binded to anything at all

[timber basin]

hmmm, classic me and my code.

this seems to be a recurring theme

and its a miracle that parts of it even work

[plucky marlin]

why can i send it through normal socket

but cannot receive from it

there is 1 socket per integration

but it says its busy

alright i'll try looking into this queue and try to make it work i guess

yeah i googled a little and it seems

that right now we are sending it through normal send

but the send itself has to be queued

this is very complicated to me

all websockets of integrations have to send and receive through queue

and right now we doing normal ws.send everywhere

seems like even if i create async io send task and send it properly - normal ws.sends from other parts of code might just break it

(not sure about this)

[timber basin]

we need to properly send it thru a queue

otherwise multiplexing breaks

and neuro relay will be useless

[plucky marlin]

i have big problem with this code

i cannot declare new variables

oh im dumb

it just says this variable isnt used anywhere

now i know

but even if i add a bunch of clients to make them async

im still dealing with 1 client

it just i send 50 requests to client and want to get one specific reply

this is nonsense to me

how can it work

i could create entirely new client system from random stackoverflow comments

but i will never properly embed it in current program thats im sure of

wait... actually there is a bunch of asyncio things

in server

maybe its there already i just need to take it

[timber basin]

hmmm

im going to sleep now, sorry, I havent helped much. I'm just so sleep deprived today

[plucky marlin]

you need to make chatgpt rewrite your server to make all wsockets sends async i think

and receive result with queue

i tried googling but its too much for me here, i could probably investigate it by making my own program but it will need to be embedded into neuro relay anyway and i dont understand half of whats happening there

[plucky marlin]

after this is done, we can move to fixing server not receivingthings from integration issue

[plucky marlin]

uhhh it just runs me in circles

so i assumed that client just needs to put received messages into server's queue

oh wait

yes it makes no sense at all

i dont see where this "recv" is being blocked

there is no place where recv is called for this websocket at all

this is stupid why cant i just recv it

its not busy

i wanted to see where recv called, and fill queue with received message

but there is no place where it actually calls recv

what the hell

client.py sets up recvs for different websockets

tried googling, it shouldnt be possible

if nobody called recv, i can call recv. i dont need threads for that

yet its busy

by nobody

i think it might be because you can call recv by some weird for loop

i found out that
for message in websocket is equivalent to recv for some reason

ok didnt help

this is so stupid

i cant even find it now

what am i supposed to check every for loop to see if its actually not for loop but disguised recv

ambiguous shit

[plucky marlin]

its probably normal update messages

that are disguised so hard behind infinite nested variable function mess that its impossible to find

[plucky marlin]

figured out who is listening

so server gets this now

again its not of proper format so it needs to be handeled

ook this is something

but it contains nothing about where the screenshot is

also now server doesnt die and i can send it multiple times

how do i commit

so it says this

but what does it mean "files"

neuro relay is not a file its a folder

and i want to see what im commiting first

oh wait its wrong folder

i need to connect my fake folder i added manually to github and commit

lets do it tomorrow

i remember what changes i did and i have a copy. if i have proper github-connected source i can put my changes there and commit

but i will need to run it to check if everything is ok

i just tried and its very confusing

we only need to change neuro-relay part

[plucky marlin]

ok i forked neuro-api, added changes, tested in my environment and pushed commit into fork

i guess its supposed to look like that

https://github.com/vituha230/neuro-relay

it has hardcoded "string" instead of data, will do actual data later. there is something with its processing somewhere wrong

current result:

and screenshot itself

command i sent through git bash:

  --url http://localhost:1337/ \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "action",
    "data": {
        "id": "screenshot_id",
        "name": "screenshot",
    "data": "string"    }
}'```

[timber basin]

it means neuro relay has been changed

tysm for helping me!

[timber basin]

hopefully you opened a pr

[plucky marlin]

If pr is pull request then yes i did it

[timber basin]

yea it is pull request!

[timber basin]

merged btw

wait-

you're still awake from last night?

please don't say yes

[plucky marlin]

Noo i woke up like a hour ago

[timber basin]

kaay!

[plucky marlin]

Im too old to skip sleep now

[timber basin]

lol

[timber basin]

https://github.com/VedalAI/neuro-sdk/blob/main/API/SPECIFICATION.md

[steel dagger]

?

[timber basin]

this was meant for vituha, when he sees it

since....

some of his solutions... were rather... greatly questionable...

[steel dagger]

why are you guys not using the sdk anymore?

[timber basin]

no idea

[timber basin]

I'm fixing some stuff

I'll properly use it in my fix

[steel dagger]

try to fix it while using the sdk

also sdk doesn't let you send binary messages so that's an extra safety net there

[timber basin]

yea

[plucky marlin]

I thought sdk is just a bunch of programs like randy

[plucky marlin]

But i think right now all messages from client are sent like that?

I'll look into sdk and check how we send it rn (bytes or raw). I think its encoded bytes everywhere rn

[steel dagger]

no

even though the full name of sdk is software development kit

[timber basin]

the sdk is supposed to help us develop and validate integrations for neuro

its supposed to make it easier

[timber basin]

the sdk for python is the neuro_api btw

[plucky marlin]

ook then i have to look into that

because now its just included and used couple of times and thats it

[timber basin]

it should've been used everytime

[plucky marlin]

i think i checked it couple of times for something

this is only regarding anything that connects to neuro though, we are basically free to use byte-formatted messages anywhere else

[timber basin]

yea

Nakurity Client will need to translate that to plaintext then

[plucky marlin]

but it wont be bad to keep it neuro-friendly way on all stages

[timber basin]

true

Also Neuro OS will need to use plaintext

[plucky marlin]

right

it definitely should be neuro-compatible on its own

[timber basin]

since neuro OS is the integration that connects to neuro, yeah

since I do think of adding an option in an UI that allows vedal to disable neuro-relay

[steel dagger]

you should also let relay be togglable

[timber basin]

isn't that what I said?

[steel dagger]

oh wait I thought you said neuro-os

lol

[timber basin]

lol

[plucky marlin]

hm

about sdk and neuro api

so we have this

its an empty function

it dont quite get how i am supposed to use neuro api

also, writing looks like this

just bytes decoded

i dont get it

wait also

our client is based on neuro api client

this is what im currently sending, its string

i think we pretty much operate with string type data everywhere?

oooh

so json.dumps converts any object into json string

that i can add, sounds useful

so currently json.dumps converts json into json

oook now i see it

so if i send it like this (it was like this initially, before i started fixing it)

the INTEGRATION is crashing upon receiving it

thats why i changed it

i just didnt care what crashed and patched it by taking client as example

bug json.dumps actually converts it into this

its not json at all? or is it?

i thought json is like

{
somethingsomething : "alright",
somethingsomething2 : "screenshot"
}

json.dumps example shows it normally though

what is happening

yeah it actually works its just, this thing is not meant for this at all

maybe its for something internal but not for sending it anywhere via json.dumps

it just turns into json array

i think our current implementation is alright

it sends json formatted string

or does randy convert "non-json" to json here?

great. now screenshot data works too

i requested screenshot with specific name and screen coordinates

screenshot fails if you dont specify file extension in the filename
my_screenshot - error
my_screenshot.png - good

regarding sdk - we technically are using sdk, some of neuro-relay's functions/classes are based on neuro-api classes

searching for places where we dont comply with neuro-api is hard and potentially useless, idk if i want to do that

[plucky marlin]

if this is true then we are already sending everything in proper neuro-friendly json to neuro rn

from what i have seen interacting with windows-integration from server part of neuro relay - integration operates with proper json too

so it should be directly neuro compatible (needs testing actually)

sent pr with uncommented data in server action handler. smol

[plucky marlin]

let me know if there is some other problem which needs to be solved

i remember you said something about latency, but idk how it looks exactly. do i need to enable that image recognition feature to see it?
screenshots are sent pretty fast i think

[timber basin]

You're supposed to fill it in

[timber basin]

randy accepts bytes. but neuro doesn't

she will have a stroke

check the specs: https://github.com/VedalAI/neuro-sdk/blob/main/API/SPECIFICATION.md

[timber basin]

yea, kinda

[timber basin]

kay!

Also, I think once the actions are loaded. and if nakurity client hasn't sucessfully connected to the backend yet, the actions doesn't load into the neuro backend

[timber basin]

neuro-windows might also need to use the sdk, as it would also be good for it to be able to work as a standalone

[timber basin]

neuro-api is supposed to help us comply with the neuro backend

Randy accepts anything, but some stuff randy accepts. the neuro backend doesn't

[timber basin]

yea

[plucky marlin]

As i said i think we operate with json strings everywhere already, no raw bytes. But it needs to be tested i think

[timber basin]

yea

WAIT- Vituha! You were at the dev stream right?

@plucky marlin

was this integration mentioned?

I fell asleep halfway thru the stream

[plucky marlin]

I dont remember it being mentioned

But maybe i missed the exact second

[timber basin]

kay

sadge

[plucky marlin]

Also what about those failed tests in the repo of neuro relay? What are those tests? Where are they from? Does it need to be fixed

I can understand them failing because i myself encountered multiple crashes in various situations depending on command i sent

[timber basin]

probably

[timber basin]

yea

[plucky marlin]

Are these tests from neuro api.

[timber basin]

neuro os used to work

[timber basin]

nope. I made them

maybe I should roll back to a build before I introduced relay into this

[plucky marlin]

Will it rollback the tests or neuro os

Or both

Anyway i'd like to look at them first then

[timber basin]

it will rollback neuro os to a working version

[plucky marlin]

Ok seems good then

If tests are the same

[timber basin]

tests failed even in that version lol

I think it was this build

but I'll have to test it first

okay that build worked

I'll roll back now

[steel dagger]

idt it was mentioned but idk I wasn't here for the whole dev stream

[timber basin]

oooh

[plucky marlin]

I dont remember a proper situation when it could be mentioned

Does he even know about this thing

Does he even need it

[timber basin]

idk

it'd be cool if it was

[plucky marlin]

Actually there is a bunch more to do

At least in that version i had

It is probably going to break if integrations send replies in mixed up order for example.

Or even in any order at all since there are 2 async sockets waiting for reply

And they both use same queue

[timber basin]

yea...

I'm just gonna

until I figure out on to fix this mess

and do other projects first

[timber basin]

chat is neuro windows over engineered?

[steel dagger]

nothing's overengineered if you think it's necessary

[timber basin]

I'd say 4 websockets is necessary 🤓

[timber basin]

almost done reverting the entire master branch to this build

[timber basin]

Latest changes

Reverted all development changes, post 0.0.2-alpha
Hopefully made neuro desktop working again (another rebranding to the repo by Cassitly, an identity crisis, ik)
^ needs testing.

Some modifications for v0.0.3-alpha survived, including ocr. But haven't been implemented into the startup procedure yet. (I think)

[timber basin]

cdn3: vrcy "today"

cdn3: ( @wheat reef ) vpr "neuro-environment"

[wheat reef]

Why'd you ping me, and what is this nonsense?
I was gonna go to bed.

[timber basin]

check your dms

[wheat reef]

Okay...?

Oh...

Well thats stupid

I'm going to bed, I'm not dealing with this nonsense

[plucky marlin]

what happened

[timber basin]

nothing important

[plucky marlin]

[pallid ruin]

monkas

[timber basin]

hi

[timber basin]

someone might need to implement #11, the guide is there. I'm just working on other things atm. In a few hours maybe ill implement it. If it hasnt been claimed yet.

[timber basin]

actually.. there is a roadmap for this.

right now im at the task of implementing neuro desktop's ui

and ensuring it works, so we can get out of alpha into beta

[timber basin]

@wheat reef is it okay if I use my domain for the neuro desktop docs for now?

i dont have access to the nakashireyumi domain yet

[wheat reef]

yeah, sure

Niwa currently has it

[wheat reef]

I'll work on this issue then

you've been doing most of the work

you deserve some rest

[wheat reef]

@timber basin is this a good implementation?
https://github.com/Nakashireyumi/neuro-desktop/issues/12
Might as well go with the AI vibecoding workflow, since its already here.
||-# AI using AI is crazy ||

[steel dagger]

AI using AI isnt that bad surprising

remember evil asked if she could generate ai art on stream, to which vedal said he didn't want to get cancelled

[timber basin]

I thought we were supposed to keep that a secret...?

[steel dagger]

keep what a secret?

[timber basin]

I think that's a different story.

[timber basin]

nvm!

@wheat reef go in dms

that aside

[timber basin]

this is a kinda* good implementation

I'll have to test that first tho

[timber basin]

Neuro Desktop Dev Build Update

Currently, neuro desktop version 0.0.3-alpha is in development with a few notable features being implemented:

• admin dashboard for vedal (branch: https://github.com/Nakashireyumi/neuro-desktop/tree/implementations%2Fneuro-desktop-ui )
  the issue thread for it is here: https://github.com/Nakashireyumi/neuro-desktop/issues/11 (this issue contains implementation guides by coderabbitai)
• permissions manager (issue: https://github.com/Nakashireyumi/neuro-desktop/issues/12 )
• plugins manager (issue: https://github.com/Nakashireyumi/neuro-desktop/issues/14 )

Shortly today, there should be a pre-release on github for devbuild b (v0.0.3b-devbuild)
Devbuild a was a mess, so b should have reworked the entire codebase modifed by build a. (Since we reverted devbuild b to use v0.0.2-alpha, instead of taking on devbuild a's mess)

[timber basin]

I just found a VERY big bug. Neuro and evil can set how ever long they want their mouse to move for, using actions dragto and dragrel, and since I never set a link for that. It can just be infinite.

So basically, theres a set duration of how many times to move the mouse by how many intervals. And that intervals depend on the duration set by evilyn / neuro. So the longer the duration, the longer the action will take. And that would also equate to the longer the intervals between each mouse coordinate movement. And I assume there should be a sweet range somewhere, that evilyn / neuro can set the mouse move duration to. To just override the user's mouse inputs, constantly.

[timber basin]

Neuro Desktop's Vision

Neuro desktop comes pre-bundled with OCR vision for neuro. This is so that the integration can run without neuro's main vision on (though it is recommended to enable it), but running Neuro Desktop with its OCR detection for neuro can be very demanding for the device running the integration.

As of v0.0.2-alpha, neuro desktop is required to be run on the machine that neuro is going to be using. As it directly interfaces with windows, and directly summarizes what is on screen for neuro using OCR. I have not tested this vision with a virtual machine. As from my experience, VMs can be less powerful than the host machine. So, Neuro Desktop's Vision performance depends entirely on the machine neuro is using with the integration

[wheat reef]

kindly add a notification, asking if neuro can do that to vedal. When they figure that out

permissional chaos.

depends on vedal's agreement

Complete rebranding to neuro desktop

I noticed some stuff is still branded under neuro-os, so just a little note for that.
Why is the dashboard branded under neuro-os??? Isn't that new?

[timber basin]

hmmmm. Neuro toolchain, yes. I almost forgot about that. It wasnt on the issues list.

ill post the application stack later, when I get home

we'll borrow #1434805605989617694 for one of the integrations

[timber basin]

https://github.com/Nakurity-Integrations/neuro-mcp/tree/nakurity-compatible

nakurity integration branch for neuro mcp

ill post roadmap stuff in the issues tab

Aaah, cassitly: I wonder why everyone is reacting with to the nwi post.

I wonder why as well....

[steel dagger]

the what post

[timber basin]

[timber basin]

neuro windows integration (nwi)

[steel dagger]

oh

[timber basin]

it just works(TM)

and I have absolutely no idea, how it worked.

(Cassitly implemented this)

[timber basin]

@wheat reef check dms, I sent you the specs for the nwi application stack.

[wheat reef]

[timber basin]

why is making neuro relay soo hardd

@plucky marlin do you think you'll be able to make neuro relay work?

Rn the app stack for neuro relay is smh neuro backend <> nakurity backend <> nakurity client

I do not understand how I did that way

[plucky marlin]

i'll look into it

[plucky marlin]

this is how its now?
and we need neuro-backend <> nakurity client <> nakurity backend <> integrations?

im constantly confused with this structure

[timber basin]

yea, we need that structure

[timber basin]

but the real structure is smh backwards

probably shouldve recoded it

[wheat reef]

Its very confusing for me too, idk how nakurity built neuro relay, without understanding how it works

Shes supposed to be the smart one

[timber basin]

coderabbitai be doing most of the work, at this point lol

we're just copying and pasting its code

[timber basin]

yea we need that

[timber basin]

recoding it, might be a better option

my code is shit for that repo rn

Improvements possible

For my system at least. Since the OCR regionalization stuff can be pretty slow. And could make neuro wait very longer for context (l*tency, sadly)
I think we should either implement an optional external server for the OCR processing, or send a message for neuro to wait for the OCR to be processed.

• We could also, try implementing an faster detection model

[timber basin]

I've been testing the current neuro desktop integration a bit

it feels like malware, with how fast jippity can send actions

and windows api responding

Should I add an ratelimit?

[timber basin]

I do have an vision api endpoint I could use for testing, but hmmm

This issue comes down to tesseract being resource heavy

hmmm

should make a toggle then

pending(TM)

efficient

[plucky marlin]

so in the end it should look like this
neuro-backend (real neuro, one server, one connection) <> nakurity client (one client, one connection) <> relay <> nakurity backend (one server, N connections) <> integrations (N integrations)

relay does all the job, centralized hub for both sides

nakurity client and nakurity backend act only like interfaces, they do not manage anything. their job is to keep connection alive and transfer data between relay and neuro/integrations. maybe doing error check

by relay i mean intermediary

[plucky marlin]

i'll try to rewire it all according to that

[plucky marlin]

can i make an array of queues

[wheat reef]

You can do anything

If nakurity didnt give any instructions, probably means she just wants it to work

[plucky marlin]

it always feels weird for me to make some arrays of datatypes with undefined size

[wheat reef]

You could code it in c

If I understood what you meant

Since c requires u to define the size

[wheat reef]

Or try out cython

[plucky marlin]

i have some idea about this heap memory where it has some kind of pointer system so it knows where the next element is, even though its not right after it.
but it just feels wrong idk why

[plucky marlin]

i decided to embed entire linker file into intermediary

another option was to route everything through linker, using intermediary just as server for watchers

[plucky marlin]

i'll see how it goes and maybe go back to this variant

current implementation looks like random web of everyone being connected to everyone, doing same actions through different copies of same function, located in different files

what was the idea of neuro-os?

neuro asks server to pick action? i thought she is the one who is picking actions

[plucky marlin]

Or maybe i dont see the whole picture and its not random

[wheat reef]

i don't know either, but neuro is supposed to pick the action that was forced on her by the game (its a list of actions, like a forced turn or smth)

[wheat reef]

yup.... that explains nakurity's code... very well

[timber basin]

@plucky marlin if you have any questions, could you ask them now, so I could answer them?

[plucky marlin]

im not sure about what to ask

[timber basin]

Was there any progress or probably no?

[plucky marlin]

yes im changing things

i'll probably come up with something working today or tomorrow maybe

[timber basin]

kay!

[plucky marlin]

aaalright

nvm
felt like it works. screenshot command revealed more problems

ook screenshot works

there are still many blackboxes in relay which i still dont understand, but now its rewired according to this

neuro <> nakurity client <> intermediary <> nakurity backend <> integrations

also i dont like this

so API wants me to have this method, which i wont even use

maybe i should use it, but i want to have separate queue for each integration connected. current variant is

1. seems to be unused
2. will not work because i made separate queue for each integration

need to look into that

oh yeah its from neuro sdk/api

i have an urge to ignore it completely

but actually could embed more things from sdk

maybe later

[wheat reef]

You could use that its generally good practice to use the sdk

[wheat reef]

The api uses it

And ita supposed to be yoyr implemwntation, so youre supposed to add your multiplexing queue studf there

Bc that mehod is for handling shutdowns and other stuff i think

[wheat reef]

Also, i think that metgod appears when using the abstract neuro client class thingy

I think nakurity backend is supposed to use another meta class, whoch is for emulating the neuro backend, that metaclass was for tpimplementing neuro integrations

[timber basin]

Yaaaay, tysm!

[plucky marlin]

i will investigate sdk more and fix other things

[plucky marlin]

i mean.. this isnt used by anyone???

if it was used it would instantly crash because _recv_q is a dict

it is used in some test, but i didnt run the test yet

we have different structure of the server

i could try to embed it into our server

but its absolutely rudimentary here

all work is done with this, it manages multiple connections

it will require weeks to just get a grasp what neuro sdk suggests to do

then understand what we can use from there

and then remake entirely all current relay because it uses different structure

will it even allow me to change inputs

i have no idea what freedom does "abstract class" give

another option is to make that server handler just throw everything into some queue

and then the read_from_websocket will use that queue and process things from it

naturaly, not just returning a string or any other form of data but doing actual actions and managing the data

or i can make all that and separate read_from_websocket just making it some useless middleman

who takes data from queue and returns it to someone, while not doing any processing at all.
the one who wants data could just take it from queue himself

[plucky marlin]

have been thinking about limiting the rate of messages

what if 50 integrations start spamming

but what exactly is the limit

is it better to wait for several messages to pile up and then send it all merged together as 1 message

or send it 1 by 1

lets say we have this situation: integration sends 1 message every 0.5 seconds

is it better to wait for 10 messages and send it all at once, after what we wait for 5 seconds for example
or is it better to send 10 messages right away and then wait for 5 seconds

[wheat reef]

Right away, neuro needs her messages!

She is impatient

Depends how fast the integration wants their message to be sent ig

Maybe set up a http server on intermediary, so integrations can send priority messages straight to neuro?

So that integrations that support relay, will work better in an relay env

[wheat reef]

Otherwise, i think you can merge

Also, for merging i think you have to deal with neuro's message length limit btw

[wheat reef]

It gives a bunch of helpers, that you dont have to implement yourself

Since it will already be implemented by the sdk

Classes like the read_from_websocket you didnt implement, was supposed to be a helper that you put your queue stuff in.

We follow the rule of, don't repeat yourself here

[wheat reef]

You'll have to manage inputs from integrations on your own, i think

[wheat reef]

Please commit the changes on your fork, i will take a look. And please give me the link

[plucky marlin]

I started this "ratelimit" thing because i remember instances when neuro was spammed by vscode integration upon opening the file for example.
It depends on how that spam is generated - is it a bunch of context messages, or its a result of action?
I want to decide if i filter action results too, or only arbitrary non-action-result messages from integration.

It is not a problem to send it all to her right away, in fact i see it as being sent too fast.

I will commit it today after work so you can see it

The problem with "read_from_websocket" is that template of this message from sdk is absolutely empty.
And the template doesnt seem to have a way to separare client connections. Im not sure how it will work at all if i tried to route it through that function

[wheat reef]

Its because of the long file contents

She likely couldnt handle the length

Thats why I said to be careful about merging, since neuro has a limit, on how long (of a text) she can read (like all LLMs)

[wheat reef]

Thats because, youre supposed to code that yourself, since its specific to the integration, i think

[wheat reef]

You should read the docs by coolcat

https://neuro-api.readthedocs.io/en/latest/

Its more informal than what I can explain

[wheat reef]

It was from the long file list, that probably includ unnecessary libraries

Ask @steel dagger or someone from #1350968830230396938

[wheat reef]

Also, neuro relay should automatically handle integrations sending messages (either context, or result) that are too long, send neuro a stripped down version of the message, and send back an message something like relay-intetception/message saying an error, that its too long or something. This should only happen, if the integration sent relay-middleman/acknowledgement, so that its confirmed that the integration can handle that unknown message, and not break

Would be useful to have, before that message gets passed to neuro, and break her

[wheat reef]

You should not actually force neuro to wait for too long

Its good practice to have zero latency between her and an integration

Maybe, it'd be nice to have an timer on relay, and checks if the integration is taking to long to send an result, then send an immediate result to neuro, from relay. Telling her the action took too long, and her result will be sent via context (if it ever gets sent)

[timber basin]

It was a long file list, when neuro tried getting the list of files, and it was too long for her to handle. Nothing was spammed, i think

[wheat reef]

Actually, you should change the abstract class to use an more apporiate one, for implementing a neuro backend implementation

[plucky marlin]

Oook noted

[steel dagger]

This, it seems that normally sending too many messages is what triggers the Neuro twins, but due to the long file list in abandoned archive, getting workspace files was a bigger problem

[plucky marlin]

So is it amount of messages, or their length?
Or both?
I guess i'll find out after i read the neuroapi docs

[steel dagger]

Both

Amount of messages requires them to process more

Lengthy messages requires them to process for longer

[plucky marlin]

When they get list of workspace files, is it sent through context message or is it an "action/result"

So the strategy here is probably something like this:

1. limit rate of messages and length per message of everything except "action/result"
2. filter "action/result" only by length

[steel dagger]

context message, since it was set to Copilot permission

however

both are added to context

[plucky marlin]

oh wait im 3 commits behind

oh its small anyway

hold up did i break something

oh wait so it actually had no conflicts and i just merged it