Neuro's Canvas (Neuro Integration for a Simple Painting App)

[elfin shuttle]

This is a simple painting app I created in Python to allow Neuro to make some doodles.

Everything is in this repository, no separate installs required:
https://github.com/Kaya-Kaya/neuro-canvas

The app has a limited set of options right now. I'm open to suggestions for improvements, and you're also welcome to pull request any changes you wish to see!

A demo of Jippity attempting to draw a turtle

[bitter skiff]

Could be cool to see her try and draw, ofc with her controlling it and not just another ai doing everything.

[celest bay]

I don't know how crazy would go if neuro could grab the pencil and say "line from coordinate to coordinate". Maybe it's way too advanced

[elfin shuttle]

That's actually already an option. If you're curious what it looks like with only that turned on, here's Jippity drawing a house. Neuro probably wouldn't be able to do anything too crazy with it but I imagine she could draw some really simple stuff like this.

[low cosmos]

Could she draw curves properly by expressing it mathematically in her head and then drawing the function graph?

Far from true hand-drawn but it's something

[celest bay]

There's also curves by points. There're tools for this. What they're called? Beizer curves?

[low cosmos]

Wouldn't the twins get stuck sometimes by trying to define infinite points? They tend to loop with words but idk how LLMs handle math.

[celest bay]

Good hecking question. I'd imagine it would work like human tools... select a point, select another, select another, modify some point, delete some point, repeat until satisfied.

The trick is... every action should be... an action. Like us. So yeah. They could be stuck 100% forever on one simgle curve unless given vision for it

[elfin shuttle]

The integration supports bezier curves as long as Neuro gives the points for it (although its not antialiased but this can be fixed).

[elfin shuttle]

They have to define all the points in one go so they probably wouldn't get stuck

[elfin shuttle]

Here's Jippity using only the curves feature. It said it was making a landscape. From what I can tell it doesn't have a super good understanding of the result of its actions for this one as compared to the other simpler tools, and prefers to use the minimum number of points

[celest bay]

But that already feels more natural. It's progress

[dusty solar]

ah yes, the new form of ai art

[dusty solar]

I feel like Evil would draw a house and Neuro would just roast it for looking bad

Or something like that

[dusty solar]

@elfin shuttle sorry for ping, I just pushed the changes you asked for + a readme change and requested your review, could you take a look at it when you can?

also what are you working on adding? planning on more contributions after this so I need to know if you are working on stuff so I don't overlap

[elfin shuttle]

Merged!

[elfin shuttle]

I'm not currently working on anything, so you're good to make any changes.

[dusty solar]

great

a friend of mine saw what I was doing and instead of expressing concern over me sneaking in more power for the neuro siblings in some fashion she gave me a list of tools she would add to a basic Paint-like app

[elfin shuttle]

based friend

[dusty solar]

agreed

[dusty solar]

@elfin shuttle maybe do feature releases in batches because I'll be PRing these in slowly

[elfin shuttle]

alright

[timber thorn]

Can the AI write a math formula to create a specific line like a graph?

[elfin shuttle]

No, the most we have are bezier curves. It sounds like overkill in terms of customizability to me. That being said if LLMs do better with using math to graph pictures it may be worth adding.

[jagged trench]

For AA I was going to recommend attempting to use SVG for the graphics, but I'm not finding the best of resources on easy ways to use it with your current stack. It is a bit more standard, plus some AI's including ChatGPT have it within their knowledge base. Just a suggestion as I was scrolling through projects.

I know it somewhat kills the idea of fully sketching, but at least it is an easier option for shapes. Plus the lines already contain the same math for drawing them.

[dusty solar]

I may have almost lost the sheet of paper that my friend gave me

[elfin shuttle]

I think using SVG is a good idea but it doesn't mesh too well with this project. At that point we'd be building Neuro an SVG editor which should probably be a separate app. I'll probably ditch anti-aliasing altogether and embrace the MS paint look

[jagged trench]

I do think there would be a way for you to use SVG as the method of drawing as it already implments the lines and curves to at least use the anti-aliasing, it would also not have to worry about saving and reloading. But alas, the MS paint look is also a valid direction; thank you for your time for considering this apporach.

As a side note, if anyone hasn't tried manually creating SVG files, I'd say it is worth your time. It is not as daunting as one may think.

[elfin shuttle]

It doesn't really make sense to me to wrap it around the functions in the app if its just SVG behind the scenes, the LLM would probably have a better time editing the SVG directly. I do think building an SVG editor for Neuro is worthwhile though

[elfin shuttle]

Good to know, I've never done it

[jagged trench]

It does entirely depend on Neuro knowing how to edit a SVG. I think the format is neat for storing the images in general. The only downside being that discord and other sites don't embed them as images due to code execution concerns of the format so one would still need a method to convert to PNG.

[jagged trench]

An SVG editor would be pretty far out from drawing and would be it's own thing I agree.

[elfin shuttle]

I would guess she probably can edit SVG. For saving drawings I think rendering to PNG shouldn't be difficult with the way the app currently works

That is a feature I want to work on soon

[jagged trench]

I hope that it is very easy for you to accomplish!

[dry locust]

very cool project

[dusty solar]

oh shoot I never PR'd in layers

[dusty solar]

@elfin shuttle I forgot to ask you to check my PR changes, I'll just ping you for that since this feels like a month overdue?

[dusty solar]

@elfin shuttle can't add @update_display to layer vis & remove actions in canvas since they aren't defined/imported

[elfin shuttle]

oh just move update_display to the top of the class

[dusty solar]

oh it's cause it's before the decorator

lemme fix that rq

[dusty solar]

@elfin shuttle dooooonnnneeeeee

sorry for the delays

[elfin shuttle]

nice

[elfin shuttle]

nah youre good ive been pretty slow to review

[elfin shuttle]

@dusty solar I've figured out how to make undos work with layers (and hopefully more features in the future). It involves some big changes to the canvas script, wanted to check if you were working on anything before I push

[dusty solar]

nope, all good to push that out if you want

[dusty solar]

hey @elfin shuttle how much are you going to keep this to "simple"? As in, what's the limit on the extensiveness of this building app?

because I might start using the little-known app called "paint" as my reference point

goddamn it i should've named the branch i-spilled-the-bucket

[elfin shuttle]

not sure tbh, what features do you have in mind? i think more stuff would be good as long as theyre usable to neuro

i might split up the actions script into multiple files if we're planning a lot of different actions

[dusty solar]

this entire toolbar (except the copilot which I blanked out because then it's not neuro drawing anymore is it?)

I think we'll split it up into "visual actions" and "editing actions" - only in case the user wants to see what was drawn a bit better they can ask to zoom it in with visual actions, editing actions are self-explanatory

share probs will also be gone for obvious reasons

[elfin shuttle]

alr im down for that

[dusty solar]

cool

I will probably get started on branches today buuut I had my blood drawn yesterday and it still is somehow hurting so we'll see how I go

I think you already have the save action right?

[elfin shuttle]

yea

[dusty solar]

does it let you save as any other file format?

[elfin shuttle]

not currently but it would be easy to implement

it should probably be a config option and not something that neuro controls tho

[dusty solar]

ehhhh I disagree? at least making it as like

list of allowed options

because i'm not sure if it will always be good to save as one file type

[elfin shuttle]

actually ive been thinking about making what actions are available to neuro configurable, ig i could do it so that you can specify which options neuro will have

[dusty solar]

right, so like permissions

[elfin shuttle]

yeah pretty much

[dusty solar]

cool

[elfin shuttle]

in case she overuses something it might be nice to have

[dusty solar]

I can see if I can work on that? probably will make it a separate file like permissions.py

and then would a config.json work as well?

[elfin shuttle]

yeah that works

[dusty solar]

alright

hm actually I might combine the logic for both into config_reader.py

[elfin shuttle]

like the saved configurations would still be in config.json?

[dusty solar]

yes

I might make it json5 or yaml instead but if those don't work out (and I'm certain they probably won't) I will just default back to json

[elfin shuttle]

not familiar with those but whatever format you prefer is fine

[dusty solar]

ok

would it also be alright to

switch to using uv for package management

as well

[elfin shuttle]

ive never used it

whats the advantage

[dusty solar]

it's a package manager written in rust (lol ofc it is)

but like it's really fast and also just really nice
also has a lockfile (the best part for me)

I don't remember what else it has

since that's all I remember off the top of my head

[elfin shuttle]

alr im gonna look into it

is there anything that would need to change in the repo

[dusty solar]

no

in fact the only thing changing is adding a new uv.lock file

[jagged trench]

I do prefer uv, but that's just because it is a one line change in my env file for very fast venv activation.

[dusty solar]

wait what do you need to change in your env file

[jagged trench]

I blame Nix for all of my issues

but yeah, at least for what I use activating it is an extra line

[dusty solar]

ah

nix is weird

[elfin shuttle]

@dusty solar ive split up the actions script so having a lot of actions should be easier to manage now, hopefully i didnt create any merge conflicts

[dusty solar]

config.json will be split between 2 top-level keys: settings and permissions

@elfin shuttle Would it perhaps be a good idea to build a separate GUI for toggling permissions and settings and such?

or do we just leave it as json-only

[elfin shuttle]

it might be but i think its low priority

[dusty solar]

I'll get config.json working first ofc

[elfin shuttle]

we'll see how difficult it ends up being manually editing the json

added other filetypes to saving btw

[dusty solar]

Hey @elfin shuttle is this a good structure for the config.json file?

{
    "configVersion": 1,
    "settings": {
        "allowed_save_file_formats": ["png", "jpg"],
        "canvas_size": {
            "height": 1920,
            "weight": 1080
        }
    },
    "permisssions": {}
}

[elfin shuttle]

permissions is which actions are allowed right?

if so then looks good

[dusty solar]

🔥

[dusty solar]

@elfin shuttle what file formats does the app officially support?

[elfin shuttle]

bmp, tga, png, jpg

[dusty solar]

cool

[dusty solar]

@elfin shuttle do I just split permissions by the files (as in split by the functions in background.py, brush.py, draw.py, layers.py and misc.py)

[elfin shuttle]

sure that sounds like a good way of organizing

[dusty solar]

I tried to implement a control class (currently testing AbstractLayerAction) (is that even the right term) to use in place of AbstractAction and it just crashes on startup, can you take a look when you wake up?

[elfin shuttle]

its trying to register abstractlayeraction as an action

i was thinking we wouldnt register actions that dont have permissions in the first place

so in the __init__.py in actions

also can you add fine-grained (per action) permissions as well? i thought you were just asking about how the json should be organized, i didnt realize you were doing permissions by groups of actions

[elfin shuttle]

fyi i wont be able to respond on discord next week and a half or so, message me on github instead

[dusty solar]

that imo is a bit much

[dusty solar]

ok well that makes more sense

[dusty solar]

hmmm sure but that does require a little workaround? I'll look and see how it's set up right now.

[elfin shuttle]

it should be pretty simple if you use the if statement in __init__.py

and add what condition you need to check

[elfin shuttle]

for some of them maybe, but for the draw actions it makes sense imo

same with misc

[dusty solar]

hm

alr

[elfin shuttle]

i can add them after merging if you'd prefer not to work on that part

[dusty solar]

I'll see if I can work on it

[dusty solar]

actually, would it be better to just attach a new property that returns the string identifying the permission it needs to check?

[elfin shuttle]

wouldnt that be either the action or file name

[dusty solar]

I just grouped some of them together still so

[elfin shuttle]

alr that sounds good

[dusty solar]

@elfin shuttle I defo did not forget about ths

[elfin shuttle]

i def didnt either

[wraith cliff]

[modern plume]

this seems cool

[distant barn]

Why not make the canvas an coordinate system and neuro could draw on individual pixels.

[elfin shuttle]

it would require a lot more actions and probably be more difficult for neuro to use, that being said it could be an interesting idea for pixel art specifically

also the canvas is a coordinate system, but it's edited using actions like draw line, draw circle, etc. as opposed to individual pixels

[timber thorn]

If she can draw a line from x,y to x,y then she could probably draw tiny dots if she wanted.

[elfin shuttle]

we could certainly add that feature but the dots would be too small to be meaningful imo given that we have a 500x500 canvas

[timber thorn]

Oh I just assumed that's was it's already doing

[elfin shuttle]

oh i misread your message mb, yeah you're right

[distant barn]

I mean, you could just return the pixel map back to neuro. And she returns the pixel map that has her edits back. Please correct me, if this is not realistic. I havent quite poked around much at the sdk yet

[distant barn]

Could the sdk return images yo the neuro backend? If so, that could be easier

[elfin shuttle]

no, only text

[elfin shuttle]

it would be a lot of data both being fed to and received from neuro so im skeptical it would work well. for a 500x500 canvas thats 250000 pixels

[distant barn]

Pixel map?

[distant barn]

Why not focus on the pixels neuro is drawing on / parts that makes the canvas into the current image

[elfin shuttle]

then you lose the context of the full canvas and may as well just use a smaller canvas resolution

[distant barn]

True.

Idk

[elfin shuttle]

i think its a good idea for pixel art though

[distant barn]

Kinda yeah

I do have an idea for image drawing from math

Idk if neuro could do math that well or not. I saw a video about someone who used math to make art.

[elfin shuttle]

she probably has some math knowledge in her weights, the same way she has coding knowledge, so it might be feasible but idk

especially if using a common format like latex

i have no idea how i would program it though

[distant barn]

I could help

How is neuro seeing the canvas currently?

[elfin shuttle]

she's not, the idea is that she would have vision enabled although even if it wasn't she can figure out what's on the canvas by what actions she's performed

the challenge with drawing from math is parsing the latex or whatever format and performing calculations which idk how to do

[distant barn]

What if we use a different algrothim for the image drawing. And neuro input the text thingy, probably would defeat the purpose idk.

I know how to, atleast

[elfin shuttle]

im not sure what u mean

[distant barn]

Like, kinda a different ai?

[elfin shuttle]

a different ai for what?

[distant barn]

Nevermind. Im too stupid to explain. Sorry, i can help with parsing the math input from neuro onto a graph as an image

[elfin shuttle]

that's alright lmao. i think math input could be interesting, you're welcome to pr

[distant barn]

Okay

[dusty solar]

@elfin shuttle resolved conflicts on my config.json pr

[dusty solar]

just realised CI failed on that branch

idk why tho lemme see

considering that code shouldn't have had tests written for it

[elfin shuttle]

its because no actions are registered, can you switch to enabling all permission by default

[dusty solar]

...

oops

ill do that tmrw when im free (im tired and need to write smth)

[elfin shuttle]

yea no worries

[dusty solar]

cool

[timber thorn]

I remember a discussion about something like this before

[timber thorn]

I'm assuming you either mean an algorithm that turns Neuro's prompt into a math equation or you mean a different integration that works along this one for the same drawings.

[honest terrace]

This is cool. Need to read through how this was done later

[dusty solar]

Sorry y'all

[timber thorn]

How different is this from turtle?

[dusty solar]

well for one all the bindings are directly as an action instead of having to script them all

[timber thorn]

Sorry for my ignorance, but is this a library that could work along turtle? Or they use completely different canvases?

[dusty solar]

They would use completely different canvas iirc

[timber thorn]

Oh that's right, those are their own integration functions and app, not a library.
I guess dynamically drawing has its own use though.

I wonder if it's possible to add other drawing libraries to this and let her write small scripts.

[dusty solar]

just saw this again and thought about it, theoretically possible but we would have to add bindings for those to neuro

and if we want to make it so that she can use any library and she just has to script it then that sounds... awfully familiar

[timber thorn]

Just the idea of her using different ways to draw on the same canvas and the ability for her to save the image and/or restart it.

Although, the problem in general with the concept of her drawing in steps is her knowing what she just draw.

[dusty solar]

I mean we can forgive Vedal for not turning on vision last time when neuropilot was used to draw with turtle, but if he literally doesn't allow vision for the canvas integration then idk what to even say

[dusty solar]

saving is already a thing, and idk what you mean by different ways

also wdym by restart?

[timber thorn]

Like, I'm saying this without really anything in mind, but I was thinking that basically, instead of each time creating a new image every time she can decide to continue on the same image, save the current image and/or restarting from blank.
She could use any library, function or action she wanted, existing or made by us.
I think for this to exist you'd have to either:

• make an app or library that let's you(neuropilot) draw in steps on top of the existing canvas without erasing and then you could create any custom function you want if they don't exist already
• or more wackily an entire app with these same features, script parsing, custom commands, etc...

Idk if I made sense.

[dusty solar]

[..] I was thinking that basically, instead of each time creating a new image every time she can decide to continue on the same image, save the current image and/or restarting from blank.

Currently not very possible unless we define our own image file format (like photoshop or gimp)
Unless you mean that she can just load an image and add her drawings on it, similar to downloading an image from google and making edits (might work)

She could use any library, function or action she wanted, existing or made by us.

see above, this also means the canvas app has to be pluggable which sort of breaks the part where this is supposed to be a simple drawing app
although we probably could drop the "simple" pretense...

I think for this to exist you'd have to either make an app or library that let's you(neuropilot) draw in steps on top of the existing canvas without erasing and then you could create any custom function you want if they don't exist already, or more wackily an entire app with these same features, script parsing, custom commands, etc...

this is literally just coding using turtle but with more steps and in an app not designed to be like that

it should be simple for neuro to be able to draw something, without needing to go neuropilot mode

[timber thorn]

I mean yeah, it's obviously convoluted because I was imagining merging all the functionalities from both ways of drawing, but I can see that their points might be too different.
The idea of saving/downloading the image and use it as background for the next drawing action is basically what I meant with the first hypothetical idea, idk if you can do that on a python library (I could just Google tbh).

[timber thorn]

My second hypothetical idea was an app with Neuro integration where she wrote code as an action similar to neuro pilot and the app basically would do the same as she did with python turtle, but I'm not even gonna delve into this one lol.

.
I reformatted my original message, just in case

[dusty solar]

iirc the current lib being used doesnt support images as bg but I could be wrong

[dusty solar]

heartheartart scripting language

but fr tho at that point just put the -pilot and canvas people together and make a neuropilot container at that point

no need to reinvent the wheel for the second time on both counts when you can just make some small changes and put the two together

[timber thorn]

The main idea of mine was sharing the same ongoing canvas with whatever method she uses.

.
Seems like turtle doesn't let you draw on existing images, but other libraries could.

[dusty solar]

potentially

if I can find enough research I'll maybe do a prototype but I have other stuff to do so...

[elfin shuttle]

I’m pretty sure it does

That would be a good feature

Also making our own save format might be doable

The app already keeps track of action history

[dusty solar]

well

I wouldn't mind getting back onto this project again

[distant barn]

I might help

[dusty solar]

then making it pluggable! wait- why does this feel familiar?

do all roads lead to api?

[distant barn]

also, what does pluggable mean?

[dusty solar]

extendable

aka "hackable"

[distant barn]

oh, kay

[dusty solar]

the reason why I said "do all roads lead to api" is because I proposed to add an api to a certain other integration

[distant barn]

aaaah, I see

[dusty solar]

anyways

3 programmers who also happen to be musicians in one way or another working on a paint-like integration

kinda odd 🤔

[distant barn]

kinda odd

🤔

[elfin shuttle]

just an fyi im not gonna have much time to work on this for a while, i will probably be slow to review prs

[dusty solar]

allg

[dusty solar]

@elfin shuttle do we hardcode the port in this app?

[elfin shuttle]

no

its read from environmental variable

and falls back to the hardcoded one if not found

[dusty solar]

ah ok

[dusty solar]

something something we should probably work on it something something damn we kinda lazy 💀

[dusty solar]

need to confirm if it's canvas or pilot

[elfin shuttle]

welp i guess not canvas

[distant barn]

Maybe vedal doesn't know this exists?

This isn't quite the top post on this forum either

[dusty solar]

Well also tbf there hasn't been any dev work on this for months now

partially myself to blame tbh

But also, with Vedal's insistence on not using vision for these drawing exercises (for some reason), idk how good it would be

[distant barn]

do you think an Vision endpoint for the neuro backend is planned?

[dusty solar]

not at all

[distant barn]

I saw someone in the neuro bot page said they'd suggest it

what happened to that?

[dusty solar]

I suspect it's way too hard to implement something like that without also being expensive or smth, idk

also it should be noted, how the hell are you gonna be able to do that with the constraints of the api as of right now?

I guess you could b64 it and send that off

but also you could b64 anything and send that off, like, idk, a piece of malware payload

also you need to set up filters so that someone can't just send [explosive images] and cause her to say something uhhhhhh bad

[honest terrace]

he did enable vision for yesterday though

[dusty solar]

when?

[timber thorn]

He has never checked the projects channel, except for maybe once or twice.

[timber thorn]

He said he did, can't prove it for sure though

[dusty solar]

I think he said he checks it on a semi-regular basis (and by semi he probably means quarterly)

[dusty solar]

this thing keeps disappearing off my list but it only took one day for it to disappear

[timber thorn]

Save the link

[dusty solar]

to this project

[timber thorn]

Who knows, it might still be useful