programming

[stark needle]

chabgpb

[tender river]

great pritish bounds

[stark needle]

so since ur a tv, can we not be friends? (tv + human = addiction)

[scarlet arch]

I like Ü because it looks like a grinning face

[ruby timber]

parasocial relation with my lg tv

[stark needle]

Perfect idea for new anime

[tender river]

fun fact lg logo is a pacman

[ruby timber]

Please no

[stark needle]

so you lied to us

[ruby timber]

I want a brand to sponsor an anime like this

Just to see what would happen

[stark needle]

I also lied I'm not actually an anime girl, I'm a real living guy

[ruby timber]

And like half of each episode is just advertising the mc's features

I guess it's time for me to come out then

I am not a bash script

There, I said it

[tender river]

i lied i'm not actually asuka from neon genesis evangelion

[ruby timber]

It would be so good

[stark needle]

my tablet has 16384

L

Nerd

[prime plaza]

I regret to inform everyone that I am not Bonnie from one piece 😔

[tender river]

let's be real it would be wacom waifu like the can anime

[stark needle]

the one piece was not real?

[final tinsel]

Western Digital 2TB WD Blue SN5000 NVMe Internal Solid State Drive SSD - PCIe Gen 4.0, M.2 2280, Up to 5,150 MB/s - WDS200T4B0E

Wait

I pasted that in the wrong place

[prime plaza]

oh no it's real

just I'm not

I've been a <gender pending> teenager this whole time (barely a teenager but whatever)
(believe it or not this is Vedal's fault)

[stark needle]

I will steal everything from you one day

[final tinsel]

Yoink man go brr

[prime plaza]

where tf are you gonna put it 😭

[stark needle]

You can live with the big ben

[final tinsel]

South wales

[prime plaza]

idek where that is

we got New south wales but that's it

[final tinsel]

In the south of Wales

[prime plaza]

yeah idk where wales is

[final tinsel]

Earth

[tender river]

https://myanimelist.net/anime/5112/Akikan

[prime plaza]

no way thank you

[final tinsel]

Yw

[ruby timber]

https://tenor.com/view/ohhh-duh-why-didnt-i-think-of-that-gif-21849807

[prime plaza]

is that in the UK or something

[tender river]

theres lots of slop like that

[stark needle]

Put it on the right side on the beach @sage crag

[prime plaza]

funny thing is I have some heritage from wales I think

[ruby timber]

I've been to London, can confirm it looks like this

[prime plaza]

and I used to think their accent was scottish

[stark needle]

Drinking anime girls my beloved

Truly the anime we all needed

This will solve so many global scale problems

[prime plaza]

literally more creative than jjk too

truly the peak of anime

[stark needle]

No

[tender river]

as long as it isnt england

[prime plaza]

it be like that sometimes

[stark needle]

I like watching sunsets

[final tinsel]

Oh for an actual more on topic thing, is more cores better for running lighter ais like mistral with llamafiles? Ik eventually I need to get it running on GPU but yk for now just curious

[prime plaza]

if an alien came to earth and offered to take me to another planet I would take it in a heartbeat

[timber dagger]

the fact that Akikan is rated 6.10 is crazy to me

[stark needle]

Ye

[amber fractal]

[stark needle]

konii be like: ✅

[final tinsel]

Do y’all use XMP for ur memory?

[stark needle]

Ye

[prime plaza]

So what you're saying is that we should throw people into the sun

[final tinsel]

I had a follow-up question for this but I forgot it

[tender river]

it's that or sun explodes, you choose

[prime plaza]

Solves the overpopulation crisis and extends our lifespan as a race, probably

[final tinsel]

Thanos aah mindset

[tender river]

theres no overpopulation crisis...

[prime plaza]

Not now

[nocturne olive]

Yes (mine is also overclocked)

[prime plaza]

But there will be

[stark needle]

i'm finally home

[prime plaza]

Probably

[tender river]

actually its overproduction if anything

[prime plaza]

Aka greedy ass billionaires who don't share crisis

That's a new word

[final tinsel]

I am

[tender river]

i for one am a strict adherent of rule 3

[final tinsel]

1984tran

[prime plaza]

Oh is that controversial

Mb

[tender river]

at least when people talk about politics here i only engage on a very cursory level otherwise i'd be banned years ago (before i joined)

[final tinsel]

I mean AI usage is controversial and good luck avoiding that (but yeah I get the point)

Tutel time

That’s the point

[prime plaza]

Man this has to be the worst possible time for Vedal to have his streams for me 😭 (completely my fault though)

4am, I rarely ever go to sleep before midnight, etc.

[tender river]

its 1am and i only have like 1-2 hours of chores left for today and then i can work ()

i dont know how people manage to get anything done on 24h schedules

you wake up brush your teeth and half the day is gone

[nocturne olive]

I kinda wonder how much Vedal is overcomplicating V3 speaking voice
NeuroSynth works crazy well and it's pretty much just a distil of Neuro's singing voice via RVC (at the moment), I see no reason a similar technique wouldn't work for a TTS

[scarlet arch]

Like idk what you're working on but sometimes I don't understand what the voice is saying when you post examples. I suspect maybe Vedal is having similar issues?

[nocturne olive]

I think that stuff is just tuning issues, which has nothing to do with the Neuro voice itself
And also partially just how we prefer to do songs focusing on complex mixing rather than clear and in-the-center vocals
I'm quite sure the voice itself could do just fine with proper tuning

[scarlet arch]

Ah

[nocturne olive]

This happens when the tuning is incomplete

Something like this with better tuning should be much better

[scarlet arch]

Idk if you intended that to sound funny but it tickles me

I still can't make out some words. Am I having a stroke

It's like uncanny valley voice

[nocturne olive]

It's BETA, it's a 2-step distil rather than the single-step it will be once the full dataset is completed
Also if I can get a 3090 or something to train on that should improve batch size and stability

[nocturne olive]

Part of it could be that my tuning is still not amazing

[scarlet arch]

Like those images from when image generation was new. You could tell that something is supposed to be in it, but you can't make it out ya know?

[scarlet arch]

Oh yeah no critic. Love the stuff you're doing. Can't wait for it to surprise even Vedal :D

[nocturne olive]

The pitch curve is manual with vocal synthesizers, and I don't know all the tricks yet

Still learning

[trim valve]

imo she sounds a bit more stressed than in the original or something

[scarlet arch]

Yes that one

[nocturne olive]

Well, we're aiming for this thing to reach Neuro RVC V11 quality eventually, so it's gonna be good when it finally releases to 1.0

Currently its quality is probably somewhere between V10 and V11

[trim valve]

lmao completely unrelated but you reminded me of the shenanegans jetbrains are doing

[scarlet arch]

Oh, also, hi bred

[trim valve]

it went from 2025.1 -> 2025.1.1 -> 2025.1.1.1

[nocturne olive]

With mentioning the RVC versioning?

[stark needle]

Cause tts is essentially a completely different tech lmao also vedal has very tight latency requirements

[scarlet arch]

Wonder what their reason for that was

[trim valve]

yeah it just reminded me that versions existed lol

[nocturne olive]

RVC can do both speech and singing, could probably distil trough that

[nocturne olive]

How silly

[scarlet arch]

Hahaha
https://cdn.discordapp.com/attachments/1067638175478071307/1372277191894761654/image.png?ex=6826303a&is=6824deba&hm=36f6344c9e6c521757fe4f90e9df483eef4e60f8361a229564c164c17623458d&

[stark needle]

i love the ____ in this image

[scarlet arch]

[stark needle]

Yes but it's not so easy

lol

Speaking tone

[trim valve]

soundboard is c#

[stark needle]

Is different from a sequence of notes

[tender river]

where did you find a photo of my room

[nocturne olive]

I would guess so, but there's probably a way to make it work

[tender river]

damn i even sold my valve index

[stark needle]

the index in your valve

Also chat

Anyone know how

[amber fractal]

no

[stark needle]

To turn off pings in a channel

[tender river]

right click

[stark needle]

I keep getting the daily #announcements ping

[tender river]

notification settings -> none

[nocturne olive]

Mentions -> none

[tender river]

or just disable everyone

[stark needle]

I already turned this off

It literally ignores

[nocturne olive]

Whuh

[tender river]

go to server notification settings and disable everyone

[nocturne olive]

Discord moment

[amber fractal]

It'd be suppress @ everyone @ here

That is in server settings

[stark needle]

i dont even rly watch neuro anymore 😭

like

I even hid the channel completely

And

It just reappears

[nocturne olive]

I call that a skill issue on part of the Discord devs

[amber fractal]

When does discord not have a skill issue?

[nocturne olive]

Actually I'm not sure

[trim valve]

real neurocord

[scarlet arch]

[trim valve]

slander of my goat light mode 💔

[drowsy ocean]

I took a screenshot and all of a sudden my app won't load

[stark needle]

Aisling

Ash

[ruby timber]

Shad

[stark needle]

Stella.sh

[ruby timber]

...

[stark needle]

light mode

[stark needle]

[ruby timber]

Stella.sh: Permission denied

[amber fractal]

I should switch to light mode mhm

[ruby timber]

Get lighted

I could never use dark mode with a background like this

[drowsy ocean]

let there be light

[ruby timber]

no chmod...

[maiden geyser]

"kanava" :D

[ruby timber]

Actually chown might be the better way

[nocturne olive]

Finnish for "channel"

[stark needle]

shad

[amber fractal]

Not as bad as I thought, the contrast of this and my keyboard is something

[ruby timber]

shad OW that hurt

advanced humor

you won't get it

posture check

[trim valve]

to be fair I am only a lightmode enjoyer when its bright

[stark needle]

Hello koniifer™®©

[trim valve]

I should really convert my remote desktop setup to use a virtual display

[amber fractal]

Now we are talking

[stark needle]

spidey

[trim valve]

that's certainly a keyboard

[scarlet arch]

Wasn't slander. I was just blinded. My discord is in the OLED theme and the sudden brightness actually was bright haha

[trim valve]

fair enough lol

[ruby timber]

Goated keyboard

[tender river]

are you really a programmer if your eyes are healthy enough to to look at light theme

[stark needle]

konii

explain

[final tinsel]

briish

[ruby timber]

! I had this when launching discord earlier

[stark needle]

stop doing that

[trim valve]

ngl I end up straining my eyes if I use dark themes on this display

[ruby timber]

failed loading english gameboy

[scarlet arch]

I have this in my face for most of the day. You'd think I'd be used to brightness

[stark needle]

it's one of the places i would consider potentially moving to some day

[scarlet arch]

Also ignore the grass growing like that. Similar to my haircut I'm too lazy to cut it

[stark needle]

bro has a regular grass plantation

[amber fractal]

Transparent doesn't overlay

[stark needle]

if i get accepted into deepmind i will move to the UK

gotta live with that

in 999 years

[final tinsel]

Brexit 2

[stark needle]

whyy

[amber fractal]

o7 konii

[trim valve]

can we just get rid of england and leave the rest

[final tinsel]

or just unify ireland

SCOTLAND FOREVE-

[amber fractal]

My ears hurt and I'm not listening to music

Maybe we can learn a key weakness from the OST

(No we can't)

[trim valve]

proof?

[amber fractal]

Taking out those kneecaps

[trim valve]

understood 🫡

[tender river]

i think all of my uk friends would agree with this approach

[trim valve]

can I also agree

with myself

[amber fractal]

I thought it would have been -30%, guess it was nerfed

[trim valve]

is this incoming out outgoing

[scarlet arch]

Message received. I am quickly approaching your location.

[stark needle]

i will hammer you to your demise (in minecraft)

[tender river]

i wouldn't fight you you'd squish me like a bug

[stark needle]

[scarlet arch]

Close and reopen discord and suddenly they're in sync :D

[stark needle]

konii death squish

[amber fractal]

Gotta use this boi for stacking damage

[stark needle]

uranium on a stick

🔨

[dry charm]

i hated the ex compressum era of skyblock modpacks

it was a cool mod, but it was in EVERY modpack

[olive sable]

i am alive!

[trim valve]

how big is the konii radius

[olive sable]

tf you mean your radius?
im nowhere near englasnd

[amber fractal]

I didn't mind ex compressum, I do wish Ex Nihilo was used less often.

[olive sable]

300-400km i guess?

[tender river]

tangentially related (russian irc quote)

- a bundle of crowbars typically sinks
- but it floats in mercury just fine
- but if the crowbars are made of uranium they will float in mercury just fine
- float uranium crowbars in mercury by yourself

[olive sable]

domain expansion

[amber fractal]

I'm not in the radius surely

[olive sable]

i dont think so

you're on the opposite side of the planet

[scarlet arch]

What happens in it? I hope it's hugs

[trim valve]

assuming konii is a londoner I'm near the border

[olive sable]

if konii is in london her domain expansion would be the london lesbian domain

[trim valve]

[olive sable]

physics

[stark needle]

400KM

[olive sable]

im imagining a gate of babylon esque scene where she spawns in thousands of yuri images

[stark needle]

i am outside

lets goooo

[trim valve]

oh I am in that one

[olive sable]

[trim valve]

spooky

[olive sable]

i am def in that one

[trim valve]

o7 my time has come

[ruby timber]

paris can go, I don't mind

[stark needle]

yes french speaking people are L anyways

[ruby timber]

[stone cedar]

Parisn't

[trim valve]

konii you talk like a plague inc virus

[olive sable]

erm actually , that's unlimited blade works, not gate of babylon

i sure can

[scarlet arch]

Not even touching Germany. Guess no hugs for me

[olive sable]

yes you did

are you sure?

well im not paying attention so idk either

[trim valve]

can they invent a monster that doesn't have the side effects of being a liquid

[olive sable]

what?

[trim valve]

monster, the drink

[olive sable]

oohohhhh

[olive sable]

freeze it, problem solved

[amber fractal]

[trim valve]

just licking a solid chunk of monster ice

surely that's an acceptable library activity

[olive sable]

as one does

[scarlet arch]

This kind of makes me wanna freeze one of my energy drinks.

[olive sable]

i dont really drink energy drinks so

[scarlet arch]

Obviously wouldn't do it in the can for explosive reasons

[trim valve]

monster ice cubes

then you can put monster in your monster

[stark needle]

konii i hope you ________

yes

[olive sable]

what?

[stark needle]

too low iq person to understand our means of communication

now you make me sound schizo

[olive sable]

i would have you know that my iq is ...

[bleak steppe]

Ok I'm curious what is the app tag on discord mean I know neuro has it because of the tweets vedal posts of their messages but I'm wondering how it compares to the bot tag

[tender river]

@round kraken

[stark needle]

neurt

[tender river]

thats one

theres the other one too

uh

i forgor

no

[trim valve]

isn't app just the new name for bots

I think it changed a while ago

[stark needle]

yea

[stiff mica]

🍪 Om nom nom
You've given me 28 cookies! | I've received 203605 cookies total!

[tender river]

@old pendant this is the other one

[bleak steppe]

Ah ok I thought it was a new type

Wait just curious what happens if someone messages neuro

[tender river]

the neuro twins (neuro and neuro)

[tender river]

you explode (try it)

[bleak steppe]

Nothing happened

[trim valve]

nothing ever happens

[bleak steppe]

It's probably a thing in vedal's code were she can only message friends

[olive sable]

understandable

[prime ridge]

@olive sable Fixing code rn. Once done should be able to have a batch size of around 200

[olive sable]

[bleak steppe]

If that's how it worked all someone would need to do is spam her to make them talk to her

[nocturne olive]

I feel like at 512M params 1024 batch size on 24GB VRAM that's like impossible or something

[prime ridge]

not 1024 batch size lol

Also it isnt. Whatchu know bout MLA and Adafactor running on FP16?

I can boost the ctx a LOT too

[nocturne olive]

I meant CTX, just confused the numbers

[prime ridge]

without much memory usage increase

[bleak steppe]

Ah ok I'm not really that experienced with code I more of appreciate the effort that goes into it

[prime ridge]

1024 ctx uses 4mb of vram

[nocturne olive]

Not that big

[prime ridge]

well in forward pass. More realistically around 10m with overhead

[nocturne olive]

The model itself at FP16 is 1GB

[prime ridge]

yes

[nocturne olive]

Then who knows how much the gradients take

[prime ridge]

2.6 gb

Adamw is very expensive with a 2x overhead

Adafactor is 1.3x factor

so more vram savings

[nocturne olive]

I guess we'll see how it goes

[hoary lion]

hi

what were we talking about

[scarlet arch]

[prime ridge]

MLA

https://tenor.com/view/its-peak-old-man-shaking-head-gif-3938163323164042530

Saving so much memory rn

[hoary lion]

where did you get it from

send the git repo 💯

[prime ridge]

It's not done yet just working on it

theoretical max batch size is around 150-200

no need to run higher than 128 tho

[nocturne olive]

Theoreticals are never hit

[hoary lion]

lel

[prime ridge]

That's why I make pessimistic assumptions

[hoary lion]

gotta count that random ass vram occupy from os

[prime ridge]

if lower end is around 150 then 128 seems reasonable

[honest vapor]

my school project is complete yay

now I need to create unit - integration tests otherwise I will get less score

[nocturne olive]

With 8x gradient accumulation that's like 1024 effective

[prime ridge]

oh well yeah I don't want as high of a grad accumlation

makes it run slow

[nocturne olive]

Low compared to commercial LLMs, but reasonable for a home-made model

[prime ridge]

but remember it's a 512m model

[nocturne olive]

It's just a factor of 8 on the runtime

[prime ridge]

so a batch that high isnt really that necessary

[hoary lion]

i need to check my previous analysis of mla

forgot the boost

[prime ridge]

MLA is an insane boost

[nocturne olive]

I ran NeuroSynth at 2x gradient accumulation for BETA-2, it took consistently linearly 2x the amount of time

[prime ridge]

Should be O(L*N) I think?

[hoary lion]

i dont think it was about modifying the complexity

[prime ridge]

in memory not time

[nocturne olive]

For the next one I may try 4x

[hoary lion]

probably

[prime ridge]

Grad accumulation is only needed really if you don't have enough vram

[nocturne olive]

Would go from base 3 real batch size to new 12 effective batch size

[prime ridge]

24gb is more than enough if you use latent attention

[nocturne olive]

Well, I have 12 GB

And NeuroSynth is not a transformer model

NeuroSynth is exceptionally memory-heavy to train

[hoary lion]

latent attention doesn't help with memory

unless you regard previous ones

and dispose them

[nocturne olive]

I really kinda want a 3090 for more VRAM for more batch size for better NeuroSynth
Because 12GB having 3 real batch size makes the util suck

[hoary lion]

2 x l x b x (n_h x d_h x s) seems to be the calculation for traditional transformer's memory complexity

[quaint venture]

yo how to i remove stuff from a dictionary?

[hoary lion]

where in MLA n_h and d_h pretty much gone and reduced to small c

[prime ridge]

it's quadratic

[quaint venture]

in python

[trim valve]

del dict[key] isn't it

or dict.pop(key)

[trim valve]

I still hate this option so much

[quaint venture]

ill try this

[hoary lion]

wait is it?

[quaint venture]

so basically GCSE computer science programming task

ill explain it in one message

[prime ridge]

traditional is

[hoary lion]

no like that's time complexity

memory is still linear

[nocturne olive]

It's crazy how NeuroSynth at 100M total trainable params is so intensive to train

[prime ridge]

memory isnt linear

[hoary lion]

but the coefficients are large af

[prime ridge]

memory is quadratic

[hoary lion]

proof please

[prime ridge]

it's a square

[nocturne olive]

I assume it's normal though, it's not a tranformer model

[hoary lion]

...

[prime ridge]

x is ctx

so x^2

[quaint venture]

i have to write a program that asks the user to list 4 vegtables then add them to a dictionary starting with index 1. then i need to print the dictionary then ask the user for one veg to be removed then sort and re print the dictionary

[hoary lion]

i think that is just attention scores?

[quaint venture]

how do i do the code thingy bc i wanna put down what i have

[prime ridge]

which is stored in memory 😂

[hoary lion]

🤦

[nocturne olive]

Not sure how exactly to remove something from a dictionary in Python, try assigning the variable type to explicitly be dictionary and see what options for methods the IDE gives you

[hoary lion]

shouldn't we store each key and value's computed results per layer

so each token takes up 2ndl

where n is head size, d is head dim, l is layer amount

[trim valve]

use ```

[quaint venture]

Ok??? it just that i dont know how to set it to be explicitly a dictionary apart from the {}

'''py

[prime ridge]

not only does it need to be reproduced in each layer but it also is still ctx^2

[quaint venture]

oof

[prime ridge]

so it scales linearly and quadratically

[trim valve]

` not '

[quaint venture]

ohh

[hoary lion]

yeah in that context it is

[prime ridge]

ctx^2 * 2 *n *d * L I think

[hoary lion]

but MLA does not optimize it

[prime ridge]

either way it's a lot

[hoary lion]

and it is not a multiplication

[prime ridge]

the cache

[hoary lion]

addition

[prime ridge]

Key value cache

[nocturne olive]

Usually you do variable: Type where Type is the class name for the type

[prime ridge]

instead of spamming more caches

you compress it

into latent space

[nocturne olive]

Something like variable: Dictionary = {}, though not sure if the class name is exactly that in Python

[hoary lion]

i know

you multiplied

[prime ridge]

that's still a lot of memory saved

[hoary lion]

ctx^2 + 2 *n *d * L

yes

[quaint venture]

a = input("whats a good vegtable? ")
b = input("whats a good vegtable? ")
c = input("whats a good vegtable? ")
d = input("whats a good vegtable? ")
dictionary = {1: a, 2: b, 3: c, 4: d}
print(dictionary)
del dic(input("which veg would you like to remove? "))
dictionary.sort
print(dictionary)

[hoary lion]

bro I did a report on MLA

[prime ridge]

I mean I skimed through it. I trust you know more on it

I just know it will save a lot of memory in the caches

but idk if I still need a large amount of memory to fit the initial attention scores

cuz the very first attention computation

[hoary lion]

TTFT?

[prime ridge]

yes

like specifically in memory

cuz no caches

[hoary lion]

the prefill stage is compute bound

[prime ridge]

no like literally it might not fit on the gpu

idc about the speed

it's only once

[trim valve]

a few notes:

• its del dictionary[key] not del dictionary(key)
• the result of input("which veg<...>") will be a string and your keys are integers, you probably need to convert
• I doubt dictionary.sort exists, and even if it does you'd need to call the method (so dictionary.sort())

[hoary lion]

"Specifically for causal self-attention mechanism, during training and prefilling phases,
batched matrix multiplications and attention computations exhibit high arithmetic intensity"

and if it doesn't fit, you won't be able to train at all

typically in research field we don't really account vram on training but this is kinda bad

which means

i think 24gb 3090 is cooked

[prime ridge]

It shoulden't be

[quaint venture]

figured out the first and second point, but the last point is because its asking me to sort the dictionary so im kinda bugging there

[prime ridge]

that is 48x the model size

[trim valve]

sorted in what way exactly

[prime ridge]

48x overhead is insane

[nocturne olive]

24x at FP16

[quaint venture]

not a fucking clue the textbook says sort it then print so im guessing that it wants to be re-indexed to like 1,2,3 instead of i.e. 1,3,4 (if i removed 2)

[nocturne olive]

And remember that Windows has to exist as well

[prime ridge]

that's still an astronomical amount

especially since I have a small ctx

[nocturne olive]

It takes anywhere from 0.5-3GB

[trim valve]

ok fair enough, in which case I only have a cursed suggestion

[quaint venture]

gulp

its meant to be basic python too 😭

[trim valve]

{i: v for i, (k, v) in enumerate(dictionary.items())}

[prime ridge]

mine takes like 200mb bruh

[trim valve]

I think that works

[nocturne olive]

Sometimes it's lower, sometimes it's higher, it's very dependent on the specific device

[prime ridge]

no yeah ik

[trim valve]

theoretically speaking that is based on the assumption that your dictionary gets iterated on in the right order but because its based on insertion order it should be fine

[prime ridge]

I highly doubt that i'd get OOM

[nocturne olive]

My dwm.exe idle usage is somewhere around 2-2.5GB

[quaint venture]

can i shove that in as it is or do i have to change the variables?

[trim valve]

>>> dictionary = {1: "a", 2: "b", 3: "c"}
>>> del dictionary[2]
>>> {i: v for i, (k, v) in enumerate(dictionary.items())}
{0: 'a', 1: 'c'}

so close

[trim valve]

yeah it should just work, you just like print({k: v...})

though you'll need to make the key one bigger

[prime ridge]

oh yeah we good

[trim valve]

looks like you can do enumerate(..., start=1)

[prime ridge]

MLA literally doesn't allocate the full n^2 attention matrix @hoary lion

[trim valve]

fwiw if you want me to explain that instead of just an abstract "here's some stupid code" I probably can

idk if they teach list/dictionary comprehensions to you

[prime ridge]

so yeah we fully balling

[quaint venture]

ok im going to get pizza w/ my dad so ill prolly ask in a bit

[trim valve]

[hoary lion]

erm

it should??

it should keep the score matrix?

[prime ridge]

Literally doesn't

[hoary lion]

proof please

[nocturne olive]

I wonder what NeuroSynth's VRAM scaling is actually like, with base NeuroSynth at 100M total trainable params allowing for 3 batch size before 12GB runs completely out

Of note is that those params are trained in parts, that 3 batch size number is for the larger 75M params part

[prime ridge]

I mean just google it lol

https://www.google.com/search?client=firefox-b-1-d&q=does+latent+attention+allocate+the+fill+matrix

plus it mentioned that in the paper

because it's a latent representation

not the same attention matrix at all

it's not even noticably related

[nocturne olive]

I don't know, I guess who knows

[prime ridge]

Do you know what the arch of neurosynth is?

[nocturne olive]

Currently secret

[hoary lion]

he made it

lol

[prime ridge]

A transformer makes the most sense tho

[nocturne olive]

I didn't

[hoary lion]

you didnt??

[prime ridge]

vedal

[hoary lion]

[nocturne olive]

I know for certain it's not a tranformer

[hoary lion]

its over guys

im packing up

[nocturne olive]

Not Vedal, NeuroSynth is my vocal synthesizer model, built using an open source tech stack

[hoary lion]

i think you should just check out their paper

[nocturne olive]

NeuroSynth is very much not a transformer

[hoary lion]

https://arxiv.org/pdf/2405.04434

[prime ridge]

it's dxd

where d is the latent size

which is wayyyy compressed

[stark needle]

who dared pinging me

[hoary lion]

did someone?

[nocturne olive]

If NeuroSynth was made by Vedal I would not have access to it

[hoary lion]

[prime ridge]

oh lol

[nocturne olive]

NeuroSynth's tech stack is open-source stuff, while me and Wispers made the model itself

[nocturne olive]

And it is absolutely not transformers-based

[prime ridge]

It probably should be tho lol

Does it have a voice bank that it makes the pitches for or does it literally generate the waveform?

[nocturne olive]

I'm not 100% sure on how it works

[hoary lion]

im pretty sure it is not transformer

[nocturne olive]

Why should it be? I see no reason for it to be a tranformer

[prime ridge]

If it generates pitches then it's a seq2seq model which transformers are great at

well actually maybe not

[hoary lion]

yep more close to be a UTAU ish stuff

[prime ridge]

cuz previous context doesn't matter rlly

so a simple RNN could work

Yeah RNN is prolly the move

[nocturne olive]

UTAU is fun to mess with too

[hoary lion]

dh and nh, l is all fixed number

so kv cache increases linearly

with heck ton of coefficients

MLA removes the 2, and compresses dh times nh into much smaller one c

[nocturne olive]

I wonder what different basic model architectures would work well as a vocal synthesizer model

[stark needle]

kv cache my beloved

[hoary lion]

diffusion

[nocturne olive]

True

[stark needle]

already diffusion

[hoary lion]

oh yeah it was

most audio these days that are not relying on transformer + vocoder

[prime ridge]

good enough. You made it seem like it was using the original matrix which it isnt

probably a misunderstanding on my bad

but yes the c^2 matrix

[hoary lion]

c squared matrix is probably something in between the process idk

[nocturne olive]

I wonder what SynthV AI uses for its architecture

[hoary lion]

let's break down the binary

[prime ridge]

did they even use NNs?

thought simple voice bank with goofy ahh audio editing tricks

I can't imagine using a NN on low spec machines anyways

[stark needle]

they do

[nocturne olive]

It's definitely some kind of AI-based vocal synthesizer

[stark needle]

fancy voice cloning

[hoary lion]

SynthV was voice cloning??

or is it just baked in

[prime ridge]

That's probably better anyways but damn

how tf do they run a NN on the machines

[stark needle]

how else u think it can do teto in spanish or whatever

[hoary lion]

idk never had synthv or any vocaloid/utaus in general

too adhd to do audio related stuffs

[nocturne olive]

Silly

I've been having a good time messing around with RVC and NeuroSynth

[prime ridge]

@hoary lion Do u wanna write MHLA implementation?

I still gotta push to git and I can add u on it if u want

[nocturne olive]

https://www.youtube.com/watch?v=3jWOyIcKz98
Especially this turned out really great

[prime ridge]

Neurows XP 💀

[hoary lion]

i can review

[prime ridge]

I'm probably going to end up fucking it up

[nocturne olive]

Soon to be the most viewed fanmade Neuro MV

[prime ridge]

W thinkpad btw

[nocturne olive]

Waya did most all of the filming

I did the cover

And the art is by like 40 different artists

In total somewhere around 50 people worked on it across a time of 3 months

[hoary lion]

can't hear it yet but it already looks amazing

[rigid snow]

smh haven't heard machine love yet

[prime ridge]

Very good

[nocturne olive]

That's crazy
We gotta make sure every swarmer sees it at least once

[hoary lion]

machine love

[prime ridge]

ngl the transitions are crazy

[hoary lion]

undominatable english teto song

[prime ridge]

so many touhou references lol

feels like everyone here is also a touhou fan

[tender river]

i met one of my friends in uni because we both liked akatsuki records (among other things)

[prime ridge]

wow

Shibayan records on top

I love shibayan's style so much

akatsuki is really good too

[trim valve]

💔 store only had white bread

[stark needle]

bread

[prime ridge]

wheat bread >>>>

[stark needle]

white bread with molten butter >>

[scarlet arch]

Nooooo. Muti-corn my beloved. Or whatever it's called

[prime ridge]

Wheat bread with melted butter with sea salt, garlic, and rosemary >>>>>>>>>>>>>

a littlr thyme too

and olive oil

[trim valve]

I aint doing allat

[tender river]

too much effort for bread

[prime ridge]

https://tenor.com/view/walter-white-falling-fast-gif-18043850

ok actually time to lock in

[hoary lion]

obesity speedrun

[tender river]

im too lazy to get fat

[uneven pulsar]

Someone exsplain MoE like I am 5

[stark needle]

moe as in mixture of experts?

[uneven pulsar]

It doesn't make sence

[uneven pulsar]

Yea

[stark needle]

what part doesnt make sense

[uneven pulsar]

I just don't understand how it does what it does

[stark needle]

well it's a feedforward network that can choose which experts to activate depending on the input

via topk routing

[uneven pulsar]

This just sounds like magic

[hoary lion]

so a gate function sees the first layer's output

[stark needle]

it's the softmax activation

[hoary lion]

and says 'this ffn seems good for processing this'

[rare bramble]

i think it just divides the parametes between experts and selects on them on the fly, for example 36 billion parameter model with 6 expers has 6 billion parametes exclusively for each expert. and this is apparently more efficient than using all 36 billion parameters to learn about each subject

[stark needle]

which for the input generates a probability distribution

[hoary lion]

and routes it via softmax

top k means k amount of most 'good' experts would be routed, total being 1

[uneven pulsar]

So it uses fancy math to guess what network is best for the current input?

[hoary lion]

not fancy

but basic

[stark needle]

basically yes

but no it's not fancy math

[hoary lion]

also each token does it

so per token, it gets routed into top k best experts

which differs every token

[uneven pulsar]

I think I can implement that

It doesn't seem too complex

[stark needle]

the batch matrix multiply will be not so straightforward

and the fact that

token dropping etc

[uneven pulsar]

Can pytorch Handel that magic for me?

[stark needle]

yes if you know what ur doing

[prime ridge]

Normal Dense models do this

Input -> Attention -> Single FFN -> Output

Moe does

Input -> Attention -> Router ("Hmm what expert is best fit for this") -> Couple of experts -> Combine the results into one

[stark needle]

https://arxiv.org/abs/2101.03961

this is for transformers specifically

if ur not doing transformers then

the principle is still the same

[prime ridge]

Massive oversimplification but should be good enough for "explain to me like im 5"

[hoary lion]

i forgot what was token dropping

[uneven pulsar]

I am doing transformer so it should be good

[uneven pulsar]

Seems simple enough

[hoary lion]

tbh i should do MoE from ground up too

[prime ridge]

Also 😵‍💫 how tf do I even implement MHLA. Barely any github repos out there with examples

[verbal grove]

erm guys. i might have a silly question

[prime ridge]

Especially since I am using xtransformers since I didn't feel like manually building out the transformer

and they don't support it

[prime ridge]

not too hard yeah u got it

[hoary lion]

search it with MLA not MHLA

https://github.com/ambisinister/mla-experiments

this is a solid start

[uneven pulsar]

Let's hope my Ai never realises it can CD into its own source code (my code is embarrassing)

[verbal grove]

you know thats something im hoping will happen someday honestly

atleast for my own Ai

because im genuently curious what would happen

[grizzled viper]

Probably try to refactor and break itself

[uneven pulsar]

My Ai has emotions and a pirated personality so it'd probably laugh at me then become skynet

[verbal grove]

A

well that probably is correct but that's why im running it on a distrobox as an extra layer so that it only has read only access to its own code and can ask me to add things

[nocturne olive]

How silly

[verbal grove]

;w;

better than giving it root access on its own system so that i dont have to refractor a few files

[nocturne olive]

If it's an LLM, it's still not that interesting

[uneven pulsar]

Should i remove my Ais write perms or just give it a rollback function (so it can't birck itself)

[verbal grove]

depends on how much brain damadge you wanna have from your own ai's stupidity honestly

if its running on your main machine definetly remove write perms tho 👍

you only make that mistake once and then spend 4 hours trying to unbrick a linux kernel while you only have a basic grub shell

[uneven pulsar]

It isn't my PC (I'm in the process of getting one) so it should be fine

[verbal grove]

then add rollback probably

[uneven pulsar]

Or I just give it write perms for the SD card it lives on

[uneven pulsar]

Yay I can brick my brother PC :3

[verbal grove]

->

ah, you should be careful about what you say honestly. since carma is not exactly nice to people

[uneven pulsar]

Who?

[vestal prism]

Is that time of the year again

When i try to learn vim

and vim kick my ass

[verbal grove]

hold up a moment il get to that in a second

[uneven pulsar]

OK

[verbal grove]

il pray for you 🙏

because i use neovim with a custom config :3

[vestal prism]

yea neovim whaever

[rigid snow]

learn emacs instead

learn ed instead

just echo "line" >> file.txt instead

[verbal grove]

you know he has a point

[vestal prism]

if i say i use the vim extension plugin for vscode you will all will say eww or some shit so i will not say it

[verbal grove]

->

ive done that before

its not that bad once you get used to it

actually @vestal prism do you know how to write in bash?

[vestal prism]

i use linux only for uni things it doesnt make a difference for me than windows

[verbal grove]

aww ok

[vestal prism]

its hard like geting to use all those things at once

at the end you waste more thing setting up things than doing things

and i cant afford losing that much time on some things

lately for example linux mint give me weird errors in all screens that aren the first one so its pretty difficult idk

[verbal grove]

can i get that error rq

[trim valve]

what would it use write perms for

[vestal prism]

lol

[opaque sigil]

pragmatic and based

[uneven pulsar]

To alter the bad source code

[trim valve]

understandable I guess

idk if I'd trust me to do brain surgery on myself though

[uneven pulsar]

If it crashes then it just automatically goes back to the last stable version

[vestal prism]

nah its not worth of your time, i think i will wipe it up and reinstall it or try popOS that works fine in my notebook

[trim valve]

my only concern with that is what happens if it bricks itself in a way where it doesn't crash but also doesn't work

[uneven pulsar]

Uhhhhhhhhh

What If i give it a separate environment to test new code before it murges

[verbal grove]

oki

[grizzled viper]

An AI playground would work

[trim valve]

that sounds like a better idea

[uneven pulsar]

I could probably write a benchmark script so it knowns when its giving itself brain damage

[trim valve]

mhm

idk the specifics of how that'd work but worth considering as an idea

[final tinsel]

…I think i am the ultimate nerd. I just sat down on the toilet and started scrolling LinkedIn. sigh

[verbal grove]

linkedin?

try github xd

[final tinsel]

…now I want to make a program that makes a “main feed” for GitHub like other social media sites

[verbal grove]

hm

interesting idea

[trim valve]

github already has a feed lol

[verbal grove]

true

but its not a nice feed

[final tinsel]

I gtg, if y’all make this and get rich I want 1% of the profit

[verbal grove]

👎

sorry man general public license is a must when i code

but oh well

[uneven pulsar]

Nerd

Make everything proprietary like me

[stark needle]

Based

[trim valve]

just don't make anything ezpz

[uneven pulsar]

(I'm using my code to make money I need to live so its kinda justified)

[tender river]

theres a better one at https://github.com/dashboard-feed

[stark needle]

I mean one can do something and paywall it and if the customer deems it a justifiable expense they will also pay for it

Otherwise just dont use

[real sierra]

hello programming

today I learned that trying to install python packages through pip for use on another platform is basically impossible

[trim valve]

😔

[real sierra]

even if you pass all the platform, implementation, abi, python version, etc requirements of your target platform

it will still refuse to download a package if it is not compatible with the system pip is running on

there is no option to disable this behavior

[tender river]

python packaging is horrible

like its the worst i've seen and i've seen C++ and Java

[stark needle]

Maven is not bad tbh

[real sierra]

idk if my use case is super uncommon or something but there's almost no information on what I'm trying to do and seemingly nobody has ever tried to do it

[trim valve]

what's the usecase btw? (just curious)

[real sierra]

i wrote some python on a Linux system and wish to run it on an offline Windows system

so I need to move compatible equivalents of the required packages over, and can't download them from the windows system

the simplest way to do this seemed to be pip download on the Linux system with options matching the windows environment

practice has not reflected theory

[stark needle]

[warped narwhal]

could you run cmd through wine and do it that way?

[trim valve]

I'd personally just give up and run a windows vm that's similar to the target machine

[tender river]

wine would definitely be easier

[verbal grove]

i would say run windows in a podman or docker container

[opaque sigil]

so what you're saying is that you need package X for windows but it won't let you install it on windows? am i dumb

[verbal grove]

if you dont need any custom configs and its a package issue thatl work

[opaque sigil]

oh wait offline

[verbal grove]

offline is harder...

[trim valve]

like to some degree I think python is in an impossible situation because stuff like build scripts will never fully work if you're "building" on a different platform to the target

[opaque sigil]

there's also mingw maybe that helps with compiling

[real sierra]

I mean all I'm trying to do is download wheels

I don't even need to install them

[prime ridge]

neovim mentioned?

[real sierra]

but pip is trying to hand hold too muxh

[prime ridge]

I've been asking for this 😩

[warped narwhal]

what commands are you running so I can have a look through the source

I'll see if it can be forced by skipping checks

[unkempt citrus]

this is why python has like 6 package managers now

[tender river]

and somehow every one of them sucks

ok i might give poetry a pass

[real sierra]

pip download -r requirements.txt --platform win_amd64 --implementation cp --abi abi3 --python-version 313 -d .

something to that effect, I'm going off memory

[unkempt citrus]

are you in a venv as well?

[real sierra]

yes

in particular, the package it's failing on does not have a platform-agnostic wheel available

but does have separate wheels for Linux and windows respectively

[unkempt citrus]

You could do a work around by making the command a script that needs a env variable

and have it choose the wheel that way

[olive sable]

im alive

back to debugging

[hoary lion]

Hi sam

[amber fractal]

I'm also alive

Back to the pits of hell

[real sierra]

it was so confusing to even figure out this was the issue because pip index versions <package name> <all the options above> would correctly list the package version required, but pip download with similar arguments would say the version wasn't found

[final tinsel]

mah PC is back online! lets goooo

[real sierra]

pip has to be one of the most crucial pieces of software in all of python development

[unkempt citrus]

Have you ever looked at the github homepage?

[real sierra]

how am I seemingly the first to hit this issue

[unkempt citrus]

I doubt you are but pip is such a mess that it never got resolved

[final tinsel]

i meant like how u can scroll reddit or instagram

but like

github

wait

nvm

found it

[real sierra]

a quick Google of "python pip ignore compatibility tags" brings up nothing relevant to my use case

like seriously nothing

mind boggling

[tender river]

easy solution MITM hijack whatever pip downloads so you keep the files even if it fails and then replay that on target machine

thats what i do for gradle

https://github.com/chayleaf/mitm-cache

[real sierra]

[unkempt citrus]

When people have a problem with pip, they don't fix pip, they just use or make something else so nothing ever gets done

¯_(ツ)_/¯

[real sierra]

oh well

[olive sable]

ive never had issues with pip ngl

[prime ridge]

You clearly never installed pyaudio 😂

[real sierra]

my workaround is probably going to be just manually downloading the troublesome packages with wget and removing them from the requirements.txt temporarily

[olive sable]

indeed

[real sierra]

repeat until all packages are acquired one way or another

[tender river]

no shade at all

[amber fractal]

All shade to pip

[raw trout]

Heyyo!
I haven't checked in for attendance today

[real sierra]

pip is great but I don't understand how this is even a bug

if I specify in the platform option that I want windows

why would it check the platform of the wheel against the pip compatibility tag

user supplied options should take priority over environment inferred ones

I don't think that's even remotely a hot take

[rigid snow]

pip is great

[real sierra]

even if they understandably think all python users have low density heads, they could at least make it an option to ignore the compatibility tags

I lost 3 hours to this dumb issue and lack of useful warning message

and the only solution I have is dont use pip

[final tinsel]

pip uninstall pip

also with my new cpu i can run my llamafile without going 100% usage

[sour lotus]

When you build something as a protoype and then dont want to share it 😅
But here goes, open to feedback and ideas if you dont mind Mods, im not trying to be promotional, just sharing something im prototyping but 👍 if you dont

🛠️ [WIP Prototype] SoulSkin – Companion/Avatar/Fun OverLayer for ChatGPT (Chrome Extension)
• I’m hacking together a lightweight Chrome overlay that visually reacts to ChatGPT conversations in real-time (facial expressions, ambient effects[WIP], etc.).
• Tech stack: DOM observer, regex-based keyword parsing, CSS/Canvas overlays.
• No server-side processing — all runs locally in the browser.
• It’s super early (V0.1, bugs included)

#hides

[final tinsel]

oooo

not personally my cup of tea (to use) but very impressive

[sour lotus]

11 Emotional states which pick up regex keywords
Thanks! 😊

Ignore the AI slop 😅

[final tinsel]

lol

[sour lotus]

Id love to get it funded or something and get real artist works in there

[final tinsel]

how do i manage to accidentally send stuff so often here (this time i noticed fast enough to remove lul)

[cosmic sphinx]

download something on python 3.9
it works
proceed to work on it
eventually reaches a project where I need a package that requires 3.11+
have to install 3.11 from ground up
have to reinstall EVERY package i've had so far but for 3.11
end up with two pythons and duplicates of packages filling up my storage

average python experience

oh and dont get me started on the PATH conflicts..

[tender river]

thats why pip on linux tells you to prefer distro package manager (i think) (at least you should prefer it anyway)

[tender river]

wouldnt venv solve any issues with that

[final tinsel]

now try this but its 3.9, 3.10, 3.11, and 3.13. Also needs to work on/have a variation for mac, linux, and windows.

[cosmic sphinx]

im too much of an amateur to use that

[tender river]

also obligatory "just use nix and you will never have path conflicts"

[tender river]

its really not hard

especially if you use direnv

[final tinsel]

EW

i just looked at a page's html

AND ITS ALL ON ONE LINE

[tender river]

thats normal though

theres no point in adding spaces

since its not for humans to edit

but for machines to read

and its easier for machines to read html without random spaces that mean nothing

[rigid snow]

you heard it here first guys, no docker = bare metal

[final tinsel]

._.

[grizzled viper]

I like this idea, it's a interesting take

[sour lotus]

Thanks 🙂 Definitely a market for it

[amber fractal]

[final tinsel]

always fun to use chatgpt to help me make anti-hallucination prompts

and have it work

[amber fractal]

Ah yes, hallucination to not hallucinate

[final tinsel]

if it aint broke...

�

dang

discord doesnt know what it is either

‘

ah

U+2018

[olive sable]

Gn

[amber fractal]

Goodnight Sam

[final tinsel]

this bed is comfortable

ill staaay

[prime ridge]

todayyyy

[final tinsel]

this waaaay

[prime ridge]

i'llll stayyyy

[final tinsel]

todaaaaaaaaay

[uneven pulsar]

[final tinsel]

musical interlude

[prime ridge]

I just fixed the code btw. I'll send it to you can you can train the morning

it's really really good now

[final tinsel]

u guys are workin on like ur own llms and im just struggling to import an rss feed lol

well

[prime ridge]

rss feed is dope

[final tinsel]

2 rss feeds without overwriting but yk

sadly i cant just use + or ,

so imma have to actually write code

aur naur

lol

[uneven pulsar]

Meanwhile I'm working with Zend mail Framework that's no longer supported

[prime ridge]

It's crazy that it can train a 128m model on my pc with a batch size of 16

couldne't even do a batch size of 1 without grad accumulation on a 30m model before

[uneven pulsar]

Legacy

[prime ridge]

never even heard of this 😁

1.4gb weights file 😱

[uneven pulsar]

Pain

[final tinsel]

how bout dis tho?

[prime ridge]

that's calm

4.2 gb is chill

[prime ridge]

smol

[final tinsel]

steals your ggufs

[prime ridge]

All of everything will be open source

[final tinsel]

legally downloads your ggufs

[prime ridge]

using h5 btw 😎

[final tinsel]

frantically googling what that means to make witty response

oh cool

[prime ridge]

https://tenor.com/view/its-rubbish-its-peam-rubbish-its-peak-gif-5313681084982718404

I am actually so insanely happy that it's somehow training

MHLA is AMAZING

THank god for deepseek bro

publishing that paper was clutch

I eventually figured it out

[final tinsel]

ok wow this is cool

part of me wants to like do the whole make it from scratch cuz like this is cool, but at the same time i know if i went for that id prob get frustrated and give up

OH THATS XML (not made as rss)

IM DUMB

hmmmm that doesnt seem right

[prime ridge]

Doesn't RSS send XML?

I honestly forget

I barely touched RSS

[final tinsel]

i think RSS is made of XML

from my (very limited) understanding

[prime ridge]

I bet Vedal is in VR chat rn 🤩

[final tinsel]

@vedal turn on your activity sharing on discord

no balls

[prime ridge]

He's too scared to do something that bold

[final tinsel]

errr google not sure abt that but ok

[prime ridge]

I mean....

[grizzled viper]

It's a valid descriptor

[final tinsel]

i mean i guess

to be fair

he is basically printing money without needing to interact (unsupervised neuro streams)

[prime ridge]

Infinite money glitch

[grizzled viper]

Neuro is a infinite money sync though

[final tinsel]

i think sink would be more correct in that context

[desert wave]

[prime ridge]

Vedal barely pays a dime for Neuro I bet

[final tinsel]

XD

[prime ridge]

compared to revenue

[final tinsel]

*revenuero

[grizzled viper]

Woo puns

Very punny

[amber fractal]

I has been given object

[final tinsel]

camera?

[amber fractal]

Yeah camera

[final tinsel]

yay (i had a better response but i forgor)

[prime ridge]

Is that the arducam?

[amber fractal]

Box says BitEye

[prime ridge]

That makes sense

[desert wave]

[object Object]

[final tinsel]

newsgrabber.pynennnnnnnnewnewsgrabber.pynewsgrabber.py

oh

wait

[prime ridge]

No. We don't do that here

[final tinsel]

i... thought i was focused on vscode...

[amber fractal]

Java hate inbound

[final tinsel]

wait

why do i have nitro

when did i get nitro

how did i get nitro?

yk what im not gna question it

[amber fractal]

Emotes

[final tinsel]

here, take an inside joke

[grizzled viper]

[amber fractal]

looks at mutual servers
Noted

[final tinsel]

friendgroup server, sorry ehehehheheheh

[amber fractal]

I'm more surprised I found another Aliensrock fan

[final tinsel]

ngl havent watched in like 2 years