programming

[amber fractal]

I can do that yeah

[unkempt citrus]

https://rxresu.me/ I like this for realitively straight forward resume building at lest

Yeah its crap but I do at least care about some of the work some of the places I'm applying for, renewable energy, medical devices, financial regulation speak to me at least

[amber fractal]

Also the fact I got over half of the members

[olive sable]

true

[unkempt citrus]

Networking

[olive sable]

employee management for a medium sized entertainment studio

[unkempt citrus]

I did have smoene reach out to me on linked in but they wanted a Senior DBA which firs tof all no interest, and also I'm barely more than a juniior DBA

[nocturne olive]

ExtraordinarilySoVerySilly

[amber fractal]

I did legit play that role, not wrong at all

[unkempt citrus]

If you want to spin more crap you can talk about your time on discord as "community engagement" and "tutoring in LLM and AI usage"

[scarlet arch]

I get contacted by web3, NFT and fintech companies all the time because my focus is Rust and apparently they all use Rust lol. It's so annoying.

the money range is insane, but my morals tell me not to

[amber fractal]

I am noting all of these for later

[nocturne olive]

Meanwhile me: I managed converting a bit of audio to a different bit of audio

And a few other audio things

[olive sable]

i would 100% bag that money ngl. sorry not sorry i need a 2nd 3090

[unkempt citrus]

Pay scales here are like 70-90 for eng, 90-150 for mining work, 250k+ for quant trdaing

for entry level roles

[stark needle]

Game Jam - Slave artist

• Drew 18 full illustrations in 72 hours under pressure
• Tuned vocoder vocals with industry standard Dreamtronic software for the creation of a full original song
• Tuned machine learning models for realistic text to speech voice synthesis with Pytorch
• Achieved first place in the audio category among hundreds of other teams

[scarlet arch]

yeah I just can't. It would eat me up knowing that products I help develop are used to financially ruin others

[unkempt citrus]

Good on you then

Everyone has their price, I just hope mine is out of their range

[stark needle]

my price is anything above 7.7$/hr

[nocturne olive]

Next game jam though, I'm gonna be using a custom-trained vocal synthesis model to help create a new original song

That's gonna be fun

[stark needle]

Not linkedinified enough

Not hired

[unkempt citrus]

Isn't the singing built on vocaloid technology

[nocturne olive]

I have no clue what that means

[olive sable]

means more people go to the art slavery

[unkempt citrus]

Too humble, not buzzwordy enough not going to get hired

[stark needle]

Making it seem like the achievement of all time

[nocturne olive]

No it is not
Vocaloid is not even AI

[amber fractal]

We'll need to grab artists (that are not Shadow)

[nocturne olive]

Whar

[nocturne olive]

Well, certainly not me, I can't art

[olive sable]

yep, relieve some of the rpesure

[amber fractal]

I can't art well enough to main

I got a bit of time to practice but who knows if I will end up doing so

[olive sable]

im probably gonna be too busy doing 3D modeling and engine stuff

[amber fractal]

Next time we are not doing that many cutscenes

[nocturne olive]

I really hope we'll be able to get NeuroSynth up to par with Neuro RVC in time for the game jam, it'd be such a great showcase of the new tech

[stark needle]

"- Developed an instrumental piece of Software challenging state of the art singing voice synthesis tools in precision, utilizing novel machine learning algorithms to support an indie studio achieve nth place in a global game development hackathon with a storytelling-focused game"

this is how to linkedinify

[nocturne olive]

That seems way too scewed for my liking
I usually like to be as direct as possible

[amber fractal]

That is the point, need to shove in more buzzwords

[nocturne olive]

Seems a little silly

Speaking of silly

[amber fractal]

Problem identification
@olive sable Surely you know what this is referring to.

[olive sable]

what?

shader error code?

[nocturne olive]

NeuroSynth do be sillyin'

[amber fractal]

Optimization

[olive sable]

optemizig Problem identification?

[amber fractal]

Figuring out what to optimize requires knowing what needs optimizing, hence Problem Identification

[olive sable]

ah

we're still talking about bussines talk

i gtg, school ending

brb

[amber fractal]

Cya

I'm gonna sleep as I need to be awake soon

[nocturne olive]

Still BETA-2, the new dataset is still in development

[tender river]

new andor is out for anyone following

[scarlet arch]

my Jellyfin is already showing it to me... but alas I must work

[tender river]

i wonder if using an arm laptop would be a good idea

i think its like 50/50, steam and ida wont run but the rest should be a slightly better experience because all of my devices except my laptop run arm

[safe path]

arm chips are also generally more power efficient, it'll probably be a dream thin client

[unkempt citrus]

I've used an arm thin client

it was also a chromebook though and very undrpowred

[tender river]

I HATE AI SLOP

[rare bramble]

i dont think arm is ready yet for general use, so many quirks and random things not working properly

[unkempt citrus]

Aren't most phones using arm

[rare bramble]

only arm processors i would trust even slightly is the apple M series, but that's only if you want to be in the apple ecosystem

[rare bramble]

ye, but it's mature enough where everything on a phone is design around the arm processors, it's different for desktop applications

[unkempt citrus]

Isn't that more because of x86 dominance

if arm had greater market share you'd find more things working on it

[prime ridge]

Not using 256 😂😂😂

[unkempt citrus]

Early adopter cost and all that

[rare bramble]

ye, but i dont think they give a huge benefit over x86

at least currently

though AMD has talked about designing ARM chips if the market goes that way and it becomes more viable, they arent married to x86

[ruby timber]

The energy saving are huge aren't they?

The smaller instruction set makes things mostly simpler and more efficient

Is what I heard

And in the current world, that's definitely a big w

Though I do agree it still needs a lot of work compared to the behemoth that is x86

[olive sable]

aight im home

lemme see if the zip file finally finished unzipping

[unkempt citrus]

Oh hey theres a framework 12 now

2 in 1

[olive sable]

yeppers

[olive sable]

wtf is this data bro

you gave me almost a million files

it unzipped into a tokens folder inside the existing tokens folder, so give me 7 min to fix it

i can understand why it took so long now, each file is only 6kb so you cant fully utulize ssd speed and everything

[cosmic sphinx]

i stopped paying attention to any online articles besides web forums/social media discussions

its such a waste of time

[maiden geyser]

the rest should be a slightly better experience because all of my devices except my laptop run arm

it's like saying "all of my devices run linux except my pc so linux should be a better experience"

[maiden geyser]

ALL phones are using arm

it's either snapdragon or mediatek

[ruby timber]

Or Apple silicon stuff

[olive sable]

on my way to make a x86 phone jsut to spite you

[unkempt citrus]

I mean, theres also some RISC V stuff

but theyre almost non existent

[ruby timber]

True

[olive sable]

ps1 is risc v iirc

[rare bramble]

some use exynos

[ruby timber]

That's funny

[ruby timber]

samsung in europe for some reason.............

[maiden geyser]

isn't x86 kind of risc-esque inside

[olive sable]

nope, just risc, no v

[maiden geyser]

there are none

well, there are, but they cost like a Boeing wing

there's rpi, but 5 needs active cooling

[unkempt citrus]

theres the microsoft surfaces but those turned out to be more expensive than I was expecting

[prime ridge]

🥬

Cooked

I warned you 😂😂😂

Its a lot

[maiden geyser]

what's the point of a pc which can't heat my room

[olive sable]

its supposed to be 948.157 files ye?
cuz idk if the exrtraction crashed halfway while i was gone

no

use spacing

[maiden geyser]

948_157

python approved

[olive sable]

to think i though of you as "smart" yesterday.
you dissapoint me

i was just hoping yall qould know an amount of physical objects cant be a float

but ye we use , for the decimals here

noy .

but i have qwerty and program so i use . most of the time anyways

its really annoying for writing essays tho

[ruby timber]

948'157 :3

Easily accessible for qwerty

And undoubtedly european

I think

[olive sable]

[stone cedar]

Added benefit is that modern c++ allows this notation as well :D

[ruby timber]

Ooooooh that I didn't know, very cool

Surprising considering how much of a pain apostophes are to lexe and parse but that's cool

[olive sable]

nope its not, it should be over 2 mil

my windows does not like extracting these

time to install 7zip

[prime ridge]

Sounds close enough 😂

Actually lemme do the math brb

[olive sable]

im not gonna half-as sit

ill let 7zip extract it cuz i think the windows one keeps crashign halfway through

[prime ridge]

Ok its def more

Although I mighta not put the entire dataset

[tender river]

???

theres nothing fundamentally different

[nocturne olive]

Ever considered appending the files to make it more efficient?

[ruby timber]

Isn't the biggest change just the instruction set? I'm not familiar with differences between architectures

[prime ridge]

I simply just forgot lool

[nocturne olive]

2 million SMOL files is stupidly inefficient compared to just 1 single big file

[olive sable]

yep

[prime ridge]

Yes I know 😭

But thats how it works

[olive sable]

why did you even do it like this then?

[prime ridge]

Its not intentional

[prime ridge]

Well yeah which inherently means that the processors have wayyy less functions it can do

[ruby timber]

Just less abstraction, but you can still do just about anything

[tender river]

and it is a better experience, so i dont get your point

[prime ridge]

Like what

[tender river]

true thats why android supports x86

[ruby timber]

Oh it does? Neat

[olive sable]

android on threadripper when?

[ruby timber]

Be the change you want to see in the world

[prime ridge]

Cuz no way in hell ur counting the "Intel Atom"

[olive sable]

i need 16K RTX clash of clans

[tender river]

i think they are no longer being made

[prime ridge]

There is no reason to anyway

[tender river]

waydroid

[tender river]

yeah it would make sense for embedded stuff maybe but i dont think it makes much sense for retail phones

[prime ridge]

100 watt phone 😂

[tender river]

you'd be surprised how many IoT devices run android

[olive sable]

if its not using a kilwatt, then what even is the point?

[tender river]

or some of android userspace at least

my robot vacuum runs adbd

[prime ridge]

Probably wouldent

Compared to arm?

Yes ofc

[tender river]

100°C
you joke but my prev laptop ran that hot 24/7

[prime ridge]

Yeah same 💀

[tender river]

i once spilled a bit of tea under it and it boiled instantly

[prime ridge]

2007 thinkpad

Riscv??? Pfffft no. We roll our own isa

[tender river]

ppc is very nice too but kinda irrelevant nowadays

[prime ridge]

Yeah ofc

No arm is def cheaper

Well idrk

[tender river]

https://en.wikipedia.org/wiki/OpenPOWER_Foundation

to clarify, it simply means anyone can create a riscv chip with arbitrary extensions and without licensing fees, not that the chip design itself is open

i mean i guses theres https://openrisc.io/

theres also mips (and loongarch i guess)

[olive sable]

isnt mips just family to risc?

or a subset?

idk anymore

[tender river]

RISC != RISC-V

RISC means "reduced instruction set computer" (not so reduced anymore, its complicated), as opposed to "complex instruction set computer"

[olive sable]

i once wrote a whole essay about the ps1's proccesor, its been a while since then tho

ps1 is a mips

[tender river]

while RISC-V is just a RISC ISA

MIPS is a RISC ISA as well, or rather there are many different MIPS ISAs

the most exotic ISA that i ever disassembled code for was probably PPC, but i don't remember where i stumbled on that code

[olive sable]

aight, i got the full zip extracted now without crashing

3 milion

let the gpu torture begin i guess

[scarlet arch]

Why would extracting a zip crash

[olive sable]

@prime ridge i need a discord_llm_config.json

[scarlet arch]

Unless it's a zipbomb

[olive sable]

it kinda acted like one

[stark needle]

[sick owl]

https://huggingface.co/PrimeIntellect/INTELLECT-2

32B param model finetune trained with public access globally distributed reinforcement learning just dropped

With public training code

https://github.com/PrimeIntellect-ai/prime-rl

And a public RL dataset

[nocturne olive]

Have a fun couple months

[olive sable]

cant

dont have a file

i need a "discord_llm_config.json"

[nocturne olive]

Do you even have the patience to not use your PC for anything except training and maybe light browsing for potentially a few months?

[olive sable]

it has a pause button apparently

i just need my gpu for vrc in 7 hours

[olive sable]

@prime ridge

[nocturne olive]

Assuming it's really batch size 1 and uses gradient accumulation to compensate for that, it's gonna take literal ages due to low utilization and high compute requirement

[olive sable]

he said up the batch size till it crashes

vscode is already crashing from just opening the project oflder tho

[nocturne olive]

Is there even an option for gradient accumulation?

Without gradient accumulation, the batch size is gonna suck and the model is gonna suck

Apparently most modern LLMs are trained with like 10s of thousands of batch size

[olive sable]

gradient checkpointing is a thing

apparently

[nocturne olive]

I don't know if that's the same thing as gradient accumulation

[olive sable]

i dont know either

[olive sable]

i think im fine with just not doing that part of the command

i read through the code and there is a fallback

not suprisingly, rip ssd

[nocturne olive]

That write activity looks unusual

Is it dataset preprocessing?

[olive sable]

i have no clue

[nocturne olive]

Look at what the code is doing then

[opaque sigil]

gradient checkpointing is for calculating some gradients on demand during backprop, gradient accumulation is for accumulating gradients and updating them fewer times

[nocturne olive]

Sounds like different things

[opaque sigil]

very different yeah

[olive sable]

[nocturne olive]

I assume gradiend checkpointing doesn't do the same thing as gradient accumulation where it increases the effective batch size by a multiple

[opaque sigil]

no you just directly trade memory usage for extra compute usage

which i guess means bigger batches indirectly

[nocturne olive]

Well, certainly seems less scalable than gradient accumulation, and also less powerful

[olive sable]

he said put the batch size higher till it crashes, so i have it at 100 rn

[nocturne olive]

Uh, check your VRAM usage in case it tries to leak into system RAM

[olive sable]

its not crashing but it doesnt look like its doing much either

1.7GB/24GB

[nocturne olive]

Probably not training yet then

Might be dataset-preprocessing, which can take ages

[olive sable]

it is using a decent amount of ram tho

5GB or so

[nocturne olive]

It's probably preprocessing
When I was pretraining one time the preprocessing step used a total of like 300GB of memory, including the pagefile

[olive sable]

[nocturne olive]

100 batch size is probably way too much though
If it's currently at 512M parameters and my 12GB card maxed out at 24 batch size with 150M parameters or so, it's not gonna be even close

[olive sable]

"warmup_steps": 200 ?????

[olive sable]

ok

[nocturne olive]

200 training steps to slowly ramp learning rate

[olive sable]

ah

[stark needle]

Gradient checkpointing is just remat

[olive sable]

does the preproccesing get saved or does it do this each time?

[nocturne olive]

I don't know, if the code sucks it redoes it every time

[olive sable]

uhhh

[nocturne olive]

That part is up to the competency of the programmer

[stark needle]

Sam

[olive sable]

yes?

[stark needle]

Do u have access to the config to change the model params

[olive sable]

yes

[stark needle]

Aka where ndim etc is defined

What is current config

U should be able to optimize the model size

[stark needle]

what hidden dim

[olive sable]

i have no clue what any of this means

[nocturne olive]

Hm, 8 gradient accumulation steps

[stark needle]

theres model dim and hidden dim

[nocturne olive]

Seems a little SMOL though

[stark needle]

Wait i mean intermediate dim

The expanding FFN thingie

[stark needle]

Nah its good

[nocturne olive]

Well, depends how much the base batch size will be

[stark needle]

It says 100

So

[nocturne olive]

That probably will crash

Sam is still trying configs

[stark needle]

Ah

[olive sable]

i got told, put the batch size as high as possible

[stark needle]

Ye but only till ur hardware can handle

[olive sable]

checkpoints

nott my code

[stark needle]

is this code oss

[olive sable]

oss?

[hoary lion]

Hi #programming

[stark needle]

open source

[olive sable]

hello

[stark needle]

Hiii

[olive sable]

i dont think so?

he dm'd it

[stark needle]

fair

Actually i better shouldn't give out free info abt llm pretraining

[olive sable]

im getting paywalled

[hoary lion]

[stark needle]

irl paywall

my rates start at 1799$/hr as a "ex googler"

[olive sable]

googley moogley

[stark needle]

this is average ex googler salary right?

[olive sable]

idk

[stark needle]

A trillion gazillion dollars per hour

Someone pls pay me

[stark needle]

in return i will let you pick me up from the floor

[olive sable]

POV

[stark needle]

Who would not give *me* a trillion gazillion dollars

[stark needle]

[olive sable]

welp, i opened the side menu in vscode to see if anything changed int he project folder, and it crashed

[stark needle]

THIS WAS FREE TRIAL BTW

[olive sable]

thats what happens when the dataset is 3 milion 6kb files i guess

[stark needle]

parquet my beloved

[olive sable]

i have parquet flooring, but i doubt that is what you mean

[stark needle]

https://parquet.apache.org/

Most insane compression

All trillion scale datasets are stored in this

[olive sable]

fuck it, im adding prints to the file so i know what its doing

[hoary lion]

parquet

[ruby timber]

parquet

[stark needle]

p

parquet

Arquet

arq

Arc

Intel Arc A770

[olive sable]

ah yes

apparnetly its stuck at

# Setup training
total_steps = self.setup_training(tokens_dir)

[stark needle]

My beloved

[ruby timber]

This is a get stickbugged level evolution of a word

[stark needle]

Yes

Thanks

Thanks for reminding me of what i want

And that I'm underqualified for

need to be like that boss of mine with 3 Masters and 1 PhD

[ruby timber]

[olive sable]

aight, i found where its stuck inside that function.

for file_path in self.file_paths:
  with open(file_path, 'r', encoding='utf-8') as f:
      data = json.load(f)
      self.total_samples += 1

this one is slow as fuck

i mean, with 3 milion files i can see why

[stark needle]

@ruby timber

lilac my beloved

[ruby timber]

I love this meme so much

[stark needle]

https://tenor.com/view/my-beloved-gif-16745057618688160042

[ruby timber]

[prime plaza]

would it be foolish to want to get some sort of programming job in the future

[ruby timber]

[tender river]

programming job? clearly wishful thinking

[prime plaza]

tbh I don't know anything about the industry lol, just that the idea of programming seems fun so far

soooo

high competition

[olive sable]

[ruby timber]

Try it out and see for yourself, if you're having fun with it in the long term, maybe it's for you

[olive sable]

i feel the need to ask, but this code isnt actualy doing anything is it?
this could all have just been a len()
its been doing this for 15min rn

[ruby timber]

Only you can decide if programming would be good for you

[prime plaza]

I mean it probably would be, it's just whether it's worth pursuing a career in

but if the competition is high then I have zero chance lmao

[ruby timber]

Is data even used?

[olive sable]

nope

[tender river]

LLM code

[ruby timber]

That's not how that works

It's always worth it if you're willing to put in the work

Sure the competition is high, as with every "interesting" job imo

Just go for something that you're interested in, if your heart's in it, it'll be easier to put in the hours

Rather than going for something that doesn't interest you but is "worth getting a job into"

[ruby timber]

Bruh

[prime plaza]

I mean like statistically I'm in a very deep minority, I probably can't get a bachelors in computer science or anything because my math isn't the greatest (at least for now), meanwhile there are people who dedicate their lives to it lol

[stark needle]

gwuh

[ruby timber]

You don't need math (kinda)

[tender river]

good maths aren't needed

[prime plaza]

right now translating is my main career choice, just programming would be cool to combine with it

[prime plaza]

eh to get a degree it requires math, and most jobs require degrees

[tender river]

if you have the basic idea of what calculus is that's enough for CS ed

[ruby timber]

You could enhance your translating using programming, that's a good idea

[prime plaza]

what's calculus again

[warped narwhal]

remember that most jobs at faang don't actually give a shit about your degrees and qualifications. As long as you can prove that you know what you're doing and that you work well in a team, then they will likely accept you.

for example, can you explain why you did something in a certain way, and explain why (at least in your opinion) it was the best way

[stark needle]

math

https://tenor.com/view/drake-gif-21355539

[ruby timber]

I'm not an expert but it's not as simple as "you have a degree you enter and if you don't goodbye" anymore, at least not where I live

[olive sable]

noway, i get an actual UI now

[tender river]

derivatives/integrals

[ruby timber]

Though I'll admit it's still very much the norm

[prime plaza]

just when I looked at Seek most of the job offers had requirements of a bachelors in computer science

[olive sable]

ahhhh

hmmm

welp, it crashed

[ruby timber]

Let me translate what the GPU is saying

[warped narwhal]

check whether those are the minimum or the recommended. Indeed only lets you set one so they usually put in their recommended. You may have to go to the companies website to find this out

[ruby timber]

...

[olive sable]

[prime plaza]

wha'ts indeed

[olive sable]

im guessing lower batch size?

[stark needle]

Try 8

[warped narwhal]

for example, most jobs at AWS recommend that you have a degree, but you'll see that their actual minimum is just 1 years experience in a professional environment, a degree is just their recommended

[prime plaza]

also I don't think I've ever touched derivatives and integrals in my entire life

[olive sable]

8 sounds like not that much lol

[warped narwhal]

a job searching service, there are others, it is just one example

[tender river]

thats like.. something they're supposed to teach you in school

[stark needle]

Then start touching derivatives and integrals

[warped narwhal]

platonically*

[prime plaza]

it might've been taught in year 12 but I kinda dropped to an easier math class by then >->

I graduated through a system that didn't care about math or english like normal schools so it makes sense

[warped narwhal]

most countries start to teach it when you are at or around 14-15, and only if you choose math as a subject

[prime plaza]

so like my main field of study right now is Japanese and translation

[stark needle]

We had no calculus at school

No one even knows what this is here

[olive sable]

damn

[warped narwhal]

[tender river]

to be fair

i forgot everything i learned about calculus in school

[warped narwhal]

same lmao

[tender river]

i had to remember when they retaught us in uni

[prime plaza]

oh wait is it sin cos tan

[tender river]

no

[warped narwhal]

nah, thats trig

[prime plaza]

damn

[warped narwhal]

useful nevertheless, but slightly different

[olive sable]

is there supposed to be nothing in ram only in vram?

[stark needle]

If you know what Σ is, u know more than a ton of swiss people

not even kidding unfortunately

[tender river]

of course i know! it's a hit song by betsy and...

[prime plaza]

I don't know that, wasn't it like the sum of something

[warped narwhal]

I've had people in my soft. eng. course that had to use a calculator to work out 12+53

[stark needle]

Yes

[prime plaza]

ok then I do know it, I think

also yeah I have never seen this ever

[tender river]

i mean, i've had moments like that
opens python
types 1+3

[warped narwhal]

https://www.youtube.com/watch?v=QfUS5E8o2oI

[olive sable]

its doing stuff now

[tender river]

https://www.khanacademy.org/math

its not a big deal to just catch up on stuff you dont know

[prime plaza]

ok so like is it worth trying to get a job in programming or not I feel like I'm getting mixed signals (also doesn't help that my math skills are subpar)

[tender river]

only you can answer that

[olive sable]

927 hours
bro said 20, he forgot the 900

[prime plaza]

oh yeah actually learning the math won't be hard, it's figuring out what to learn that's an issue

[tender river]

we all like programming in this channel

[stark needle]

Lowe it till no shared memory is used

[tender river]

but what we talk about here aren't our jobs

[prime plaza]

if it's in high demand and difficult to actually get a job in then I don't see a point, objectively/lh

[olive sable]

oh, okay

[tender river]

(unless something interesting happened at our jobs of course)

[warped narwhal]

you're unlikely to get objective answers to that question in #programming

there may just be a slight bias

[stark needle]

I "like" programming

[prime plaza]

fair enough

[tender river]

difficult doesn't mean impossible, the rest is luck, effort and connections

[warped narwhal]

with a double serving of connections

[prime plaza]

yeah connections is funny (I have none)

[stark needle]

Connections are OP asf

[olive sable]

we have connections in here so idk

[tender river]

i have an internet connection

[prime plaza]

I live in Australia so uh

[olive sable]

rip

[prime plaza]

I sometimes have an internet connection

[olive sable]

I no connections with anyone or any identity of asset in this discord. I am not here under my own knowledge, more than likely added by a third party. I have no sympathy for the things found in this discord, and affiliated peoples

[hoary lion]

i only have internet connections

would you be my online friend

[warped narwhal]

Only if Mercury is in retrograde

[olive sable]

i really want to

chug chug with you

[prime plaza]

that's a lot of big words

[olive sable]

its going

[prime plaza]

I'm stupid what does that mean

[olive sable]

no shared mem, but only 15gigs vram now

from 10 to 8 i lost half the vram util

[olive sable]

llm shit

[prime plaza]

[olive sable]

tbh i have no clue either, this is not my code

[prime plaza]

real

[stark needle]

hello i can be your online friend yes

[olive sable]

(shadow, is your headset charged?)

batch size 14 seems to be the sweet spot

oh nope, justy slightly went over

i guess 13 then

[prime plaza]

I can't wait for when I understand all these big words

[olive sable]

i dont know most of them, but im learning

[prime plaza]

tbh I just wanna make an ai lmao

[olive sable]

94 hours, so 4 days

[prime plaza]

there's no way in hell I could make a good one with the specs my pc has though

8gb vram and 32gb normal ram

[olive sable]

im doing this ai training rn with someone else's code cuz they didnt have a good enough gpu

[prime plaza]

ah

[olive sable]

ngl the code isnt too optemized but im not cpu limited rn so its fine

only had to do small changes

[prime plaza]

I think I'm everything limited lol

[olive sable]

i got 64gb normal ram and 24gb vram

[stone cedar]

As long as you're not motivation limited, anythings possible

[warped narwhal]

I have 64gb as well, but not for AI.

I have it because DCS eats all the ram in your pc

[olive sable]

i got it for blender mostly

mainly do gamedev tbh

[prime ridge]

Oh

[prime ridge]

Beautiful

[olive sable]

hello

you can expect it to be finished in 90 hours

[prime ridge]

No its cooked

[prime plaza]

[olive sable]

btw, did you use ai to write the code, cuz one part of it was a bit whack

[prime ridge]

Its using wrong model size

[olive sable]

oh

[prime plaza]

I am completely normal and mentally stable.

[warped narwhal]

is this using ratatui?

[prime ridge]

Lemme fix the json

[prime ridge]

Cuses

Curses

Its Python

[prime plaza]

haha lack of money and income go brr

I make zero money and am currently in a little debt lmfao

[olive sable]

same

[olive sable]

its 512 rn

was it not supposed to be?

"model_dim": 512

[warped narwhal]

its meant to be 1024 (I know absolutely nothing about AI)

[hoary lion]

model dim means intermediate FFN projection size

what you actually need to do is to keep track of total # of parameters

[prime plaza]

I just gotta hope my mum decides I've done enough chores to give me anything

[prime ridge]

Its a 30m model rn 😂😂😂 thats cookee

Should be 512m parameters

[nocturne olive]

Yep, way too much batch size

[olive sable]

its doing fine rn

at 13

[prime ridge]

Loading the entire dataset

[nocturne olive]

Get it out of shared memory, should go way faster

[olive sable]

but you werent using the data you got out of it

[prime ridge]

Yeah make sure it doesnt leak into shares mem

[prime ridge]

Validation ik its cooked

[olive sable]

it is

[olive sable]

what?
validation for what?

it just wastes 2 hours beforehand

[prime ridge]

💀

Why 2 hours

Took me like 20 min

[olive sable]

cuz there are 3 milion files, and its python

[prime ridge]

U can remove it. I forgot about that

Coding in one big file has its disadvantages 😵‍💫

[olive sable]

i didnt actually wait 2 hours, i gave up after 30min and replaced it with a len()

[prime ridge]

Alr

Thats cool

[olive sable]

you might have more single core idk.
i got a 5950x

[prime ridge]

Hopefully ur batch size will be at least 10 😬

[olive sable]

its 13 and doing fine

[prime ridge]

On new model size?

[nocturne olive]

30M parameter model at 13 batch size and already using all the VRAM......

's training code was better

[olive sable]

yes

[prime ridge]

Idk how

[nocturne olive]

It could do 150M at 24 batch size on half the memory

[prime ridge]

Still doin 30m param

Did u pass in the config json file?

[nocturne olive]

Like, what? How can you be so much worse than 's garbage code

[olive sable]

i dont have a config json, im just changing hte default config in the file

[prime ridge]

Oh right I deleted it cuz it was wrong

Im not at my computer rn so I cant send it

Thatll work tho

Sorry for this shitty code

[olive sable]

just tell me what to change and ill do it

[olive sable]

nah its fine

[prime ridge]

Should be in dms

[nocturne olive]

It's truly garbage if it's so much worse than 's very bad code

[prime ridge]

Just model depth and ffn size

[nocturne olive]

Your code: 24GB. 30M params, 13 batch size
: 12GB, 150M params, 24 batch size
Though now that I think about it, it's possible it's the CTX

The CTX scales horribly inefficiently

I don't remember, was it expontential if you don't optimize it at all?

[prime ridge]

No bro

Not 30m

Its not exponential its quadratic

And I didnt implement any additional optimizations

[nocturne olive]

You said Sam was running 30M params there

[prime ridge]

It might be a ui glitch

Cuz I might have accidentally hardcoded 30m param

Maybe

I hope not

But during debugging there is a chance

[nocturne olive]

Did you by any chance use an LLM to write the code?

[olive sable]

"Model size: ~" + f"{self.config['model_dim'] * self.config['depth'] * 8 / 1000:.1f}M params"

in the code

[prime ridge]

For a lot of it

[prime ridge]

Yes

[nocturne olive]

So it's definitely trying to calculate it
Which would mean it's either calculating incorrect values or training 30M

[prime ridge]

Damn how tf is on 30m 😭

[olive sable]

should be around 245

[prime ridge]

512 ish

[nocturne olive]

Well, whatever you're doing, it seems like you're encountering some sort of silliness

[prime ridge]

Fs 😂

[olive sable]

1280×24x8 / 1000 no?
about 245

[scarlet arch]

Guess who can rebase things again tomorrow. Wohoo....

[prime ridge]

Squaree

Squared

[olive sable]

ohhh

ah ok

[prime ridge]

1280^2

Times 24 timesn 12

[olive sable]

its not getting squared in the code tho?

[prime ridge]

Def is

Its using the set parameters not default

Ohhh

Ik what its doing

[olive sable]

squaring would be self.config['model_dim'] ** 2

[prime ridge]

In the main section

Its reading my set values

Pretty sure

Did u download the json file?

[olive sable]

nope

[prime ridge]

Ok so I think its in the main section

Reading code on mobile is fucked

Gimme a min

Hmmm

💀💀💀

Its a ui glitch

So we chillin

Probably...

Just make sure u are rerunning the program

It wont auto update or anything

And make sure u clicked save on thr file

[prime plaza]

can someone tell me what it means by "faster" when the line of code is literally done in a fraction of a milisecond

[scarlet arch]

optimized code is often measured in nanosecond improvements.

at least in my code, idk

[prime plaza]

damn really

aren't those like

really small

[scarlet arch]

also, idk how this works in python but in Rust a list and set are on the heap, while tuples are on the stack

[scarlet arch]

yes. but if you have really really hot code, it adds up

[noble zodiac]

you might not really notice a difference when the amount of data is very small but that can change drastically when we are talking about vast mounts of data

[scarlet arch]

some optimizations we do appear like arcane magic to outsiders

[prime plaza]

ig that makes sense

I just don't know how much higher that can stack up

[olive sable]

fixed it.
but like i said before you arent squaring it here so you just get 245

idk why it even needs to be squared tbh

[noble zodiac]

look at the big O notation complexity of operations on List, Set and Tuple to get a better idea

[scarlet arch]

While maybe not translating quite directly to Python, the collections documentation of the Rust standard library is great for learning.

https://doc.rust-lang.org/std/collections/index.html

[stark needle]

Just count params manually

Also 30M model should be doable on a 3090

My 3090 handles 300M param pretraining ez

[olive sable]

it seems like mine really doesnt like these settings

im still going over vram at batch size 6

[daring nebula]

Bro, tbh I'm listening to this from moment you sent this... Why the f I like it lmao

[nocturne olive]

Whuh

It's just a NeuroSynth test

[olive sable]

batch size 2 is still going over vram

[stark needle]

Something wrong is happening somewhere

[daring nebula]

IT WAS TEST EVEN

[nocturne olive]

Whar

[olive sable]

idk what im doing shadow

[stark needle]

[olive sable]

i can provide the hardware to train on, but the code is borked

and i dont know llm code

[daring nebula]

Keep going tho... I like it

[nocturne olive]

Me sillyin'

[olive sable]

its obvious this was made with an llm too, i feel like there's no saving it unless an actual profesional looks at it

[nocturne olive]

Incomplete and still heavily WIP

[stark needle]

[olive sable]

it sounds so good

[nocturne olive]

LLM code: takes longer to debug than it would have taken to just write yourself

[olive sable]

ye, kinda

[nocturne olive]

It's just a silly thing I've been doing

Here's one test where I used both NeuroSynth and Kasane TETO

[olive sable]

@prime ridge in its current state its not really worth running

something is borked

[daring nebula]

🔥 oooh yeah

[nocturne olive]

Was mainly me testing Kasane TETO, but decided to mix it properly and it turned out really well

[stark needle]

It would be so over if somehow my Bluetooth malfunctioned and leaked into the office space the songs I'm listening lmao

[daring nebula]

NeuroSynth, is that your tool?

Sorry if I look like an ignorant or dumb

[nocturne olive]

NeuroSynth is the model I use
Current latest is BETA-2

It's one I'm developing with Wispers

Neuro vocal synthesizer model

[daring nebula]

Is that RVC model?

[nocturne olive]

No, vocal synthesizer

[daring nebula]

Got it, yup

[nocturne olive]

Gonna replace the old SynthV + RVC chain with a full-chain vocal synthesizer to remove the dependency on an expensive piece of software

[stark needle]

I love meetings by higher ups when they inform of a bunch of changes and i'm uninformed by basically all of them

I'm like

Ok nice but i got no idea what ur talking about

so good for u

[trim valve]

real

[stark needle]

@sage crag just got a book with stuff I'll be looking at the next whole year in math and they said if i can complete this book i should have no problems in math for the whole year

The book in question:

yeah here it's grown ass adults😭

Also this is the only thing left between me and university

If i complete the next year i can do the uni exam

[tender river]

if my headphones disconnected while i was listening to music everyone within 50m vicinity would suffer levels of psychic damage dangerous to their lives

[hoary lion]

holy vocab

[tender river]

(except that never happens since i never have no-headphones volume at nonzero levels)

[tender river]

wdym vocab this is basic english

[hoary lion]

vicinity is not everyday word

or is it

idk

[stark needle]

Konii am i cooked

[tender river]

honestly i bet if you learned it in school you'd forget it by now anyway these topics are worth revisiting even if you learned them already

in general maths is so big you're bound to forget some parts, you're mostly learning so you can easily revisit it when you encounter it again

[hoary lion]

this lmao

[tender river]

i personally don't remember how to solve quadratic equations

[hoary lion]

just search up the internet would be enough no?

basic math moment ig

[tender river]

i'd be able to derive it if you give me 10 minutes (surely ) but i dont remember it off the top of my head

maybe 5m

[stark needle]

@sage crag https://www.swissinfo.ch/eng/society/pisa-survey-swiss-students-perform-well-in-maths/49032102

"But 25% of Swiss pupils failed to reach the minimum level in reading."

[hoary lion]

quarter of them are unable to read lmao

[stark needle]

Unable to read AND math

[hoary lion]

https://tenor.com/view/dog-dog-staring-at-sun-cute-dog-dog-closed-eyes-dog-staring-at-sunset-gif-10310895250239531072

[tender river]

let's just say i didn't do some of my homework

[stark needle]

Switzerland

i will never become an actually useful ML research scientist at this rate

[tender river]

easy enough yeah

[stark needle]

can't even read

[prime ridge]

Wdym

[tender river]

surely discord is reading

[hoary lion]

i mean i personally think it is about experiences

[prime ridge]

Ive got the code to work

[hoary lion]

tbh math in ML is more like general usage of equations instead of twisting it and make it hard to comprehend

[prime ridge]

It trained and it worked

[hoary lion]

so you got it shadow

👍

[prime ridge]

Its not that hard

[stark needle]

Not if u wanna do revolutionary shit in foundational ml

[hoary lion]

🤔

i mean

[stark needle]

def relu(x):
if x > 0:
return x
else:
return 0

[hoary lion]

i am undergrad and I can comprehend most of attention stuffs so

you good

[stark needle]

Attention is not hard

[prime ridge]

💀

[stark needle]

[olive sable]

The vram usage is out of control

[prime ridge]

Thats not because of the code

[hoary lion]

oh yeah ofc attention is but like new ones yk

[stark needle]

Whats hard abt attention it's literally just dotproducts

[prime ridge]

Thats normal

[hoary lion]

like underlying logic of RoPE and such

[stark needle]

and topk

[prime ridge]

Its not

[hoary lion]

they are alright

[olive sable]

Batch size of 1 is even over 24gb

[prime ridge]

The optimizer is fucked

Thats just adam being shitty

[stark needle]

?? what is hard abt this

[prime ridge]

I can try to add more optimizations tho

[prime ridge]

Back prop

[hoary lion]

but ts is not okay for me:

[prime ridge]

Adjusting attention matrices ia hard

[stark needle]

Ok sure but the algo is the same backprop

Unless ur talking abt optimizers then sure that's harder

[prime ridge]

With attention is wayyyyy harder than weights

[stark needle]

????

Attention is linear layers

[prime ridge]

But its still a matrix

Let me complain damn

Lile im a hs student 💀

Multi variable is hard enough

[stark needle]

There's no diff between the layers used in mnist and attention

[prime ridge]

Fuck tensor calculus

Manually deriving is hard

[hoary lion]

vanilla attention is 🆗

[stark needle]

but that doesnt make attention hard

[prime ridge]

There is juat a lot

[stark needle]

That makes the optimization part hard

[prime ridge]

True

[hoary lion]

we don't derive by hand all those gradients

[prime ridge]

In academia you do

[stark needle]

???

[prime ridge]

When optimizing

[stark needle]

Only if ur trying to invent something entirely new

[prime ridge]

Understanding how gradients are calculated is important

[stark needle]

Or formal studying material

[prime ridge]

Moreover knowing how they work is still kinda complex

Queries,. Keys, and values is kinda confusing at first

[stark needle]

yeah if u know 0 abt it

[hoary lion]

[prime ridge]

Ok if attention isnt hard what is

[stark needle]

If u take a look at it for a while it becomes clear

[prime ridge]

Thats true for anything tho

[tender river]

oop

[stark needle]

Second order optimizers

certain loss functions

[prime ridge]

Loss functions 💀💀💀💀💀💀💀

[hoary lion]

i never look at optimizers

just use adamw

[prime ridge]

Ngl prolly gunna need to roll my own optimizer

[stark needle]

so real

minicry

2

[hoary lion]

i mean it is extremely rare to even have SGDs fall into local minimum

[prime ridge]

Adamw uses so much memory

[hoary lion]

bc there are billion parameters per model

[stark needle]

Ye but relatively stable

Scales extremely well

[prime ridge]

Exactly

[tender river]

in russian, students call calculus "mathan" (short for "mathematical analysis" which is the russian name for calculus) or differential equations "diffeqs" (or rather "diffurs" for "differentsialnoye uravneniye"), are there stupid abbreviations in maths in english or are english students serious people?

[stark needle]

To gazillion params

[prime ridge]

But..

Vram is a bit short rn

[hoary lion]

english only uses calculus -> calc no?

what is decision here

[stark needle]

Then use optimizer that doesn't use all those extra params

[prime ridge]

Yeah ima do that

[stark needle]

Deciding what to do with ur life

[tender river]

the japanese abbreviation culture is even more hardcore than russian one

[hoary lion]

oh

[prime ridge]

True

[hoary lion]

correct name ig

[stark needle]

Cant wait till i understand optimizers properly

I'll be reading a text and have 0 idea what's going on

[prime ridge]

Idk if anybody does atp 😭

[hoary lion]

optimising in math <- worser than ML

even than writing optimizer from ground up

[prime ridge]

ML is pretty easy usually

[tender river]

i was about to be happy game semantics got mentioned but then realized game theory is a different thing

[prime ridge]

Getting good data is half the challange

[stark needle]

bros saying "oh yeah we used 5 steps of the newton Schultz iteration to solve x problem" in optimizer papers and I'm like "damn good for u or bad for u idk what this is"

[hoary lion]

[tender river]

idk about abbreviations in russian empire but in ussr they went CRAZY with abbreviations and naturally it's still there in the language to this day lol

[maiden geyser]

more like with long ass names

abbreviation is just a consequence

[stark needle]

https://calvinmccarter.wordpress.com/2021/11/18/the-newton-schulz-iteration-for-matrix-inversion/ like wow thanks for the easy explaination

I 100% understand whats going on here

[hoary lion]

Not a single representation is comprehensible

[tender river]

speaking of random math stuff, i like dual numbers https://en.wikipedia.org/wiki/Dual_number

they can be used for differentiation

[hoary lion]

[prime ridge]

Anything past calc 3 is purely meant for torture

[stark needle]

i understand 0 words in here

yea the epsilon limit thingie makes sense

[tender river]

abstract algebra and category theory are fun you should totally learn them

[maiden geyser]

they can't keep getting away with creating random impossible numbers and pretending like they exist

[prime plaza]

how is it that every time I look at math there's always a new symbol for it I've never seen before

[stark needle]

i have 0 idea what sin cos tan is, am i cooked

[prime plaza]

it's just triangle stuff don't worry, geometry or something

[maiden geyser]

sinus cosinus tangent

[tender river]

(these are basically "ok let's see what things mathematicians made up and maybe generalize to see what's common and what's different between them")

[stark needle]

Ye but what they are used for

[maiden geyser]

for messing you up

[prime plaza]

isn't it to find the angles on a triangle

[prime ridge]

For propaganda

[prime plaza]

something like sin(cos/tan) idk

cos(sin/tan)
tan(sin/cos)

idk I'm guessing lmao

[tender river]

for just about everything in geometry

[prime ridge]

S tier rage bait

[prime plaza]

as soon as I graduated I just forgot all math ever sorry

[tender river]

they still do they're just lying to sound cool

[prime ridge]

Its just more work

Usually

[prime plaza]

wait was I the one ragebaiting

ok cool :3

[prime ridge]

Caca

[prime plaza]

imagine using trigonometry

like protractors literally exist

[tender river]

every minecraft player uses trig surely

[prime plaza]

who tf is leaving out random angles for no reason anyways

[stark needle]

trigonometry pentagonometry

[prime plaza]

that sounds nightmarish

[tender river]

honestly though other than minecraft i can't remember the last time i used trig...

[prime plaza]

how do you even use trig in minecraft

everything is 90 degrees there

[tender river]

finding the fortress with 2 pearls

[prime plaza]

ohh ok

forgot about that

I've never beat minecraft

[tender river]

checked my .python_history and rust/*/src, no signs of sin or cos thre

[stark needle]

trigonometry =
tri (three)
gon ("to say" in japanese)
o (oh)
metry (the act of measurement)

the act of measuring Three saying oh

[maiden geyser]

but it isn't trigonometry, you're simply trying to find where two lines meet

a.k.a. a math equation

[trim valve]

[tender river]

and that's done using trig

[stark needle]

why so mischievous

[trim valve]

I did A level further maths w/ decision

so unfortunately yes

forgot most of it now though

[prime plaza]

man math is hard I just wanna make code do stuffs

yeah but maths hurts my head

unlike learning to code, learning to math is just painful and annoying

[stark needle]

https://tenor.com/view/anime-girl-eating-cute-kawaii-gif-17162975

[prime plaza]

well code I just type words and it do funny trick

[stark needle]

pov ai slop chatgdpr coding

[tender river]

you don't know programming enough

[prime plaza]

with math it's like "you see this letter? Do the most complex equation possible to get the number 2 :)"

[prime plaza]

I do not

-# I literally just started and I'm scared

[stark needle]

A lot of CS can be represented as some math equation

[prime plaza]

why do math when I can just make the code do it for me B)

[stark needle]

me when Nvidia fp4

[tender river]

programming is just applied maths, computation is just doing proofs

[trim valve]

when can I make my computer do integration

[tender river]

now

[trim valve]

[stark needle]

What is what times what?????

[tender river]

WINE WHAT SAYE OU TODING

[stark needle]

Why

[rare bramble]

math is fine for me, I have never had problems with math. But I think programming is much more comprehensible than math, at least for me.

[ruby timber]

debugging

but
the bugs......

true

My file has no bugs
it is 0 bytes long

Just don't program

There, problem solved

fudge

[stark needle]

@sage crag https://www.swissinfo.ch/eng/education/why-the-swiss-school-system-is-so-successful/89129497

[ruby timber]

Um

Don't create the file in the first place

don't make me face the hard facts...

[tender river]

this was just a small bit of all the winedbgs it spawned btw, suffice to say i'm reverting back to the old version

[ruby timber]

I can't deal with it

A file? in my filesystem? ludicrous

free eduction

...eduction.... mmmmmmmh

[stark needle]

@ruby timber do you also think swiss schools are so good

We learn so much

[ruby timber]

sounds like a mix of education and abduction

CLICK THE CIRCLES

[tender river]

click the squares (insert masterpiece)

[ruby timber]

I think the mandatory part of schooling is pretty good and also terrible in certain aspects, as with all education systems-

[stark needle]

BETTER THAN NOTHING

[stark needle]

Do you also love learning the things konii learned when she was like 10 but 10 years later

[ruby timber]

If it's useful to daily life, nope

[prime plaza]

just got back from fighting a cockroach after taking pain medication but will do 🫡

[stark needle]

We did NOT learn how to do our taxes in school

Literally last year

[tender river]

here taxes are automatic (unless you're self employed) (i've never done my taxes so i wouldn't know)

[stark needle]

We spent 10 weeks

Watching a netflix series

[tender river]

yumy

[trim valve]

somone in a lower year did the same with copper sulfate

[prime plaza]

I think where I live you can get someone to do your taxes for you

[stark needle]

not for free

Anywhere

[trim valve]

he was dared to for a whole like 50p

based

[ruby timber]

I did learn in my last year of middle school some politic stuff about how the country works, that was pretty neat
like votations and stuff

[trim valve]

all I remember about titrations was people complaing about the math

[ruby timber]

no taxes though

[trim valve]

(I also had no idea how to do the math but I just shuffled numbers around until I got the answer)

[prime plaza]

well yeah not for free, but at least you don't gotta do them

[stark needle]

You will survive one day just cause of that and the rest will perish

[trim valve]

our one was pink I think

[stark needle]

One place where i saw the progress is insane is

In chemistry and stuff for some reason

Like

We go from 0 knowledge to

[prime plaza]

wait in what context

[stark needle]

Frigging protein structures or something

For some reason

[tender river]

i skipped all physics and chemistry in highschool 🔥

[stark needle]

Yet basic ass math is ommitted

[tender river]

also biology and geography

[prime plaza]

I did like 3 subjects in the end of high school and 2 of them were forced upon me

[tender river]

honestly i skipped just about everything i could it's a miracle i passed

[prime plaza]

yeah because school is bs

[tender river]

i did go to some classes but i never went to a single PE class but luckily never got a failing grade

[stark needle]

What if i just quit and become a DHL driver

Or just move to the alps and become a cow milker or something

[scarlet arch]

Would I get a 🥈 medal if I posted this?
https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/

[stark needle]

Borg my beloved

[prime plaza]

so like does AI actually provide any jobs or is it only replacing them

because I saw a video earlier talking about how it's the same as like the invention of the automobile, a computer, the internet, etc.

[stark needle]

WHAT

32.5%????

[scarlet arch]

Yeah. Kinda wild.

[prime plaza]

so it just creates a more competitive environment for finding jobs for everyone 😭

[tender river]

wrong

it's the second coming

[prime plaza]

does ai require you to know programming?

[tender river]

very little

[prime plaza]

sigh

[tender river]

the thing is, LLM writes bad code and you have to be an expert to tell when its good enough and when its bad to the point of not even being fit for the job

[stark needle]

Konii

If it wasn't for llms you wouldn't be here

[lofty hill]

The problem is when the llm hallucinates and give a bogus code and you can’t tell what’s wrong

[tender river]

its not even about being bogus, the code can be bad while still doing what's asked