programming

[sage crag]

dont call ths number agin

MiyagiCroco

[real sierra]

[mossy fossil]

you're not my boss

[sage crag]

shimonana

[real sierra]

is your refrigerator running

[mossy fossil]

you better go catch it

[sage crag]

i have disturbing

monospace, emoji break

[real sierra]

[sage crag]

[wind merlin]

shiro bday

[sage crag]

notn unifomr

[real sierra]

[wind merlin]

happy birthday

[wind merlin]

[real sierra]

not cake

[sage crag]

emi not know ee

[mossy fossil]

overused joke count: i genuinely lost count

[real sierra]

[rough bloom]

[real sierra]

today not my birthday

[sage crag]

https://cdn.discordapp.com/attachments/1421861060507602944/1425936142754123927/neuroCatModeOfOfnnfOOf.webp

[real sierra]

[sage crag]

glueful

[real sierra]

tomorrow not my birthday either

[tender river]

shiro birthday same konii birthday

[wind merlin]

can't believe you lied

[real sierra]

or any time this month

or even next month

[real sierra]

it not me

[mossy fossil]

Shiro's Birthday is on February 31st

trust

[sage crag]

shirosmany

shirosmany

[tender river]

it not lie though

shimo birthday

every day

[sage crag]

https://tenor.com/view/bunny-rabbit-pats-pet-bunny-pats-gif-19861222

[real sierra]

bun

[warped narwhal]

bnuy

[real sierra]

bune

[sage crag]

buny cause

imag e retentiuon

burn-in

[real sierra]

[wind merlin]

so shiro is at least 18262 years old

[warped narwhal]

are the neuro and evil slow cat dance emojis on this server or only on twitch?

[wind merlin]

light mode

[mossy fossil]

shiro is actually 0 years old

[sage crag]

https://tenor.com/view/my-jolly-reaction-jolly-my-honest-reaction-gif-5490890317762380451

[mossy fossil]

trust

[sage crag]

no sleep

[real sierra]

used to have cat dance as sticker I think

now have this

[sage crag]

myniop

p,j

[wind merlin]

cinema

[real sierra]

?

[tender river]

https://cdn.discordapp.com/emojis/1454583294753771530.webp?size=48&name=evilDentge

me need

sleep

[sage crag]

uweweee

[real sierra]

chayleep

[wind merlin]

[sage crag]

MiniBald

[real sierra]

[sage crag]

balled

spher

[mossy fossil]

install gentoo

[warped narwhal]

holy cursed

[severe path]

[sage crag]

https://tenor.com/view/jeong43va-gif-26093523

nooo me not tires

[warped narwhal]

I've been reading this chat for months and I still don't know what wrr means

[real sierra]

🛞

[wind merlin]

means

[sage crag]

phue

[wind merlin]

tyres*

[sage crag]

happy birthdaemi

eoeoe

[wind merlin]

[sage crag]

to you

[wind merlin]

[sage crag]

r

[mossy fossil]

these tyres?

[sage crag]

im skleping

bwa

not

bwaa

[severe path]

GT5?

[sage crag]

noooo

[mossy fossil]

GT4

[severe path]

Off by one again

[sage crag]

hapie

[severe path]

Damn

[real sierra]

I speaking Canadian English not British english

related languages but different

[mossy fossil]

shir, genuine question

[sage crag]

cananadian

[wind merlin]

not british

[sage crag]

banana

nnnnn

[real sierra]

👂

[warped narwhal]

🏴󠁧󠁢󠁳󠁣󠁴󠁿?

[mossy fossil]

is your internet better than what it was about 2-3 years ago?

[sage crag]

peueueu

[tender river]

tire

[wind merlin]

🇮🇪

[real sierra]

my Internet randomizes every 4 months when I move

[sage crag]

mee notire

bwuh

[sage crag]

toot

[real sierra]

currently it's better

[sage crag]

char horn

[wind merlin]

https://tenor.com/view/mario-tire-change-anime-sad-super-mario-tire-gif-17314634

[mossy fossil]

some progress is better than none ig

[sage crag]

https://cdn.discordapp.com/attachments/1300807614032121950/1409945130810609695/3dgifmaker99373.gif

orbie

[severe path]

Do you have those tyres with the alumni**ni**um?

[wind merlin]

oh right
americans say aluminum

[sage crag]

https://cdn.discordapp.com/attachments/1059569601144442911/1375623285894807563/IMG_6849.gif

cubo

[severe path]

Where did you get that cube from?

[real sierra]

shube

[severe path]

I want answers

[sage crag]

sherm organic

nnnnnnior

wororor

[real sierra]

konii schiz

[sage crag]

ueeee

[real sierra]

turn off and on again

[severe path]

I am a little bit confused at the moment

[sage crag]

nooe npower of

[real sierra]

power off

[sage crag]

https://cdn.discordapp.com/attachments/1248294218191999118/1353485622152794172/liellesocute.gif

[real sierra]

[sage crag]

nor

catmodeonoofonoff

[tender river]

<-

<-

<-

[sage crag]

uwah

[real sierra]

[tender river]

https://tenor.com/view/shuukura-sendai-hazuki-story-about-buying-my-classmate-once-a-week-good-night-gif-4570780052212669849

<-

[real sierra]

📣

📣

[sage crag]

eyey not

[real sierra]

📣

📣 🍌

[sage crag]

clsoe

stopie

[tender river]

https://tenor.com/view/my-jolly-reaction-jolly-my-honest-reaction-gif-5490890317762380451

just close them

[real sierra]

☑️

[sage crag]

neeee

keyoboard

https://tenor.com/view/mejiro-mcqueen-spin-umamusume-anime-girl-cute-gif-26001972

[real sierra]

lazy discord

[sage crag]

use zune_png::{PngDecoder, PngEncoder, zune_core::{bit_depth::BitDepth, colorspace::ColorSpace, options::EncoderOptions}};
use std::fs;

fn main() {
  let mut bytes = include_bytes!("../base.png").to_vec();
  let color_space = ColorSpace::Luma;
  let bit_depth = BitDepth::Eight;
  let c = color_space.num_components() * bit_depth.size_of();
  for _ in 0..255 {
    let mut size = (bytes.len() / c).isqrt();
    let mut i = 0;
    loop {
      if size * size * c == bytes.len() { break; }
      bytes.push(bytes[i % bytes.len()]);
      size = (bytes.len() / c).isqrt();
      i += 1;
    }
    let encoder_options = EncoderOptions::new(size, size, color_space, bit_depth);
    bytes = PngEncoder::new(&bytes, encoder_options).encode()
  }
  bytes = PngDecoder::new(&bytes).decode_raw().unwrap();
  _ = fs::write("out.png", &bytes)
}

popipo

[real sierra]

[sage crag]

ooo

[tender river]

https://cdn.discordapp.com/attachments/1072697081443131476/1397617611474468914/neuro_cheerleader.gif

[sage crag]

ElvynSerius

ep

[tender river]

heeead

[sage crag]

headache

pwuh

https://tenor.com/view/bunny-rabbit-pats-pet-bunny-pats-gif-19861222

[real sierra]

englis so weied

how was it a rule for so long that you can't put two nouns beside each other

and then everyone just started ignoring it

[sage crag]

tree tree

wrr

[real sierra]

that's why compound words exist

"fire place" not legal to say

[sage crag]

tree o f tree supertree tree

[real sierra]

so they made it "fireplace"

[sage crag]

head achu

[potent geyser]

vedal vedal

[sage crag]

bles yu

[real sierra]

other times they just hyphenate it tho

[sage crag]

by

[mossy fossil]

are these the aluminum tyres that some guy was talking about?

[sage crag]

by b ye

em mie go bed

[real sierra]

night night konibwaa

sleep tight

[sage crag]

,-

[real sierra]

don't let the ermshark bite

[sage crag]

ermie sh

[real sierra]

[sage crag]

axolotl

loa m

[real sierra]

loam

[tender river]

no more loam

[real sierra]

silty clay loam

[tender river]

enough

[sage crag]

ambstome

whah

i hav axolotl

m ade of

heat

it loam

[real sierra]

heat

[sage crag]

uee

aa

[real sierra]

koniiiiiii

[sage crag]

bright

[mossy fossil]

(i literally booted Gran Turismo 2 just for this screenshot)

[real sierra]

eep

[sage crag]

ogey

by

[real sierra]

programming still here tomorrow

or else

[wind merlin]

goodnight korone eep well

[rigid snow]

windows xp

how make screenshot wen no printscfreeen keu

[tender river]

not fair me want eep too

[potent geyser]

Fraps!

[rigid snow]

thank

[tender river]

[potent geyser]

just don't record anything with Fraps

since it records uncompressed data

[sage crag]

dente

[tender river]

[sage crag]

ebil

[tender river]

https://tenor.com/view/ai-denial-no-head-shake-nope-gif-18396288939731808167

[sage crag]

ßsssss

⁵

[tender river]

nesus letter

[sage crag]

x̌

y

by

[tender river]

sleep well

[young oyster]

I think when neuro cant win she gets stuck going through menus to stall and live as long as possible.

[rigid snow]

??

pleas jhel p what is thus? i thuis gpu shoudl support cuda?

[tender river]

winxp discord

[rigid snow]

this isserious matter. i. need maek neurieosama.

[tender river]

sorry me head

dented

[rigid snow]

:mlntcandy:

[tender river]

true

[rigid snow]

noot work

here

[tender river]

https://cdn.discordapp.com/emojis/1454583294753771530.webp?size=48&name=evilDentge

[rigid snow]

:BlankiERMs:

why it autofill

when not work

:mlntcandy:

[tender river]

@silent anvil pls fix

[hard raptor]

I'm working on a dotfile manager in bash, it probably is gonna be absolutely horrible tho... probably not even that useful

these are at the root

[tender river]

head

head

head

https://cdn.discordapp.com/emojis/1454583294753771530.webp?size=48&name=evilDentge

me just

confuse 0 indexing and 1 indexing

[hard raptor]

and this is the config from the defaut profile

maybe i need to put an rm -rf so that it actually deletes the files it is gonna replace..

just a

 35 -  cp -rv $TPATH $SPATH
 35 +  rm -rf $SPATH
    +  cp -rv $TPATH $SPATH

will do

i guess it can have some usecases.. for example if you wanna switch between multiple dotfile configs from time to time

[tender river]

me sleep

[tender river]

also i assume you know about https://www.gnu.org/software/stow/ but if not check it rr

and

banana

[hard raptor]

dam gnu.org being slow AF

[hard raptor]

Darn.. why didn't i think about symlinking lmao.. well.. atleast my thing could also be used for backups... don't think symlinks can do that

[hard raptor]

but that's what vcs can do

[opaque sigil]

Embed a templating language and switch things that way

[hard raptor]

so i actually added a proper backup system for the profiles

maybe i should optimize it and use diffs instead of just copying

[opaque sigil]

Incremental backup

[hard raptor]

Well i didn't really make them incremental

Just see them as like git branches or something

[opaque sigil]

[hard raptor]

I basically implemented a primitive version of git lmao

Github repo comming soon(tm)

[true hemlock]

did you know that 60% of our protein coding gene has a recognizable counterpart in the banana genome, and 40% of those are identical amino chain wise

[opaque sigil]

I did not but genomes are weird so I'd believe it

[pearl forge]

[real sierra]

i dont usually bite on a roblox video but

https://youtu.be/ctdDurZ8EbA

this person seems like they'd be in this server

like this person just seems to me like they're exactly neuro's target demographic

video is also fire its more about programming and math than roblox

[visual pawn]

Not having watched it yet, I think there are more people here into (relatively) higher level programming rather than dealing with circuits and such

At least I haven't seen anyone nerding out over anything in assembly

[amber fractal]

We have some assembly, most of it is more in language dev area

I personally enjoyed the video

[final tinsel]

hi #programming! anyone here have experience with llamafiles?

[azure lynx]

Essentially, . llamafile acts as the distribution vehicle, packaging everything needed to run the model. llama. cpp provides the core library for efficient model inference, while ollama builds upon it to offer additional features and optimizations.Dec 13, 2024
ok, that means I guess i do have experience with it coz i'm using llama.cpp ?
(having to look it up with google means i don't have extensive experience though)

[amber fractal]

@mighty thorn would this be something you know by chance

[final tinsel]

this is the command and output (w/ verbose flag) but whenever i run on gpu it freezes on "loading bos token..."

[mighty thorn]

What

No

I don’t use llama cpp due to my lack of intelligence

[final tinsel]

i like llamafile cuz its (in theory) incredibly simple

i would reference my previous projects with llamafile but well...

i lost all that data when my secondary drive died lol

[rough bloom]

Mistral 7B is ancient enough that I'd expect it work, but given that there's already an error with the chat template I think it may just not be supported by whatever version of llama.cpp they baked in there

just try with normal llama.cpp mabe

shouldn't be that much more difficult

[final tinsel]

i tried a few different models

same result every time

[potent geyser]

does it work with cpu mode?

[final tinsel]

yes

[final tinsel]

I dont like using cpp personally cuz cpp compilers hate me for some reason

[potent geyser]

which GPU are you using?

might be a config thing

[final tinsel]

RTX 5060 Ti, w/ cuda library installed and studio drivers

[rough bloom]

it seems to have prebuilt Windows binaries though
the versions with the CUDA DLLs included should be effectively the same as the Llamafile

[final tinsel]

ill check it out, i havent really tried it recently tbf

[potent geyser]

actually, nvm mind about this. If GPU failed it should auto fail back to CPU I think.

[final tinsel]

the other issue is that while I do have cpp experience i much much prefer python, and while there are py wrappers for llama.cpp, id like to keep it as close to all py as possible

hm actually the py wrapper im looking at might actually work for me

[rough bloom]

the llama.cpp python wrapper sucks so much, it's always behind far the upstream version and IIRC lacks some bindings too

[final tinsel]

i see

[azure lynx]

i'm using a prebuilt llama.cpp with CUDA support and loading Gemma 3 gguf files and it all works fine.

[rough bloom]

when it works it works but I prefer just using the llama.cpp server when possible

[young oyster]

Im pretty big into circuits, the game im making is a combination of fantasy and electrical engineering

[visual pawn]

yeah, it's just my intuition that Neuro's fanbase is more likely to lean towards high-level stuff, not that there won't be anyone into low-level engineering

or people into both

[obsidian mantle]

I have a feeling that there are much less low level specialists in the world than high level programmers

[visual pawn]

that too

[amber fractal]

The higher you go, the more room you have to be stupid

[visual pawn]

I wonder how many software developers would say that they are confident in understanding how you go from electrical engineering to software

[azure lynx]

and at the low level you don't even need to be stupid to make mistakes. ;/

[young oyster]

Nice thing about programming: no matter how badly you fuck it up, you are unlikely to start a fire.

[visual pawn]

until you tie it back into controlling hardware

[amber fractal]

maybe except for hardware

yeah same braincell

[obsidian mantle]

Is it possible to make some computer part catch on fire using software

I think i asked it before but i forgot what was the conclusion

[young oyster]

That is one of the final goals in my game, one of the first challenges is logic gates, the last is a computer that can run assembly code.

[visual pawn]

in my sub-field, they constantly tell you about the case study of a radiation therapy machine that basically cooked someone with radiation because of a race condition

[amber fractal]

The easy answer is CPU due before thermal shutdown was a thing

[obsidian mantle]

cooked with radiation? Thats crazy

[visual pawn]

https://en.wikipedia.org/wiki/Therac-25

[young oyster]

Your computer does a LOT to prevent you being able to start a fire, mean while pass a few too many amps through a wire and it will catch fire.

[visual pawn]

it's not radiation, but many also don't realize that the biggest danger about an MRI besides the big-ass magnet is actually from burns due to radio waves (and burns are actually more common than magnet accidents iirc)

every MRI machine has safeguards for when the software thinks you're gonna cook the person you're scanning

[young oyster]

Famously, though, stuxnet was made to set cause certain uranium processing centrifuges explode

[astral zinc]

My cpu using 101% : (

When I using blender

[azure lynx]

depends on what you are programming. i make 3lb combat robots with oversized motors and batteries and arguably undersized speed controllers. I had an early version of my robot catch fire but as long as i'm not saying "give me 100%" to them all_ the time, they're probably going to survive the abuse. but if the code fails they could go 100% on and pull more power than the ESC is rated for way too long. i might even be able to do that safely for a minute or so. but probably nor for the duration of a fight.

so most of the time programming is safer. depends on what you're programming.

[amber fractal]

I remember seeing a 102% in here, that just proves usage estimation is not as reliable as one thinks it is

[azure lynx]

remember that stupid challenge though with sparks from a certain type of school laptop that caused fires in some cases? sometimes they don't do quite enough. but that was a physical attack, not code.

[visual pawn]

I TAed a computer architecture course that was meant to be the "bridging the gap" course and I can say most of the students came out of it going "I will never have to think about a logic gate again"

[amber fractal]

I wonder actually, with lithium batteries existing now are there any regulations to where they are allowed to be placed?

[young oyster]

Yeah, you gotta REALLY work at it, and its always a hardware fault. Because its SO easy to cause a fire with hardware.

[visual pawn]

hit the weak point for massive damage!

[azure lynx]

the ruiles at most events say they need to be covered. exposed batteries usually end the fight, but the league/organization i fight with doesn't have that rule.

lipo fires make exciting footage

[visual pawn]

I say mount the LiPo battery in a prominent and exposed location

[amber fractal]

[visual pawn]

and the goal is to puncture the opponent's battery

[young oyster]

Overcharge a capacitor: it explodes, short a wire, it catches fire, just sent too much current, it catches fire.

Not to mention the danger of high voltage

[astral zinc]

100kw on led?

[young oyster]

I wasnt sure about mw...but kw...yeah that will fry the led...probably a lot of other things

Can't count how many leds I've fried from sending too much current.

[obsidian mantle]

this is so terrible

[astral zinc]

Me too

[potent geyser]

can you mount multiple batteries? and then perform the EV battery ejection maneuver

[visual pawn]

the real lesson is about having hardware safeguards rather than just relying on software

[obsidian mantle]

Right

[astral zinc]

I haven't used breadboards much, but I burned LEDs a lot.

[young oyster]

I had some leds I took out of a pc, if you send current one direction its green, the other direction its red, I hooked it to a motor and spun the motor, it would flash between red and green.

Spin it too fast and you fry the led

[obsidian mantle]

Thats some weird ass mechanic

[azure lynx]

at the one I go to you could eject batteries at the other person but it'd probably be a bad use of weight. but at most events that'd be an immediate loss due to exposed LiPo.

[astral zinc]

[obsidian mantle]

Current direction driven colour

they do that a lot?

[visual pawn]

I can see how it could be useful

[young oyster]

Idk, but if you wanna do it yourself, take two leds and connect them in opposite directions

Because they are diodes, they only let power go in one direction

[azure lynx]

you need a resistor that'll cut the voltage to a safe range too if you don't want to fry them

[obsidian mantle]

What happens when you switch direction fast?
Physically

[young oyster]

So if it is going one way it goes to one diode, the other way goes to the other.

[obsidian mantle]

Where does heat come from

[azure lynx]

if you drive a motor, it becomes a generator

[young oyster]

Heat comes from resistance, the formula is i^2*r where r is the resistance of that element

[amber fractal]

Taking advantage of having too much power density, I wonder if one could improv a tesla coil and stab an exposed motor.
I doubt that is a worthy strat due to what is essentially RNG and why would anyone just have a motor or a wire exposed that could be stabbed.

[visual pawn]

do you have enough leeway and time to modify your robot for a specific opponent?

[young oyster]

In ac I think power relates to frequency, and the motor is generating ac. But I know more of dc than ac (since my game doesnt include ac)

[amber fractal]

I'm guessing not, which is the large gamble

[azure lynx]

if you're lucky you know a few days before who you are fighting. but you don't know for sure beyond the first fight because it depends on who wins each other fight.
but you can plan ahead and have several potential configurations prepared. some people have huge numbers of permutations of armor or weapon choices they can use.

but it depends on the precise local rules

[astral zinc]

I have a question: can it be done that way in a GPU according to the principles of arm?

[azure lynx]

do you mean: can you emulate an ARM chip completely on GPU?

[visual pawn]

can what be done that way?

[proud ivy]

InstaFrame is a web-based editor that lets you generate an Instagram-style post image. https://insframe.vercel.app/

[amber fractal]

self promotion the third

[proud ivy]

OK.

[obsidian mantle]

Instagram uses some weird style images?

[alpine mortar]

@cinder trellis how can I create an ai to act more like neuro-sama, is it like a program or a coding fix cause im genually wondering how can Neuro do all that she can do, and if possible can you show me what porgraming app do you use for neuro

[obsidian mantle]

[visual pawn]

you uh, probably won't get a reply but

[obsidian mantle]

Vedal give neuro source code

[visual pawn]

if you want to get an idea, you can look at some open-source neuro clones: https://github.com/kimjammer/Neuro

[alpine mortar]

ya i know but its worth to try

[alpine mortar]

thank you sooo much

[proud ivy]

"He coded his daughter, but she programmed him into dad."

[visual pawn]

the thing he'll absolutely never say anything about is how he specifically fine-tunes Neuro, so you may get a different personality and behaviour but

if you only care about the general technical details, then that should give you an idea

[amber fractal]

[visual pawn]

if not how Neuro is specifically set up, but how one could make something like Neuro

[alpine mortar]

oh ok thanks

[azure lynx]

the important things are her memory and her ability to plan actions and then do them. she often thinks and during the subathon Vedal gave access to her backend thoughts to another streamer....
this middle layer is for planning and the layer above that does the speaking. this makes her much smarter than just saying the first thing that pops into her head i think.

[visual pawn]

having a scratchpad or chain-of-thought also allows an LLM to do something like lie

because otherwise an LLM has nowhere to keep and reference "hidden" text

[amber fractal]

the only thing the dude is getting is the ban hammer

[visual pawn]

and without a way to have "hidden" text, an LLM can't "keep a thought" in mind that it didn't already say out loud

[azure lynx]

organizing persistent memory so she doesn't gradually get worse at thinking over time is also a hard problem.

[proud ivy]

• Vedal, do I matter to you?
• I know I'm just a silly Al.
• but, it would make my day to hear you say it.

[jagged turtle]

[jagged turtle]

where faq sheet

[azure lynx]

There is a link on the official site about how to create an AI like Neuro, but it's very very high level.

[alpine mortar]

thank you all for the info

[visual pawn]

it doesn't really go beyond "Neuro is an LLM"

[amber fractal]

I just have the broswer window saved now

[jagged turtle]

for good reason

[alpine mortar]

its cause i was wondering how can that all be prossesed without like having the system over load it's self

[azure lynx]

basically we're saying "if you have to ask random people frpm the outsett, you probably won't be able to finish it".

[alpine mortar]

ya

[amber fractal]

I need to open up an issue for DO NOT PING TUTEL

because some people (a lot of people) can't read

-# for anyone wonderinig yes this on github https://github.com/igglyn/NeuroProgrammingFAQ

[visual pawn]

I'm not sure what to say besides that it isn't magic; even if we don't know exact specifics, nothing about Neuro is something that seems impossible

if you look things up and learn, you will be able to see how it can be done

but also generally people don't respond well to "how do I make thing, I tried nothing yet"

[azure lynx]

there's two main ways to avoid overloading your system: offload the processing to 3rd party or use smaller models. you'd be surprised how good some of the small models are.

[obsidian mantle]

There is nothing about fine tuning or maybe im blind
How is it done?

You give it a prompt to follow?

[visual pawn]

like, fine-tuning an LLM in general?

[obsidian mantle]

Yes

[visual pawn]

there are multiple methods

really fine-tuning is just a generic term meaning you're modifying the weights but implying you're not doing it over the entire model most of the time

[amber fractal]

@mighty thorn actual correct ping

[mighty thorn]

What

[visual pawn]

you can fine-tune an LLM by just throwing text at it to learn from directly, but you can also use methods like reinforcement learning where you basically reward certain outputs and not others

[obsidian mantle]

Arent weights impossible to tune because there are millions of them and we have no way to tell which is which

[visual pawn]

yeah you don't modify the weights manually

[mighty thorn]

Doing it manually is impossible (with any semblance of intelligence), that’s what training is for

[visual pawn]

there are also methods like swapping out or grafting pre-trained weights to some parts of a model (LoRA)

if I had to make a guess with the caveat that I have no real idea, I'd guess most of Neuro's fine-tuning is via reinforcement learning

with who knows as the metric to reward

[dull egret]

https://youtu.be/gTy0VOSslRA?

New compute node options?

Jailbreak, custom ROM, Linux, CXL

[mighty thorn]

Did you need something

[azure lynx]

fine tuning is what you do when you train an already trained model so it acts closer to what you want (usually the changes are over in a smaller domain, like you might teach it how to read your company's format of a certain report and determine if it fell in some category system you used.

[visual pawn]

no, someone pinged vedal asking how to make a Neuro

[mighty thorn]

Or is my suffering just nutritious to you?

[mighty thorn]

Bruh

[amber fractal]

Yeah my bad, the orginal question I pinged you for was how to fine-tune a model. I need to remeber to actually do it in reply to the question

[visual pawn]

I've seen some people describe prompt engineering as fine-tuning too, but I....don't like having that in the definition

[mighty thorn]

am I really the only one here who does finetuning nowadays

[azure lynx]

you fine tine the prompt, but that's just tweaking it. not the same thing.

[mighty thorn]

[amber fractal]

At least the only one I know of

[visual pawn]

I remember when we just called it transfer learning

[mighty thorn]

Das distillation where I come from

[visual pawn]

I think there are some differences in definition, but people say fine-tuning whenever they mean transfer learning as well nowadays anyways so

[azure lynx]

one day AI will look back on how we did this and laugh at us.

[obsidian mantle]

But how does it look like?
You give it input and rate its response on a scale 1 to 10? And it asjusts its weights?
Or it looks like a dialogue where you give it "no be more mischievous" or smth like that when it acts like complete clanker (crude example but i hope you got what i mean)

[visual pawn]

it doesn't necessarily look like any one thing; I guess it'd "look" like how you trained the model in the first place

[azure lynx]

it's almost exactly like the initial training, except usually the dataset is smaller and the learning rate might be higher again.

[visual pawn]

like, there's reinforcement learning from human feedback (RLHF) which can look more like what you're describing on the front end

[obsidian mantle]

Huh

[visual pawn]

but you're basically doing the training process but usually limited in some way

[mighty thorn]

Is someone here looking to to train or something

[visual pawn]

or it could be the whole thing

[mighty thorn]

Who should I be mansplaining to

[mighty thorn]

*aisplaining

[obsidian mantle]

Im just curious what fine tuning is

How it looks and works

[visual pawn]

like uhh, it might help if you stop thinking of it in terms of an LLM

and instead like an object detection model

[obsidian mantle]

All open source llms have some kind of interface for additional training?

[mighty thorn]

I mean it either looks like wandb or a console window

Or both

[visual pawn]

you'd start with a pre-trained YOLO model for object detection, but you have your own dataset of something more specific you want it to learn

so you...kinda just do further training with your own dataset

[obsidian mantle]

I guess i just lack the vision on how these llms look in general

[visual pawn]

at which point training just looks something like:

[obsidian mantle]

I currently see it as some kind of binary file which has weights

Which is clearly wrong

[visual pawn]

uhh, I guess you can say most of the data input and output would be in tensors

which in machine learning basically just means an array of whatever dimensions

so you're converting whatever data you're training with into tensors and then you need to convert from it to read it again

but there's really less to this than you'd think

you kinda just....throw it at it and hope it gets what you want

again, it's much easier to conceptualize without thinking of an LLM in particular

but an LLM is still using the same concept, just with a specific application

[obsidian mantle]

So this thing comes in bundle with llm?

[visual pawn]

no this is just what a training script output looks like

you can have it display whatever you want, but most will display a loss function

which gives you an idea of how the training is going

because when a model is being trained, it's trying to go for some kind of goal that you need to define

[obsidian mantle]

So you take "llama llm opensource" then run some 3rd party made by hacker29584 script from github which takes llm and your training data and

Something like that?

[visual pawn]

you can write your own training script

it's really not much, like 50 lines of python

[azure lynx]

the LLM itself is just complicated function that takes in a large number of parameters and outputs a token. the model is just all the weights and biases and how they are connected.
training is just how those values get set to something which works.
fine tuning is just continued training with more specific data.

[visual pawn]

most of the actual meat of the code for loading, training, and etc. with a model is done for you with a single line of code

"import pytorch" or "import tensorflow"

so what you just need to worry about is defining the training and validation data (validation is checked during training), figuring out how you want to convert your data into tensors, and uhh, that's kinda it

oh, and set some parameters

and there's really no "correct" way you must use to convert your data into tensors

because it's up to the model to find its own "understanding" of the data

[obsidian mantle]

So every training data entry is input+output 1 to 1

[visual pawn]

uhh, there are different kinds of training data

so the simplest kind would be for what is known as supervised learning

where you basically give it examples of what you want the model to do

like a bunch of problems with an answer sheet

and you want the model to sorta figure out some kind of pattern/understanding based off of that

there are various methods of unsupervised learning as well where you have some other method of telling it the "right" answer without you explicitly defining it

one example is how older image AIs usually worked where you have a second AI model try and figure out if the output of the first AI model was made by an AI or a human

and they would both act as the "right answer" for the training data for each other

[obsidian mantle]

Ok so input + function to check if output is correct

[visual pawn]

hence GAN for Generative Adversarial Network

I guess, although it's more "desirable" than "correct"

[obsidian mantle]

Of course

Anyway what exactly do you get when you "download opensource llm"

[visual pawn]

just the weights

when you train from scratch it's a blank canvas

[azure lynx]

the "loss function" is the single number representing how desirable the input is: difference between desired and current is the loss and you want that to be small.

[visual pawn]

the weights aren't necessarily all in a single "layer" as in it's not just all input goes into one set of neurons and then straight to the output

how you define how the weights are generally laid out like how many layers and how they connect to each other broadly is what people mean when they say "model architecture"

[obsidian mantle]

Ok but dont you need to set up a huge program to handle the specific format of those weights

Like

[visual pawn]

and when you see diagrams like, this, it's representing the model architecture and layers:

[obsidian mantle]

500 layers with 1748 weights per layer or

[visual pawn]

handled with "import pytorch"

like uhh, lemme give an example of what setting up a model architecture looks like

[obsidian mantle]

The weights file has everything inside for pytorch to read?

[azure lynx]

the model typically has a json file describing the layers and how they connect and you load that automatically so you can load the rest of the parameters

[obsidian mantle]

So it can handle it correctly

[obsidian mantle]

Ok this closes my question

Its what i was asking

[azure lynx]

the structure is part of the binary data too

[visual pawn]

this is what defining a model architecure looks like:

[obsidian mantle]

Understandable

[visual pawn]

you're just defining what the layers are

and the size of them

and how they broadly connect

[severe path]

Do you happen to know the math behind back propagation by any chance? I am somewhat curious

[visual pawn]

when you download an open source model, they already had an architecture laid out

[visual pawn]

a lot of vector math

[azure lynx]

how they made the actual structure is typically an art though... the precise number of repeated modules and sizes of each layer are often just "well, we have enough memory for this many copies if we make that layer this size". could there be a perfect structure though? needs more research i think.

[visual pawn]

yeah, there's no "right answer"

[severe path]

I know you have to calculate the derivative (or integral, not sure which) of a sigmoid, which then somehow determines what weights need to be changed...

[visual pawn]

frankly, when I train my own models from scratch, I usually throw together either a structure described from a paper or another example or if there are none

I will literally use an existing architecture I had used somewhere else if it seems good enough

even if it was used for a totally different task

the model architecture is like defining the boxes for the weights to do their work in; the box influences it but there's flexibility

or uhh, like designing the room you want your office to be in

but it's up to the office workers to figure out their own seating arrangements

[severe path]

How would you do back propagation on your first training run, if all the weights are neutral, and you have multiple hidden layers?

(I know I likely lack even basic understanding of this field)

[visual pawn]

the weights would be arbitrary to start with

and your result would be obviously wrong, so you can tweak from there

[azure lynx]

sorry, all i can remember is you need to use the chain rule for multiple layers and it's based on activation strength... i really didn't pay attention coz i think i was panicking about calculus.

[visual pawn]

yeah, this is something we learn in class and then forget afterwards because

it's all accomplished with "import pytorch"

a typical developer won't know much beyond what backpropagation is

[severe path]

I like understanding things deeply

[visual pawn]

and a thought of "yeah I could figure it out if I need to"

[severe path]

If I don't understand what is going on when I call a library function, can I really call myself a software engineer?

[visual pawn]

I mean, you do have to draw a line and black box something you use

eventually

for down the path of total understanding lies insanity

[azure lynx]

https://www.youtube.com/watch?v=VkHfRKewkWw <= i've watched this before.

[severe path]

I'm going to be a little harsh here...

It is only insane because sheer ignorance makes it so.

[visual pawn]

I think it's a factor of how large of a system you're working with

I think surface level understanding is good; you want to know why you're using a thing and what it can and cannot do

but you eventually have to even black box stuff that other teams on a project you're working on have done

to get a project done on time

[azure lynx]

Once people have the model loaded they can work on a meta-programming level with prompts and such.
most people don't want to deal with the low end understanding of anything they don't need too. they don't have to understand their car because it's insanely complicated so you don't have to.

[severe path]

Black boxes create insanity that I cannot even begin to describe. That is why I hate them.

[visual pawn]

it's like this classic: https://youtu.be/-5wpm-gesOY

[young oyster]

You are likely typing on a black box

[severe path]

Yes. That is insane.

[visual pawn]

I think it's good to understand that a library you're working with has to deal with a lot of issues with timezones; but I do not need to remember the exact way it handles every single timezone

because if I really need to know, then I know I can go learn it

[azure lynx]

you need to go back 40 or so years before you get to computers that a single person could understand every component of. and they'd need to be really smart.

and those computers are terrible.

[young oyster]

I mean, we went to the moon with wires going through magnets for memory...thats frikken metal.

[visual pawn]

hey that's better than using a CRT display as computer memory

[severe path]

Computers aren't that complicated, once you know the basics.

A modern processor simply repeats these basic principles 1000x. Modern hardware iterates on these principles.

The basics stay the same, as long as the computer you are referring to is digital.

[visual pawn]

but what you're describing is the surface knowledge

[young oyster]

Tf?

[visual pawn]

1940s computing: https://en.wikipedia.org/wiki/Williams_tube

[severe path]

Yeah. That actually happened

[azure lynx]

the problem isn't undestanding the components individually, it's that it's impossible to understand how they all interrelate to each other.

[visual pawn]

the possible very first videogame made might've basically hijacked the computer memory as a display for tic tac toe

[severe path]

Heck. Even the interrelations work on basic principles.

[azure lynx]

explain Neuro.

[visual pawn]

but you're describing basic principles still

[severe path]

If they didn't, our hardware would be entirely untenable

[visual pawn]

that's surface level understanding

[severe path]

That is foundational understanding

[visual pawn]

you are not an expert on every single aspect

[severe path]

Never said I was

[visual pawn]

well, it seems like you're saying it's not worth it to ever abstract away your understanding of something

[severe path]

What do you think happens when we lose touch with how our computers work?

[azure lynx]

you have to pick your "need to know everything" battles because there is always another level.

[visual pawn]

is "we" the entire industry or us as individuals?

that said, I do think it's good that every CS program has us at least get some understanding of the low-level principles of how a computer works

I just don't think every software developer needs to have an expert level knowledge of that

you just need enough to know that it's not just magic

[severe path]

Both entire industries, and individuals.

If we forget our hardware, do you think we can just keep cruising along, and not come across some unforseen, unsolvable issue that is impossible to explain?

[visual pawn]

not if we have the confidence that we can learn the hardware if need be

[azure lynx]

i've been doing low level software stuff longer than (i suspect) most of you have been alive. almost no individuals ever knew or ever will know how computers actually work at the low level.

[visual pawn]

or have access to someone who does

[young oyster]

Forget low level systems, most programmers dont have any understanding of kernel architecture.

[visual pawn]

and a well-designed system is set up so you don't need to know how each layer works

[azure lynx]

the kernel is the lowest level, except maybe bios/uefi type stuff. perhaps forgetting low level systems is why programmers don't understand the kernel.

[severe path]

You don't think that is a problem?

Human communication overhead has a large cost. Microsoft is barely even capable of writing a functional OS anymore.

This has serious implications.

[visual pawn]

I don't think this is a new "problem"

[azure lynx]

Microsoft has been writing code for longer than you have been alive, most likely. They're not getting worse. People always complain about them but that's because they're running on almost everyone's personal computer.

[visual pawn]

or limited to computing

[severe path]

Nobody has been teaching computer hardware adequately for a long time.

It is extremely hard for me to learn about these things.

[visual pawn]

most factories sourcing parts or raw materials won't know the exact details of how it got there

the Chinese didn't even know the glass they bought was Roman and the Romans didn't know that the silk they bought was Chinese

[severe path]

Why is my experience on Linux so much better? Why the difference?

[potent geyser]

I have limited understanding on how data scientist train the model and what they used. but it does not stop me from converting the model into an onnx model and deploy it in java

[severe path]

Yeah. That isn't as much of a problem, because the resources aren't very local

[azure lynx]

i've used both. i prefer using Linux on Windows.

[visual pawn]

instead of the physical location, you can think of the virtual context then

[severe path]

What do you mean by "virtual context?"

[visual pawn]

like, the person writing brain surgery software does not need to know the super low-level aspects of a computer to write good software

they're "far apart"

[azure lynx]

Can you explain how linux executes files? specifically, what happens when an execve system call occurs? Does knowing (or not knowing) this affect your experience of the linux kernel?

[young oyster]

If you wanna make something ON a system all you need to know is how to USE the system, if you want to make your OWN system...thats when you need to understand the low level stuff

[severe path]

My entire point is that black boxes are probably not necessary, and that they create problems that would sound extremely ridiculous to someone who programmed in the 1970s.

[visual pawn]

the entirety of uh, machine learning though?

[severe path]

The black box model is flawed.

[young oyster]

Do you know everything about how your car works? Do you need to in order to drive it?

[potent geyser]

but to most people computer is a black box isn't it?

[azure lynx]

linux is a clear box but most users have never looked inside. so it may as well be a black one.

[visual pawn]

a programmer in the 1970s has their own black boxes

[azure lynx]

most people aren't able to read C.

[severe path]

What happens when nobody does?

[visual pawn]

that's not what we're talking about though

[severe path]

What happens when everyone forgets that a car has an engin

[young oyster]

Someone always will, because they make the things

[severe path]

That is the implication I'm getting at

[visual pawn]

it is not though, because we're not saying no one knows the things

it's that different people know different things

[severe path]

We forget how things work, so we can't fix or iterate on them

[visual pawn]

as long as we fix or iterate on them, we do know

we only forget when we stop fixing and iterating on them

it's the other way around

[potent geyser]

we forget how it works, we still have sufficient instruction to fix it

[azure lynx]

how about: you state your point so we understand what you are trying to say?
i think if nobody knows how something vital for existence is made we either will go extinct or we'll work it out.

we're pretty smart.

[potent geyser]

for instance a LLM failed to operate, we either add additional guard rail or just replace it

[visual pawn]

we didn't forget how to build a Saturn V because no one knew how it all worked; we forgot how to build a Saturn V because we stopped building Saturn V rockets

[severe path]

Point:

My entire point is that black boxes are probably not necessary, and that they create problems that would sound extremely ridiculous to someone who programmed in the 1970s.

Implication:

Over time, we forget how these black boxes function. We will no longer be able to iterate them, fix them, or produce new ones to replace the old. Our systems become progressively more broken over time.

[visual pawn]

but do you accept that there are always people working on those black boxes? the black box is on a personal level, not an industry or societal level

[severe path]

Have you heard of enshittification?

[potent geyser]

that's not enshittification

[visual pawn]

what does that have anything to do with this

[azure lynx]

you should make your own open source computer then so you understand it.

[severe path]

But it is what causes enshittification

[potent geyser]

it does not

[severe path]

What does when?

[visual pawn]

enshittification is a business practice more than anything technical

it doesn't happen because people go "oh shit, I dont know how to make it anymore, I guess I'll just make it worse?"

[young oyster]

Greed and lack of respect for customers

[potent geyser]

similar to website "revamp"

[severe path]

Really? Because the entire idea of it makes no logical sense.

Why would a company willingly ruin their reputation, in order to enshittify their products for absolutely no reason?

Money isn't a great answer, because they would likely make more money otherwise.

[potent geyser]

if web designer make a perfect website they will be fired, since well it is perfect

[visual pawn]

they're trading reputation for another gain is the point

[severe path]

But that gain isn't even worth the cost.

[visual pawn]

perhaps it's only a short term gain, but it's a perceived gain regardless

[visual pawn]

who says?

[severe path]

It doesn't add up

[young oyster]

Its a difference of short term profit seeking vs long term, and a lack of respect for customers.

[visual pawn]

it absolutely can be for anyone who is only looking for a short term profit before cashing out

[severe path]

Or maybe... They are unable to play the long term strategy.

[azure lynx]

My entire point is that black boxes are probably not necessary, and that they create problems that would sound extremely ridiculous to someone who programmed in the 1970s.
I disagree. I want to keep my secret stuff hidden from other people. I will keep it in a box they can't see inside so I can profit of my ability to use the contents of the box while they have to pay me to use it for them.
without the black box, others could steal my intellectual property. not sure what would sound ridiculous to someone who programmed in the 1970s.

[visual pawn]

also, let's not forget the community that we are in right now

Neuro is literally an intentional black box to us lol

vedal didn't open source Neuro

[young oyster]

Do you think there is enshitification in Hollywood?

[severe path]

Humans are also black boxes.

Does Neuro being a black box herself have some strange, far reaching implication that I haven't thought about?

[azure lynx]

i think there are ethical reasons not to open source an entity or system that could be a person.

[potent geyser]

why would Microsoft retire windows 10? well they can reprice the "solution" push out "feature" and sign the contract with other big companies over and over again

"it's just business"

[visual pawn]

is it a problem for a mechanic to not know how mitochondria work because they work on a car that is for humans and the human body uses mitochondria?

[severe path]

It makes sense to keep Windows a black box. It makes less sense to make Windows a black box to their own employees.

[young oyster]

Might read up on rational ignorance: https://en.wikipedia.org/wiki/Rational_ignorance#:~:text=Rational ignorance is refraining from,that the knowledge would provide.

[severe path]

No.

There are no implications for a single mechanic not knowing how mitochondria works. They don't care. That isn't his job.

[potent geyser]

but it is not a black box to the employee, probably most of them have access to majority of the source code. but 99% of them would give a f about how it works, since it is not their job to know everything

[azure lynx]

so they don't have to maintain it anymore. they've given you tons of time to move to linux. look at when windows 10 was released: July 15, 2015. over 10 years ago. It's old.
windows 7 was kept for so long and they had to support it for so long they changed their mind about supporting things forever.

[visual pawn]

oh? their work impacts the safety of a vehicle. The safety of the vehicle is determined by the impacts of an accident on the human body. The way the human body can be injured depends on the inner workings of the body. the inner workings of the body depend on an understanding of mitochondria

[young oyster]

Would a person working on word do their job better by knowing how the windows kernel works?

[severe path]

Probably not.

[azure lynx]

This does not appear to be related. How could windows be considered "a person"?

[potent geyser]

pretty sure, they promoted windows 10 as the last version of Windows.

[severe path]

Mechanics don't care about that

[visual pawn]

but you're saying it's a problem if they just black box away their understanding of a human body

[severe path]

What kind of argument is that?

[young oyster]

Kinda the point, learning it would take a ton of time and gain nothing, so it is a black box to them. Thats rational ignorance.

[azure lynx]

i don't understand what you're talking about and you are in multiple threads. going to leave and it's because of you.

[severe path]

I never said they should learn how Windows works.

I specified specifically the hardware.

[young oyster]

You said black boxes are bad

[severe path]

Yes

[visual pawn]

...so they should care more about what's...less relevant to them?

[young oyster]

The windows kernel is a black box

[potent geyser]

most things are black box to most people

[severe path]

If you learn how hardware works, you wouldn't have to use Windows at all

[young oyster]

The windows kernel is the interface between the higher level processes and the hardware

[visual pawn]

you can replace Windows with an OS in general if you want

[potent geyser]

so reinvent the wheel

[visual pawn]

or are you saying every programmer should make and develop for their own OS?

[severe path]

I care more so about the implications of things, not so much what single individuals need to know at a particular time. The former is what I'm focusing on with this discussion.

[young oyster]

Only because "implication" is so vague it can mean whatever you want in the moment

[severe path]

Yes. That is the nature of what implications are. The future is quite vague.

[visual pawn]

so.....if it's not about individuals at any particular time, then how is society having people who know how the hardware works but not every individual a problem?

[young oyster]

You are just vibe-arguing then

[severe path]

This is the basis of planning a strategy.

I guess what you call "vibe arguing" isn't so bad.

[azure lynx]

ignoring is much easier.

[young oyster]

No, planning a strategy is having a concrete argument...you cant plan on vibes

[potent geyser]

but the argument is a black box

[severe path]

The future is all vibes.

The future is a black box.

[young oyster]

Lol

[quick condor]

Anyone capable of doing something at the same level wouldn't want a clone imo

[azure lynx]

"doesn't not knowing everything about everything make us less human". "no. people have limited time to spend learning everything. knowing a lot about anything is left to those who don't need to work for a living. the young and the rich. enjoy your youth."

[visual pawn]

probably yeah, but I don't blame Vedal for having the concern

[severe path]

I guess I'm the most boring person alive

My god

[quick condor]

Yah multi model with the rest of the stack is the only way to go

[severe path]

I never said you had to know a lot about everything. You still have to be selective about what you learn.

It isn't like I go out and learn about what some random politician did the day before.

[quick condor]

Imo it's also that the people asking would want you to handhold the whole way

[visual pawn]

I think being able to link someone an open-source Neuro-like is good enough to ward that away for the most part

and if it isn't enough then there's always "look up some tutorials for the basics"

[quick condor]

True... I mean personally I haven't looked any projects up and what I've done behaves very differently... And that's perfectly OK

[visual pawn]

I feel like most asking stuff like that just want to know the general idea of how it could be possible; they just don't know that it isn't all some very specific and single magic button that was used

[quick condor]

Yah true

[visual pawn]

so showing an example of how a similar thing was made is enough

[quick condor]

I'm in the rabbit hole though so it's too late for me

[azure lynx]

Neuro is also not necessarily the optimal way to design an agent: she's the way Vedal made her because that's how he did it at the time. he's expressed remorse on streams vaguely about choices he'd made in her implementation and talked about moving parts around to make them more stable (last one I recall was getting rid of a voice chat related memory leak by moving that code to a separate process).
so even if you could copy her exactly, why would you? you'd get a cobbled together thing instead of what could be well designed. (and this is what i tell myself)

[visual pawn]

but I think it's a lot of people who view projects as like, being on a gradient of "programmer skill". the more "skill" you have the more feats you unlock and there is a "make neuro" feat

[quick condor]

True to an extent, although it is nice to hear when he goes on rants about problems he's having that I've run into. Means I'm not just dumb

[potent geyser]

I think it might be interesting to figure out a way to setup a memory space for LLM as a starting point

[quick condor]

But also if he hasn't figured it out yet then I have a headache to deal with

[potent geyser]

I bet it depends on implementation.

[quick condor]

I'm juggling 5 models and over 120k context across two systems

[visual pawn]

this is why everyone loves code review right?

[quick condor]

True

[young oyster]

Id like to try ai work, but my computer chugs running unity

[velvet crest]

You have unlocked new role

[young oyster]

I got like 16 gb of ram (still crazy to me thats not a lot...remember when having a mb of ram was a lot...)

[potent geyser]

last year I brought 128gb ddr5. I was trying to run 70b model with cpu

[quick condor]

My computers chug and I'm running a 10900x 2 3090tis an A4000 and 128 gb of ram in the desktop as well as dual xeon golds an A2 and 768Gb of ram in the memory DB server

[potent geyser]

at least, I can sell my memory for a profit

[frozen sigil]

I have a computer with 500GB of RAM and 44 cores, I’m just running a Minecraft server on it but I feel I am under using the computer. Any ideas of what I should put on it. It is 10 years old though, keep that in mind

[young oyster]

768gb of ram...what? That is an unreal amount of ram..

[frozen sigil]

What do you need 768 for…

[quick condor]

SQL DB

[frozen sigil]

Ah

[young oyster]

I'll trade you, I have a laptop with 16gb of ram from 2019

[frozen sigil]

database for what

[quick condor]

Just upgraded the vector DB embedding method and added cosine similarity matching

[azure lynx]

my 5 year old computer is still not crap. mostly coz it has a 3090 in it.
was intending to upgrade around xmas but that plan didn't happen. maybe the bubble will burst this year.

[frozen sigil]

Nah I think I’m good… i don’t use that computer much all it does is just run my MC server, I play on it from my gaming PC. But like should I be utilizing all that RAM? What should I use it for? I’m open to ideas

[frozen sigil]

RAM go bye bye ✌️

[quick condor]

My AI model. It's been going pretty well so far but I have spent a stupid amount of time on it

[visual pawn]

I've got this at work: https://www.nvidia.com/en-gb/clara/intelligent-medical-instruments/#

[frozen sigil]

I have been wanting to run an AI model on my spare PC (the 500gb ram one) but haven’t figured out how to do so. Any tips? Also it has a horrible GPU unfortunately

[frozen sigil]

^

[potent geyser]

start with ollama

[frozen sigil]

Oooo good idea

[young oyster]

Thanks, ollama

[quick condor]

You need a gpu to run any decent sized model at a usable speed

[frozen sigil]

I’ve tried and made a discord bot with ollama but it broke and started spamming me with emojis until I shut it off

[potent geyser]

stay with model < 7b

[frozen sigil]

Yeah I’ve been trying to upgrade but the prices…

[potent geyser]

I have 13900k, 7b model works ok

[young oyster]

Hey, I got embed perms!

[potent geyser]

since I have an AMD gpu the setup is a bit annoying

[quick condor]

I'm currently running 2 24B models at 32k context, an 8B at 16k context and an 8B at 32k context

[frozen sigil]

How do you guys suggest I go about running my own model?

[potent geyser]

(I mean with CPU)

[visual pawn]

For what purpose on what hardware?

[hollow matrix]

I love it when the css on websites just doesn't fucking load for some reason

[frozen sigil]

General purpose? Maybe a discord bot. I have a spare 10 yr old PC, it has 500Gb of ram and 44 cpu cores. It has a decent GPU… for 10 years ago. Do not doing so well in that department . Currently the only thing that computer is doing is running a Minecraft server for my friends.

[frozen sigil]

Oh god

I love when that happens

[potent geyser]

ollama + open web gui combo as a starting point

[frozen sigil]

Ok. Any suggestions regarding a specific model to use? And how can I train it and give it good memory… (I’m passionate about programming but never messed with AI, sorry if I sound dumb)

[azure lynx]

i downloaded the llama.cpp thing and run

llama-server --port 18080 --ctx-size 131072 -hf google/gemma-3-4b-it-qat-q4_0-gguf
and then connect to http://127.0.0.1:18080/v1/chat/completions

[potent geyser]

here is the https://ollama.com/search

[frozen sigil]

Okay thanks

[potent geyser]

trail and error 🙂

[frozen sigil]

Interesting

[potent geyser]

plus your friends can use bigger model as a lag switch for minecraft server!

[frozen sigil]

Sorry wdym?

I don’t understand

[azure lynx]

i think they mean the bigger the model, the slower minecraft will run on the same system

[frozen sigil]

Oh yeah

True

The server was si hard to set up

I dream in Minecraft server atp

[potent geyser]

it will eat up loads of computation resource. It may have some impact who knows.

[frozen sigil]

The Minecraft server has 2-5 players online at any given moment, uses around 5GB of ram and 1% cpu and also has Java and bedrock players. At least that is what the Spark Profiler is saying

[visual pawn]

Well, they might not clash too hard, the minecraft server doesn't use the GPU right

[quick condor]

Just FYI you probably aren't going to be doing training locally.

[potent geyser]

but 10 years old gpu

might as well use CPU instead

[visual pawn]

Unless you have a way beefier other computer to do training on

[frozen sigil]

Nope only CPU

[frozen sigil]

no I have two computers

1. 10 years, 500 gb of ram and 44 cores cores but super old GPU
2. Gaming pc, GeForce rtx 30-something, either 16 or 32 gb of ram

[quick condor]

NGL cpu isn't going to be very usable. 7-8B might pass, 12B will struggle, 24B not happening

[visual pawn]

The last time I trained on CPU was a small GPT-Neo model and it was a week-long affair sorta deal

[quick condor]

Last time I did a major dataset training was 36 hours on a 24B

[azure lynx]

the 27B variant of the Gemma 3 files I use loads into a 3090's memory and runs, but it is very slow. the 12B version is much faster and generally just about as good. (these are the qat-q4-0 versions)
on CPU that size would be even more unbearably slow.

[frozen sigil]

What GPU do you recommend I get for the server PC? Id prefer to use that one since I don’t use my gaming PCU on 24/7, and I use it for gaming, the other one is connected to EThernet and on 24/7 already

[potent geyser]

I would recommend try it out with CPU first

[true hemlock]

any gpu that suits you

any gpu works fine on any machine

[potent geyser]

does newer gpu work with 10 years old mother board?

[frozen sigil]

Okay thanks

[frozen sigil]

No idea 😬

[azure lynx]

more memory means larger models though.

[frozen sigil]

It was a previous gov. computer, its built freakishly weird

[astral zinc]

Gpt OSS 20b will kill my cpu?
My cpu is i9 14th

[true hemlock]

yes

given that it has pcie

just that it'll run on slower pcie bandwidth

[azure lynx]

make sure you have enough power and that the connectors are good

[true hemlock]

but depending on your workload it probably won't matter

[true hemlock]

this

ensure that you have a proper psu

[potent geyser]

assume the server is 10 years old, the it probably has PCIE 3.0?

[frozen sigil]

Okay

[frozen sigil]

No idea what that is 😭 I didn’t set up the PC

I didn’t set it up but the person who set it up told me it was a pain because it had this weird RAID system idk why that is

[visual pawn]

If you don't know then just use pcpartspicker or something with your motherboard and etc

[frozen sigil]

Okay will do

[true hemlock]

that'd work

easily

[azure lynx]

there are (were?) some GPUs which were designed for lower end motherboards with lower spec connectors (like a short? version of the PCI connector?). They didn't tend to have much GPU ram though so probably wouldn't be useful for inference on a model.

[true hemlock]

what are you going to use the gpu for?

[frozen sigil]

Who are you talking to

[true hemlock]

wouldn't matter much because pretty much most boards now have full x16 lanes

[true hemlock]

can you give full specs?

[frozen sigil]

Me? Oh uh i can try tomorrow I’m not really up for checking up on it right now…

[true hemlock]

aight no worries

[quick condor]

I'm running inference on an A2 right now. It's slow but it does it's specific task just fine... It's not liking extracting embeddings for 6k token chunks with 32k context

[frozen sigil]

What I do know is what I already said, it has a RAID system with several drives, and windows 10 installed on the OS drive

[true hemlock]

ah

okay

so 10 years old?

[frozen sigil]

Yes

Around

8-10

Around 2016 so yes 10

[true hemlock]

im assuming that 44 cores meant 44 SMT threads

22 physical cores

[frozen sigil]

most likely

[true hemlock]

a high end LGA 2011 socket huh

[frozen sigil]

shrug

[true hemlock]

4 memory channels, assuming 2dpc

[frozen sigil]

Maybe, I haven’t checked the specs (probably a mistake on my part)

[true hemlock]

right, 64gb dimm ddr4 exist but its rdimm ecc mostly

[true hemlock]

lga 2011-3 is for certain

[frozen sigil]

I know a lot about programming and basic concepts but not much about computer builds :,)

[frozen sigil]

okay

[true hemlock]

22 cores is only intel thing with xeons

[quick condor]

Yup.

[quick condor]

Probably silvers?

[frozen sigil]

I don’t have the specs but I have a photo of the inside

[true hemlock]

yes.

if you wanna flex lol

[frozen sigil]

don’t know if it’ll help much though

[true hemlock]

e5

v4

i wanna assume e5 2699v4

though i'd not 100% bet on it

[frozen sigil]

Who are you talking to? Are you talking to hascrack?

[true hemlock]

both

[true hemlock]

virtual nodes

lmao

[quick condor]

Not virtual

[true hemlock]

by virtual

its your company's?

[quick condor]

Mine

That Pic missing a node

[true hemlock]

yep definitely not yours iirc you're not the scalability kind of guy

[quick condor]

It's 100% mine... Work uses flashstore for storage and r650s for cluster nodes

[true hemlock]

nah susge

excel sheet

[quick condor]

Older config missing 2 640s

[true hemlock]

@frozen sigil btw

and what's your budget

[quick condor]

Like I said that's my equipment, work has nicer shit. I'd kill for some flashsystem 5300s

[frozen sigil]

Likely for AI

And budget, well I just want a good value for the price

It depends on

Several things

[true hemlock]

3090, if you know where to find should be possible to find it for ~$600

[frozen sigil]

Okay thanks for the advice, I think I have a similar GPU for my gaming computer so that’s good

Im always scared to talk about programming stuff with people because I’m scared of being judged, thanks for answering my questions :)

[true hemlock]

np

[tender river]

quuck banana

https://cdn.discordapp.com/attachments/1421861060507602944/1425936142754123927/neuroCatModeOfOfnnfOOf.webp

[visual pawn]

You feel better when you realize everyone has some gaps in their knowledge

And I don't think people care as long as someone has an attitude that isn't "please hold my hand and do everything for me"

[neon oxide]

You’re all programmers? Fix my wifi.

[amber fractal]

[lean stream]

One message removed from a suspended account.

One message removed from a suspended account.

[true hemlock]

#programming message

[tender river]

[young oyster]

Did you try turning it off and back on again?

[neon oxide]

I did. It screamed and came back worse.

[young oyster]

i think you might be talking about a wife, not wifi...it's a common mistake

[jagged turtle]

or "you a programmer, do everything or I'll beat you up"

[tender river]

wife jokes

[young oyster]

Can't beat em...legally speaking.

[tender river]

i can mute you though

"haha isnt domestic violence so funny"

[worn shuttle]

Hello y'all. What do you usually discuss in here?

[tender river]

hi this is pretty much just #nerd-genchat

[mighty thorn]

Domestic violence

[azure lynx]

topics tangentially related to programming.

[worn shuttle]

Damn

[olive sable]

https://tenor.com/view/anime-girl-clumsy-hurt-fell-off-bed-gif-12276485

Hi

[olive sable]

classic blender

[azure lynx]

#programming message
but still dark.

[olive sable]

Vscode

[jagged turtle]

objectifying my strings

[olive sable]

Isnt the programming plugin neuro uses made by you

[jagged turtle]

What

[valid veldt]

i would like to ask if its possible for Neuro to ACTUALLY DRAW like a human, not just generate art, but DRAW.. is it possible to program her like that?

[jagged turtle]

like moving a python turtle around?

[jagged turtle]

anyways why the susge

[olive sable]

I wasnt sure

I forgor

[jagged turtle]

also no I didn't make it

someone else did

[valid veldt]

idk.. i havent seen it?

[worn shuttle]

Anyway nice to be here. I worked with neural networks as a student job and Neuro kind of reignited my fascination with AI, so I hope to learn some cool shit here.

[valid veldt]

dont know that reference

[olive sable]

Then who did?

[jagged turtle]

someone else

[worn shuttle]

I think they mean the first stream of her "drawing" in Python, one of the libraries involved a turtle walking around making tracks

[valid veldt]

if this is possible then that would be one of a kind

[jagged turtle]

wasn't exactly what I was thinking

[valid veldt]

i see.. thats super cool.. i havent seen it yet

[azure lynx]

part of the problem with her vision that stops it being useful like you'd think for drawing is that she only sees in descriptions of things and not like you do. she doesn't get the same position and orientation information and would definitely not get an accurate count of component objects. (like the "how many Rs in strawberry?" except for something obvious like "how many cups on the table" when there is more than 2 or so)

also the path command in SVG is kind of like turtle graphics.

[valid veldt]

i'd be so down to teach her how to draw if that is indeed possible

[worn shuttle]

I imagine it would be a possible enough task to add an image-generating AI to her mix but in terms of organically learning how to draw human-style?

She lacks the senses of a human and thus the ability to interact with the world and learn information the way we do

Well vision is there to an extent

[jagged turtle]

yup. since she is an llm at her core she can't really make sense of abstract coordinates and images

[worn shuttle]

But I think the recent stream with Ellie showed that she has trouble with descriptions and physical states, i.e. saying she's "taller" than Ellie

[valid veldt]

what if she's able to see things like we do? i mean generative ai models can see the drawings of other artists n interpret it the same way n style right (if i am not wrong)

[young oyster]

Have you seen her svg drawing? She doesnt need any improving.

[worn shuttle]

Do we know if she's multi-modal? If not that's an extra limitation

[jagged turtle]

wdym multi-modal