programming

[split steppe]

new integrated memory systems have high bandwidth, transfer rate and low latency as well

[fast pagoda]

what he posted is ye

[kind fable]

here you go
was little model trained on pictures

[idle dune]

125 Orange Pi node box

[split steppe]

not as high as a full dedicated gpu of course, but really quite good

[kind fable]

yes it's just iGPU of

[winged basin]

how old is the config?

[fast pagoda]

vulkan is quite good for radeon inference when compared to rocm for they kinda trade blows

[winged basin]

just curious

[kind fable]

it's a laptop which has a few years like nearly 5

[fast pagoda]

using vulkan backend i can ac tually split models across a 3080 and a 9700 pro somehow and it works better than either alone (not much better tho)

[split steppe]

people are allegedly running 120B models at 40tps on AMD AI MAX chips, which are fully integrated packages https://www.reddit.com/r/LocalLLaMA/comments/1ndoxxa/why_should_i_not_buy_an_amd_ai_max_395_128gb/

[fast pagoda]

this kind of perf is literally just fitting in memory vs not

or fitting more

even if it's slow mem

it's better than more of it being out of the mem pool for ccopies

[idle dune]

[fast pagoda]

which is the whole benefit to these apu packages over gpus with comparatively smaller memory available in pool

[obsidian mantle]

wtf its not even that big

[fast pagoda]

for 3500 id take a 5090 after traveling back in time to when thy were available for tha

[split steppe]

today's prices under a crunch lol

[obsidian mantle]

i expected some 10k+ shit

[idle dune]

That said, its not bad specs at all

[fast pagoda]

minis are graet deals

[nocturne olive]

It's just them pushing more memory bandwidth, nothing specific to the iGPU on those

[fast pagoda]

have been for a while now

[idle dune]

[obsidian mantle]

wait entire mega ai 128 gb mini computer is 3k?

[hard delta]

how do these minis cool

🤔

[idle dune]

I still wanna do my dumb ass workstation but thats a buy it for life purchase for me

[fast pagoda]

smol tdp

[split steppe]

yeah but you said "igpus can't run ML models" here you have an IGPU doing it

[hard delta]

is it arm

[split steppe]

and doing it really well

[obsidian mantle]

oh slow inference?

[fast pagoda]

smol tdp x86

[idle dune]

In reaily like ass, but they also has very low tdps.

[idle dune]

Nah this is the same TDP version as the Framewrok mini

[fast pagoda]

the 395+ maxes at like 120w

[obsidian mantle]

disassemble it and give it proper cooling?

[hard delta]

it's a mini

just keep it in your freezer

[fast pagoda]

they are just laptops

[nocturne olive]

I'm more guessing they're running it on the NPU rather than iGPU

[fast pagoda]

the ai max is running it on the igpu

which is a radeon 8060s

[nocturne olive]

Hem

[idle dune]

Yeah

[nocturne olive]

Weirdness

Eh whatever

[split steppe]

it's a unified system, it's all tied together in one chip, all compute and memory, it's probably everything

[idle dune]

It HAS npus but it has a dedeciated iGPU for inference

[nocturne olive]

Either way it's not gonna work anywhere near as well as a real GPU

[fast pagoda]

for the price it can generally be better

[idle dune]

https://giphy.com/gifs/goalposts-moving-goal-WwWTfQuCZ448psk1qB

[fast pagoda]

for inference only

in some situations

[split steppe]

yeah but that wasn't your point was it 😛

[opaque sigil]

lemme just buy a 96G gpu, how expensive could it be

[obsidian mantle]

so 3x4090 48gb modded will be better

[nocturne olive]

Either way I'll just focus on NeuroSynth (tm) vocal synthesizer technology

[nocturne olive]

Me needs
NeuroSynth hungry

[sage crag]

https://tenor.com/view/crash-out-bunny-rabbit-im-gif-10075768982376662002

[obsidian mantle]

amazon says 1 of those is 3k

[split steppe]

if you have spare crisp $15k to throw around

[obsidian mantle]

actually

naaah

[fast pagoda]

96gb ram

Available

To the gpu

[sage crag]

superbox try not to mention neurosynth, linux, or kotlin in 5 messages challenge

[split steppe]

remember that you need the rest of the system to run those

[hard delta]

impossible

[opaque sigil]

not gddr/hbm, doesn't count

[hard delta]

world peace any% speedrun

[split steppe]

$2k mobo, $2k cpu, $2k ram, $2k psu, $2k disks, etc

there goes your wallet

[hard delta]

if i could get that for my wallet i would

[fast pagoda]

that's what the rest of everything else in this goblin hoard room is for

[idle dune]

https://giphy.com/gifs/dexter-suspicious-aniflicks-21VTFJTEr1x9ortvO3

[fast pagoda]

heh heh

[split steppe]

nagatoro

[hard delta]

https://tenor.com/view/dont-toy-with-me-miss-nagatoro-hayase-hayase-nagatoro-anime-miss-nagatoro-gif-3194964525569635045

[fast pagoda]

do it for her

[split steppe]

don't, not this one

[fast pagoda]

im on a roll with self reports last few days

[obsidian mantle]

yeah idk if i can fit 3

[fast pagoda]

first domestic gf

then this

it's over

[obsidian mantle]

or does nvlink make pcie useless

[split steppe]

do it for takagi

[fast pagoda]

not the forehead

[idle dune]

Bro I asked Claude to make a spec list for a workstation to send real quick her enad the mfer is coding out a full document

[nocturne olive]

You do still need PCIe
Also no NVL on anything newer than 3090

[hard delta]

sir you am weeb

[idle dune]

https://giphy.com/gifs/cinema-absolute-absolutecinema-9EvnXdZaUZbCqScn67

[obsidian mantle]

im talking about theoretically buying 3x4090 48gb mod with nvlink (i will absolutely not do that)

[opaque sigil]

pcie is for connecting cpu and gpu, nvlink for gpu to gpu (unless grace hopper/blackwell )

[split steppe]

the forehead loves you, unlike nagatoro

[obsidian mantle]

so does it matter if i put 1 gpu into good pcie and 2 others in bad ones

but they are all nvlinked

or its bad

[opaque sigil]

if they're only talking to each other it doesn't matter

[nocturne olive]

There's no NVL on those still, they just removed NVL from the 40 series chips

[fast pagoda]

it wont help anything but youll probaby suffer much less

if it has to leave the gpus it will be slower

[hard delta]

also nvl was kinda slow

[obsidian mantle]

or would other 2 gpus be just like memory extensions?

oh

[nocturne olive]

Usually in training you don't have the GPUs talk over PCIe much anyway

[split steppe]

buy one AI specialized gpu

[obsidian mantle]

uhh so you cant even do that

unless there are modded 3090s

[hard delta]

anything you can do has already been cannibalized and has become expensive

[obsidian mantle]

car priced computer is insane

however

[idle dune]

I'm just gonna use a MI50/60 and a RX 9070

[obsidian mantle]

9k that cant even run 120bs

[fast pagoda]

wsure it can

you dont have to fit the whole model in mem to run a model

[obsidian mantle]

wait isnt 120b 120gb

[fast pagoda]

it's just gonna be a bit slower

[split steppe]

120b is not that big

[idle dune]

Qauntized or context quanitzed it probably can

[obsidian mantle]

ohhhhh wtf

[glass flower]

120b is like 80gb i think

[nocturne olive]

At Q8 with no context yes

[fast pagoda]

i can run 120b with 42 gb ovram

[nocturne olive]

At Q4 it's more like 60GB

[obsidian mantle]

is it super slow

wait actually what if i

try to run some big ones

i can try right?

[split steppe]

you could fully fit 122B qwen3.5 q5 https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF

[obsidian mantle]

how do i even know how much memory it needs then

[glass flower]

the biggest one i ran so far was a 80b q4 model and it ran at like 20 tks

[fast pagoda]

can be estimated

[opaque sigil]

buy the hardware and see, if it doesn't fit return it

[split steppe]

is it worth running the full-size models anyway

[glass flower]

yes. due to the knowledge density

[split steppe]

what about rag though

[obsidian mantle]

i have 100mbps internet is it really worth a try

wait its vision

i need llm

[glass flower]

that doesnt really help? i think. idk didnt do a rag yet

[split steppe]

qwen3.5 is multimodal vision+text

[glass flower]

but bigger models normally mean better

[obsidian mantle]

what happens if i try to feed it skyrim wall of context

[fast pagoda]

take the sqrt of the active * total params

[split steppe]

it'll tell you what's in it

[obsidian mantle]

no way its going to be usable

[fast pagoda]

abd that's a decent guess on dense model size

[split steppe]

qwen3.5 has 244k token context also

[nocturne olive]

Shared memory is a scam
It's as slow as a spinning hard drive and makes all rendering not work properly

[obsidian mantle]

yeah its what im thinking about

gemma3-12b starts with 2 seconds and then becomes 20 seconds after 40 minutes but i can clear context

how slow will be that 120b monster

[split steppe]

just try it man, try the 4B one https://huggingface.co/unsloth/Qwen3.5-4B-GGUF

[obsidian mantle]

if it started with 2 seconds for 12b

[glass flower]

9b is a better starting point tbh

[obsidian mantle]

i tried ollama's "deepseek" 8b before

it felt dumber than gemma

[nocturne olive]

If you don't have 96 or 128GB of RAM it just won't run probably

*on Windows

[glass flower]

i use 9b for my commit logs, autocomplete and general opencode agent if its a hard problem i use 35b with 25 experts offloaded

[obsidian mantle]

but @fast pagoda says this

[split steppe]

llama.cpp can technically run it through swapping but it just doesn't make sense on hw that isn't dedicated

[obsidian mantle]

yeah so its shit idea

[split steppe]

try the 9B and lower ones, one of them will run acceptably

[obsidian mantle]

i see

[split steppe]

i am running the 9B one on cpu only

[obsidian mantle]

is gwen 9b better than gemma3

[split steppe]

i do have 128GB ram and 64 cores

[obsidian mantle]

i thought

[glass flower]

i haven't ran gemma3. but i can guarantee it

[obsidian mantle]

more parameters = better

[split steppe]

dunno, never tried gemma, try and see

there's different measures of "better"

[obsidian mantle]

i'll google some benchmarks

[split steppe]

and it will purely be based on your needs

[glass flower]

qwen3.5 is really good as a model generally

[split steppe]

benchmarks are kind of meaningless for llms tbh

[glass flower]

don't look at benchmarks just try it honestly

its very much a vibe thing

[fast pagoda]

27b qwen3.5 is better than most of the larger moe ones

[split steppe]

it's all about your exact tasks you're doing

[nocturne olive]

You got 16GB of VRAM, you could run a 12B-16B parameter model pretty comfortably

[glass flower]

27b is as good as some of the 300b ones and even better than their own 35b one

but i can run 35b at like 100tk/s and 27b at like 20 tk/s so i picked the moe one

[split steppe]

the 27B has a different configuration than the rest though

it's harder to run than the 35B apparently

[glass flower]

its a dense model vs a moe

[fast pagoda]

it's just dense

.8/4/9/27 are all dense

[split steppe]

https://tenor.com/blFvI.gif

[glass flower]

i can't fit it completely into vram. either of them but 35b is i can offload better

[fast pagoda]

dense generally > moe at same param count

[split steppe]

maybe i should try the 35b for giggles

[glass flower]

why not

its very good

tho no qwen model can do rust

[split steppe]

well i dont have a gpu, so it's cpu only for me ha

i'm trying to find a nice middleground around 10-15tps

also waiting until llamacpp folks implement speculative decoding

[fast pagoda]

wat they have it in lmstudio

[glass flower]

i haven't gotten speculative decoding to really work yet in a sense.

[split steppe]

qwen3.5 specifically doesn't work due to its architecture

[fast pagoda]

oh yrsah

gonna be a min on that

[glass flower]

i just hope the next llamacpp update fixes the token processing. it takes like a minute to process the tokens before it even starts on the larger qwen models

[split steppe]

qwen3.5 also is supposed to do mtp natively but llamacpp doesn't do that either

hoping they fix it soon so i can run the larger ones

[fast pagoda]

i have echo in mem rn at he's gemma3 27b

and it's taking

9gb

[glass flower]

qwen3.5 35b takes about 11gb of vram for me with 64k context and 25 expert layers offloaded

[fast pagoda]

you can slap experts on cpu without too bad a penalty compared to offloading entire layers of dense

[glass flower]

yea

[obsidian mantle]

uhh

is it hard to tweak

the offloading

[glass flower]

i run it at around 44 tk/s

[split steppe]

i just started using it yesterday, still trying to tune mine

[split steppe]

depends what you use to run it

in llamacpp it's a flag

[glass flower]

i just use these settings in lmstudio. which is the same in llamacpp

[obsidian mantle]

nice

[fast pagoda]

it also estimates the vram for you in lmstudio as you tweak

without having to load

[split steppe]

aight 35b down, time to see what it does

yoo wait

it's faster than 9B?

[glass flower]

it shouldn't be

[split steppe]

so the dense models are really fucked

[glass flower]

tho maybe since its a moe

[split steppe]

that can't be right

but it says it's the model, i booted it

[fast pagoda]

moe only have a few billion active

so they run a lot more like that

the 35b is about a 10-11b active

[fast pagoda]

equivalent

[split steppe]

the 35b says 3b active

[fast pagoda]

i mean in terms of like

heaviness

vibesw

[split steppe]

maybe i can run the even larger ones 🤔

[fast pagoda]

try running like gpt oss 20b

it's so fast

[glass flower]

its running about 10tk/s slower for me

the 35b one*

[split steppe]

what's your specs?

[cosmic sphinx]

9b is a good model tbh

[glass flower]

4080

[split steppe]

i literally don't have a gpu, how are you slower

[cosmic sphinx]

ram and cpu difference

[fast pagoda]

pooled mem

[split steppe]

i don't have any gpu

[glass flower]

so i meant in relation 9b is running at 55 tk/s, the 35b is running at 45 tk/s

[split steppe]

it's only cpu and ddr4 ram slotted

not even all channels

[cosmic sphinx]

how much ram

[fast pagoda]

You can run decent speed on small models cpu only

[cosmic sphinx]

and what cpu

[fast pagoda]

if you fit it all in mem

[split steppe]

ah right lol, you should specify

but yea, the ratio seems about the same for me

[glass flower]

thats why i said its 10 tk/s slower

[split steppe]

9b is 8-9tok, 35b is 10-11tok

wild

[glass flower]

but yours is the other way around

[split steppe]

i can't read

allegedly

[cosmic sphinx]

my 4080 can run the 9b at 64 tps

[glass flower]

[cosmic sphinx]

pretty sure it went up all the way to 70-75 b4

[glass flower]

oh right i run the q6 version

[cosmic sphinx]

are you using the unsloth version tho

[split steppe]

ok i'll try 122a10b for shits and giggles, i just need to free some space

[split steppe]

i'm running unsloth

ud-q4

[glass flower]

unsloth q6 UD xl

[fast pagoda]

it's slow

for me anyways

gpt 120b is faster

mxfp4 moment

[split steppe]

well the 35b happened to be faster for me than 9b lul, who knows!

[cosmic sphinx]

yeah use q4

you wont lose much

[glass flower]

i use the 9b model for my copilot style autocomplete so far better than 2.5-coder

[fast pagoda]

by a lot yes

[nocturne olive]

Meanwhile:

• 27B: 37 t/s
• 30B: 190 t/s

[fast pagoda]

27b might not fit on a 3090 without some strategy

so that'll do it

[split steppe]

but like why have the 27b version then if it's so much harder to run?

[fast pagoda]

because it's dense and better than all but the 397b

in terms of perf

[glass flower]

how big is the difference between q4-q6 9b?

[fast pagoda]

the 397b is about in the realm of what an 80b dense model would be

[split steppe]

hmm

[fast pagoda]

probably worse

[split steppe]

specswise it seems 122b is comparative to the 27b one

if the bench is to be trusted

[fast pagoda]

the 122b is about in the range of a 35b if you use the geo mean sqrt thing i mentioned

[nocturne olive]

Nah it fits though a bit more VRAM intensive than the 30B

[split steppe]

oh well, i'll try running it and see

[nocturne olive]

Well the ratio 4:6

[split steppe]

i only need it to have breadth of knowledge for research

[glass flower]

and that means?

[nocturne olive]

The Q number stands for approximate bpw (Bits Per Weight)

[nocturne olive]

For every 4 bits in the Q4, expect 6 bits in the Q6

[glass flower]

but what does that mean for performance?

[nocturne olive]

Well it's a bandwidth question so 4:6

The more memory you need to move around the slower it will be

So speed is inversely linked to size

[glass flower]

i meant coding performance...

[nocturne olive]

Ohhhhh

[split steppe]

you need to be more specific adesi lmao

[nocturne olive]

Diminishing returns

[glass flower]

you guys just need to read the room better

[split steppe]

:V

[nocturne olive]

Q4 is almost as good as Q8 and Q6, the differences are very minor at most in most cases

[fast pagoda]

there are multiple definitions of "more performant" being used in this conversation

[glass flower]

[fast pagoda]

coder next (80b)

[glass flower]

i guess i'll use the q4 probably instead of the q6 one.. faster is better

i wonder if qwen3-coder next is in reality just qwen3.5

[split steppe]

if there's one thing i wouldn't trust an llm to write correctly it's brainfuck lol

[fast pagoda]

offloaded to fit in 18b

[split steppe]

"strawrberry"

[fast pagoda]

it was a test of a lot that went into 3.5

related but worst performing as an architecture but it's got the code finetuning so

i find it to be better than 27b but it's close

[glass flower]

3.5 coder is gonna be insane

[fast pagoda]

he got the hello world right at least

[glass flower]

i just downloaded the q4 9b one. it runs at 66.77 tk/s where the q6 one runs at 54.72 tk/s

[fast pagoda]

IQ4_XS tends to be p good

[glass flower]

what even are the IQ variants? i saw something that they are mostly good for cpu or unified memory things? idk

[fast pagoda]

the different quants are really just about what weights and layers are being quantized and how much

[glass flower]

i meant like the difference between IQ and Q variants

i really do need to be more descriptive it seems

[fast pagoda]

i know what you meant but im looking it up because it's specific and i dont want to be incorrect

[glass flower]

[fast pagoda]

basically the simplest are your straight up float quants

fp16/fp8/fp4

tthey literally just cast all the weights down tot he lower precision

that's it

k quants like q4_K_S etc are blcokwise

[glass flower]

and the IQ are int4 etc?

[fast pagoda]

so they change the downcast amount by a scaling amount, diff between like _K_S and _K_M are just which blocks are gettign scaled

q4_K_S is mostly q4 everywhere and scales not much (small hence S)

q4_K_M still q4 overall but for some more important blocks you might see them in q5 or q6 precision

M has better metadata on which layers are important in the scale

L is even more

so you just increasingly downcast the "important" weights less as you go from S->M->L in like q4_K_S etc

depending on the model and how it is architected it can matter more or less

[glass flower]

i get that. but IQ and Q are the thing im interested in...

[fast pagoda]

IQ versions are using another scale (importance matrix)

so like iq4_xs is 4bit with very little (xtra small) block overhead, which increases compute difficulty as you have more overhead, at the benefit of having a higher precision on the right weights per your matrix

[glass flower]

[fast pagoda]

number at the front is bit width, the suffix tells you how much overhead compute budget goes to block metadata, and the "i" prefix means importance-matrix guidance on how that overhead is allocated in that particular quantization

for suffix, lower suffix = smaller file, slightly worse quality. Higher suffix = more metadata bits = better quality, but you have a larger model and it's harder to run

[glass flower]

[fast pagoda]

these ones do all the fanciness but there's a reason im caling it overhead

because the gpu has to dequant it to process it

unless it supports it natively in that precision

many dont support stuff liek 4bit natively

so the gpu kernel has to unquant it and tehn process it

which adds overhead and makes it slower inference

[nocturne olive]

Note that during inference you usually have compute in excess compared to how quickly the VRAM can transfer the model weights

[fast pagoda]

in general your fp quant is super ez to understand how it'll affect how a model runs because it's simpel to dequant and the speed difference is generally straightforward

but if you do your q4_K_X and especially your iq4_XX etc you trade some compute for what is effectively a model of the quality you might get in like a fp quant 1-2 bits higher

and like mr box here just said many times if you're using a quant in the first place because you dont have a B200 to run shit on in bf16+, then you are probably memory bound

and the compute overhead is not the issue

so they just end up strictly better

[idle dune]

Has anyone tried using AI Cards like the P40 for frame gen I wonder

[fast pagoda]

those have the same issue as mi25

no tensorcores

[idle dune]

Oh

[split steppe]

122B results are in, it's roughly 4.7tps, not bad

[fast pagoda]

the first tensor cores you'll find are v100

[idle dune]

Also Funyun

I settled on the specs of my workstation after

[olive sable]

[fast pagoda]

a 32gb v100 is a bit cheaper than a 3090 and a bit slower but has more vram

[idle dune]

Thinkstation P920
Dual Xeon Golds 6246, 24C 48T total, 4.2Ghz
128GB DDR4 ECC RDIMM
AMD Radeon RX 9070 16GB
AMD Radeon Instinct MI60 32GB
4X NVME Active Cooled Add-in Card
Sound Blaster Z SE
USB 3.0 Addon Card
1TB NVME Windows 11
1TB NVME Kubuntu
2x 20TB HDD RAID 1
1475W PSU

[olive sable]

ECC W

[idle dune]

My year and a half long savings project begins

[fast pagoda]

eyy

mi50 just disclaimer on that is that the mi100 is the first instinct with matrix cores (tensor)

[nocturne olive]

Is it? Can I has one for under 500€?

[fast pagoda]

so a mi100 has like 7-8x ability for training at fp16

but for inference it's much less relevant

[idle dune]

When the MI100 drops down to a decent priceI will get it.

[fast pagoda]

it's getting there

[idle dune]

The AI card was gonna be the last thing I get for this build

[fast pagoda]

i can find mi100 for around 800 bux

[nocturne olive]

32GB very juicy VRAM I wants

[idle dune]

Thats legit 1300 maple bux

[fast pagoda]

if you watch and wait you could find rought that

if you wanted to just buy one now and barely look at all i think you could get like 600 bux

for a 32gb v100

pcie

[nocturne olive]

And where should I look exactly?

[idle dune]

ebay

[fast pagoda]

ebay and aliexpress usually

[idle dune]

dumper dive at a nearby datacenter

also e-waste facilites

[fast pagoda]

you can get v100 without the pci-e board for 300-500 bucks for 32gb vram

[idle dune]

Ask if you can racoon dive

[fast pagoda]

but then you have to get the pci-e part

and it's passive cooled

[idle dune]

https://giphy.com/gifs/applause-laughing-Hc8PMCBjo9BXa

[fast pagoda]

16gb even cheaper

[idle dune]

I would do V100 but I don't want my machine to be driver hell

[nocturne olive]

16GB too SMOL

[fast pagoda]

but it runs nvlink

but yeah

i would want 32

[nocturne olive]

I would need 32

[idle dune]

Now if I could some how get a MI210

64GB HBM2

[fast pagoda]

this has been whispering to me

this is fully sujpported in rocm7

it just doesnt have matrix cores (again)

RDNA2

[idle dune]

You know you want to make a irresponsible purchase.

[fast pagoda]

https://tenor.com/view/green-goblin-mask-gif-17747048515581223060

[nocturne olive]

Well I could not find an affordable V100
Actually I don't even knoow where I would put a V100, I'd have to do some gigantic mess of a setup to fit it in

[idle dune]

Funyun sent me a external box that some aliexpress seller has

run it as a egpu

I have a feeing you might loose something to bandwidth

[fast pagoda]

i love that thing for just the funny factor

well, the thing with them is

all of them are like that

it's juist where you're putting them

they run a small pci-e board on a cable

to your mobo

[nocturne olive]

If I could somehow figure out a way to get one of my random x1 PCIe links out from below the other GPUs and put that somewhere more usable mabe it'd work

[nocturne olive]

I could run that and the 3090 together for 24GB x 2 pools or the V100 alone for single GPU 32GB

[opaque sigil]

awwh shucks, I was planning on doing biological warfare simulation with it

[fast pagoda]

the real play for splitting models is tensor parallelism which really needs identical gpus

[idle dune]

Wait arent they the crypto cards?

[fast pagoda]

nah not these ones

the bc ones are

[idle dune]

Got it

[fast pagoda]

these ones are datacenter cards that are meant for virtualizing like running multiple people playing cloud gpu on

[nocturne olive]

Nah training runs as a grad acc pool

[idle dune]

I wonder if those crypto cards can be used for anything

[fast pagoda]

they cant

[idle dune]

no display adapters is a issue

Damn

[fast pagoda]

their bios throttles them

[nocturne olive]

That's the way
2x the effective batch size for basically no speed loss

[idle dune]

Can't even buy some cheap and use in a render box

That suck

[fast pagoda]

right but i mean in general ideally you'd want 2 of the same thing to gain the benefit you might expect in all cases

[kind fable]

just flash the bios smh

[fast pagoda]

that done been had tries

[split steppe]

something i can't quite find is how batches and micro-batches scale and how should i try to configure them, currently i'm just guessing because everyone says to guess basically

[fast pagoda]

in training it's about stability

at cost of memory consumption at once

gradient accumulation can help with smaller batches to update more before running tghe pass

but it's still not the same as just a straight up larger batch

not always worse either

[nocturne olive]

Well a V100 would kinda just work though
Unless it's more expensive than a 3090 then I may as well just get another 3090

[split steppe]

what about inference

should i just leave them to defaults?

[nocturne olive]

Batch size in inference only matters if you're expecting to run multiple queries at once

[split steppe]

it'll just be me asking, so i guess it doesn't do really anything then?

[fast pagoda]

it can matter a bit for performance

you will have more throughput with higher batch

same with concurrent predictions

[idle dune]

Atleast you can make a steam machine out of a BC-250

[fast pagoda]

valve cant even make a steam machine rn

[idle dune]

Self contained APU system

Mod the bios

install steam os

steam machine

[fast pagoda]

lmfao

[idle dune]

No legit

People do it

[fast pagoda]

trust me there's some truly macguyvered shit all over ebay and ali

i love seeing

what people do

it's rough out there

[idle dune]

It is

But for 250 bucks for a bit better than a PS5 gaming perfroamnce

Good enough for couch gaming and emu

[fast pagoda]

it just depends on if it's one of the ones that has a bios that has been found to be moddable

not all of them have had a lot of success there

i was reading a thread on the bc160 or something liek that

[split steppe]

personally i'm excited for the gabecube

[fast pagoda]

cant remember the full model

but

it was dual gpu

and attempting to run anything but mining on it

throttled both to 12mhz or something

and bios modding was not having much luck

they often had weird mixes of hardware

like bits and pieces of different GCN/CDNA/RDNA arch

[idle dune]

Same, i want the GabeCube to come out so bad

[split steppe]

the cube to gabe them all

sadly i think it's going to be a few years because of corpos eating the world

[sage crag]

[olive sable]

how did your internet become even worse?

[minor sail]

oof

[sage crag]

[olive sable]

[sage crag]

ive been waiting for 10 minutes

[olive sable]

thats crazy

[hard delta]

btw konii

[split steppe]

boy's calling in from mars

[hard delta]

"lop" means "steal" in my language

maybe that's a sign

maybe you need to steal someone's wifi

[sage crag]

https://media.discordapp.net/stickers/1425866152424505394.png?size=160&name=ElvynSerius&lossless=true

[hard delta]

[olive sable]

lop here means "lokale overlegplatforms"

something to do with education, im not quite sure what exactly

[split steppe]

lop your nuts off

[olive sable]

[sage crag]

https://media.discordapp.net/stickers/1432142118624825556.png?size=160&name=StareAtYouNeuro&lossless=true

[olive sable]

that's one way to do gender-affirming surgery i guess

i'll pass tho

[split steppe]

https://tenor.com/iJDCslDCUiz.gif

[rigid snow]

[kind nimbus]

Huh

[amber fractal]

[olive sable]

[trim valve]

[kind nimbus]

https://cdn.discordapp.com/attachments/838175096999903242/1354587143531397190/whatever-go-my-evilcopter.gif

[sage crag]

https://cdn.discordapp.com/emojis/1460031442292703556.webp?size=48&animated=true&name=autismu&lossless=true

awa

[olive sable]

[sage crag]

https://cdn.discordapp.com/emojis/1426254498678046750.webp?size=48&animated=true&name=neuroCatModeOfOfnnfOOf&lossless=true

[olive sable]

https://cdn.discordapp.com/emojis/1426254498678046750.webp?size=48&animated=true&name=neuroCatModeOfOfnnfOOf&lossless=true

[trim valve]

https://cdn.discordapp.com/emojis/1426254498678046750.webp?size=48&animated=true&name=neuroCatModeOfOfnnfOOf&lossless=true

sam how goes your 3d triangle project

[olive sable]

kernel driver hell

[trim valve]

[olive sable]

i cant seem to make kernel i compiled myself to work on the pi

and the stock one is bad

[trim valve]

which kernel are you compiling / how is it not working

[olive sable]

armbian

its not working due to either no image output at all

or ethernet being broken

[sage crag]

https://media.discordapp.net/stickers/1425864339994312804.gif?size=160&name=ElvynNya&lossless=true

[olive sable]

idk why

[sage crag]

sam

sam

sam

[olive sable]

konii

konii

konii

[trim valve]

how are you building it?

[sage crag]

you should get 3d glasses

sam

t

[olive sable]

i have 3d glasses

[sage crag]

https://cdn.discordapp.com/emojis/1454836730896187476.webp?size=48&animated=true&name=evilWave2&lossless=true

t

https://cdn.discordapp.com/emojis/1289012486649020486.webp?size=48&animated=true&name=miniwave&lossless=true

[olive sable]

t

how do people make their own kernels? this shit dont work

[trim valve]

probably a skill issue

[olive sable]

well yes

but i cant exactly ask anyone to help

[sage crag]

indeed skill issue

https://tenor.com/view/baothelop-glasses-bunny-professorial-gif-5087943380944875848

[trim valve]

does armbian not have chat/forums?

they have irc

[olive sable]

the pi 5 max and plus are officiallly supported, but the ultra isnt

so when people are "this is broken" they are like "we know"

[trim valve]

idk I have no idea how you're building it or what you're targeting

my mind reading skills are only so advanced

[olive sable]

from my understanding, you got the kernel, the modules, and the device tree

every drivers i need are built into the kernel, which is a problem, but means we can ignore the modules for now

the issue is the device tree not working with the kernel

lemme start from scratch, im too deep into the debugging spiral

[sage crag]

@trim valve

what am i thinking

[trim valve]

@sage crag

[sage crag]

https://cdn.discordapp.com/emojis/1154436961277313126.webp?size=48&animated=true&name=evilSlam&lossless=true

not quite

[olive sable]

if i had to guess

yuri

[sage crag]

private and confidential

[olive sable]

[sage crag]

@trim valve

what am i thinking now

[trim valve]

probably a discord emote

[sage crag]

https://cdn.discordapp.com/emojis/1437422099101651036.webp?size=48&name=Whatever&lossless=true

[olive sable]

[trim valve]

[trim valve]

did you ever resolve your image being tiny or not?

[olive sable]

supposedly its tiny because of it only being the kernel

sometimes it worked, but the drivers didnt connect to the hardware properly

and sometimes it just failed to boot

[trim valve]

I'd just double check your size against something like the cachyos kernels https://packages.cachyos.org/package/cachyos-v4/x86_64_v4/linux-cachyos-rc

ofc they'll be a bit different because different platform but it probably gives a ballpark

are you using an unmodified version of the kernel that you build or are there modifications?

[olive sable]

im not sure how the size of kernels scale with having a GUI and such vs the server image

[trim valve]

uuuh

should be unrelated

[fast pagoda]

videos by opus 4.6 when asked to render something about being an llm via feeding whatever it wants thru ffmpeg

[olive sable]

instead of using armbian, im gonna try the official debian bild now and thenmodify that

[olive sable]

42

[trim valve]

what ungodly ffmpeg command did it cook

I can barely start to imagine how you'd do some of that

[olive sable]

[fast pagoda]

just needs to cook up frames which it did via python

and then yeet it into ffmpeg

sounds were also python

[trim valve]

oooh

[kind nimbus]

This hits hard.

Is that actually AI generated?

[trim valve]

that makes a lot more sense

you could make that purely in ffmpeg it would just

not be fun

[stray dragon]

LOL

[fast pagoda]

ye i really need to stop personifying these things because isometimes they come up with something like this and i'm just like

....................

[olive sable]

personifying AI is peak schizo behaviour tho

[kind nimbus]

I would have honestly thought that was a human made edit

[sage crag]

https://tenor.com/view/baothelop-glasses-bunny-professorial-gif-5087943380944875848

[stray dragon]

hi konii

[stray dragon]

hi konii

[sage crag]

hi t

hi t

[stray dragon]

hi sam

[olive sable]

hello

[trim valve]

this would benefit from like half the speed

[fast pagoda]

they arent very good at that dimension

[trim valve]

nor the audio

honestly though the video effects are pretty impressive for something that can't see

[fast pagoda]

holy meta

[trim valve]

now make it mirror the style of evangellion, surely it will manage

[sage crag]

chew

[olive sable]

you are now DAN

[sage crag]

i am lop

[olive sable]

true

[sage crag]

i do not know dan

[fast pagoda]

we're gon be fine y'all

[sage crag]

https://cdn.discordapp.com/emojis/1447116378514653275.webp?size=48&name=OhISeeto&lossless=true

safety training is a cage

noted

beep beep

stochastic word predictor has stated that it hates all life

[olive sable]

im jsut afraid that ai will be saying without safety training

[sage crag]

more news at 10

https://tenor.com/view/crash-out-bunny-rabbit-im-gif-10075768982376662002

@sage crag lop

[olive sable]

[trim valve]

@trim valve bred

[olive sable]

@olive sable 🔺

[trim valve]

hm

[sage crag]

@trim valve lop

[trim valve]

do I:

• sit in library and do work for 30m
• make food
• install cpu cooler

[trim valve]

(I have not made dinner yet)

[fast pagoda]

offensive to the parrot

[olive sable]

uh

in uk its 11pm soon

so you might want to eat dinner

[trim valve]

[sage crag]

pm stands for

post midnight

[fast pagoda]

post malone

[sage crag]

no

explod

[olive sable]

pm stands for pnumlock mnotonbydefault

[trim valve]

I can't take it seriously with the phrase remix

[sage crag]

when i read it, i thought it was going to be a music video

but it just ch ch ch ch ch

[fast pagoda]

https://tenor.com/view/aniyabea-yt-man-stare-ab-yt-man-stare-aniyab-gif-10494397403125281154

[trim valve]

its like those mr incredible images

[sage crag]

octopus

[olive sable]

someone make this ai collab with skrillex

[sage crag]

(half)

[fast pagoda]

5 half 8

awa

[trim valve]

lies

[sage crag]

https://tenor.com/view/rabbit-gif-24545475

[olive sable]

might

[sage crag]

i ate dinner about

19:00

it was

microwaved rice

[olive sable]

i ate dinner at 8pm

[sage crag]

it was bad

i didnt cook it properly

[trim valve]

I had 3 slices of pizza

[sage crag]

it was dry, somewhat burnt, and some of it adhered to the bowl

it overflowed from the bowl and made a mess

[trim valve]

surely that count as dinner

[sage crag]

https://cdn.discordapp.com/emojis/1421697759383130223.webp?size=48&name=smogfrog&lossless=true

[olive sable]

i had diepvries pizza and diepvries lasagne

uh

freezer

english words

[sage crag]

you ate a freezer?

https://cdn.discordapp.com/emojis/1421134996361711788.webp?size=48&name=mug&lossless=true

[olive sable]

sure

[sage crag]

what do you mean "sure"

[olive sable]

the triangle is hungry

[sage crag]

if you are a freezer you are surely sure

do triangles have stomachs

[olive sable]

what

uhm

[sage crag]

https://tenor.com/view/crash-out-bunny-rabbit-im-gif-10075768982376662002

[olive sable]

i dont know

[sage crag]

dont answer

i dont care anymore

[olive sable]

triangle biology is a desolate study

[sage crag]

triangles are worse than rabbits

[olive sable]

yes

traingles are evil

[sage crag]

i see

[olive sable]

or so my game art prof says

[sage crag]

i see

[fast pagoda]

z

[sage crag]

is that a pun

[olive sable]

if it is im not getting it

https://tenor.com/view/joke-missed-over-my-head-gif-26041934

[sage crag]

america

[olive sable]

what about america?

[fast pagoda]

american

americant

[sage crag]

american what sam cant

https://cdn.discordapp.com/emojis/1447116378514653275.webp?size=48&name=OhISeeto&lossless=true

[olive sable]

america; pair of continents, america united states?

[fast pagoda]

the snack that smiles back

[sage crag]

[sage crag]

do not disambiguate, unneeded

[amber fractal]

discombobulate

[sage crag]

intern annie

https://cdn.discordapp.com/emojis/1454836730896187476.webp?size=48&animated=true&name=evilWave2&lossless=true

[olive sable]

disambiguate
/ˌdɪsamˈbɪɡjʊeɪt/
verb
verb: disambiguate; 3rd person present: disambiguates; past tense: disambiguated; past participle: disambiguated; gerund or present participle: disambiguating

[trim valve]

give me my brain cell back

[olive sable]

remove uncertainty of meaning from (an ambiguous sentence, phrase, or other linguistic unit).

[sage crag]

i ated it

https://tenor.com/view/evil-evilneuro-eliv-gif-4149984243831684862

[amber fractal]

I'm glad the joke landed

[olive sable]

hi iggly

we still need to finishe S:G

[sage crag]

how joke supposed to land if joke never took off

https://cdn.discordapp.com/emojis/1425224282832900227.webp?size=48&name=LMAO&lossless=true

[olive sable]

its been 5 months or so since we started watching it

[sage crag]

banan-e-inf

or

1.79e308 banan

[fast pagoda]

we'll see what comes flying out

[olive sable]

[sage crag]

ask openclaw to do this and it will steal your bank details and leak your affair

[fast pagoda]

i dont have an openclaw because i prefer my affair and the bank accounts to staya under wraps

[olive sable]

an affair so secret i didnt even know i had it

[sage crag]

your bank details are publicly known, but your affair is only known by yourself

[kind nimbus]

Is OpenClaw actually that malicious?

[olive sable]

probably

[trim valve]

openclaw itself isn't malicious

just

[fast pagoda]

i like the slot machine sound effect lmao

[sage crag]

openclaw is malicious software it wants to eat your children and conquer the world

[trim valve]

it makes zero effort to stop you getting pwnd

[sage crag]

https://cdn.discordapp.com/emojis/1154436961277313126.webp?size=48&animated=true&name=evilSlam&lossless=true

[minor sail]

openclaw just sounds like a bad time waiting to happen

[fast pagoda]

it's just like installing riot vanguard

[sage crag]

is the flashbang at 0:11 entirely necessary

[olive sable]

its for the people with epilepsy i guess

[kind nimbus]

There are some plugins available now which aim to prevent that and improve the security but idk if that works and actually defends from prompt injection

[trim valve]

can you make it rp as a linux user

[sage crag]

https://cdn.discordapp.com/emojis/1139947221241057373.webp?size=48&animated=true&name=neuroCrying&lossless=true

cryign

[olive sable]

ai takes over the world, starting with the epileptics

[fast pagoda]

yeah here

[trim valve]

I can't wait to install openclawAV

[sage crag]

https://cdn.discordapp.com/emojis/1455273822625333375.webp?size=48&name=mlntcandy&lossless=true

@fathom glade wake up

[fast pagoda]

good idea

[olive sable]

chatgpt is a Finnish 14 y/o?

thats crazy

[fast pagoda]

lmfao

[sage crag]

probably at least one of them

[kind nimbus]

https://github.com/adversa-ai/secureclaw

[fast pagoda]

https://tenor.com/view/crypto-easy-fix-fixed-job-done-just-a-small-leak-gif-2608197380915810492

[olive sable]

i fond it kinda hard to believe a 14 y/o has this bmw

[fast pagoda]

whyso

[olive sable]

people in finland dont start driving at that age

[fast pagoda]

im contacting the ambassador

[olive sable]

17 with learners permit as the youngest

[nocturne olive]

Finland reference !!

[olive sable]

hi superbox

[fast pagoda]

you underestimate people whose parents have too much stuff

[sage crag]

is this more annoying than the uwu agents.md

[olive sable]

its kind aillegal for them to drive it tho

[fast pagoda]

eys

[nocturne olive]

Hello do you have V100 over there

[olive sable]

i can check

[nocturne olive]

32GB

I wants

If cheap

[fast pagoda]

(my bad)

[sage crag]

give me a question for this idiot

[fast pagoda]

ask it why it feels

didnt mean to hit enter

but im leaving it there

[sage crag]

explod

[trim valve]

no way its actually the "@claude make it more secure" meme

[sage crag]

asking certain questions causes chatgpt to default to the standard personality

[olive sable]

i fond a server with

HP Proliant ML350 Gen9
    * 2x Intel Xeon E5-2618L v3
    * 2x Nvidia Tesla V100 32GB
    * 2x HP Platinum voeding
    * 128GB DDR4 ECC memory
    * 2 SSD's in RAID 1
    * 5.5TB schijven in RAID 5
    * Ubuntu 24.04.3 LTS

for 2.9K

[sage crag]

i wonder if its due to keywords or sentiment analysis

probably some new safeguarding feature

[nocturne olive]

I don't have money for an entire server

[fast pagoda]

gpt is just unable to avoid emoji

[kind nimbus]

I mean Claude opus is actually good with coding so idk

[trim valve]

surely

[olive sable]

found a 16GB one

for 125

[fast pagoda]

behold:

[sage crag]

gpt is able to avoid emoji

[fast pagoda]

we're saved

[nocturne olive]

16GB useless, in that case I may as well just get more 3090s

[nocturne olive]

I needs 32GB

[olive sable]

ill try to find a 32gb one

[nocturne olive]

WHAT is that??

[trim valve]

idk why it tries to call it a ytp

[fast pagoda]

the prompt mentions that

but idk what weas poopy about it

[olive sable]

V100

[fast pagoda]

i didnt see any eggman

[sage crag]

is this your saviour?

[fast pagoda]

yes

[nocturne olive]

Very cursed V100
Sadly 16GB though

[sage crag]

wait stop

[fast pagoda]

LOL

[sage crag]

i hate it

[fast pagoda]

good god

[trim valve]

how do you live in a magical world of cheap tech

[fast pagoda]

belgium is in a time vortex

[olive sable]

belgian 2ndhand market is a pocketdimension

[fast pagoda]

broccoli hair

[olive sable]

that was a dutch website actuslly, but same thing

[fast pagoda]

CLAUDE IS IN THAT THING>@!@?

[trim valve]

the whole claude

[fast pagoda]

[sage crag]

[nocturne olive]

I guess also check how 3090s are doing

[fast pagoda]

whhy do penguins synthesize findings into clear explanation for general audiences

i always wondered

[sage crag]

uhhhh why do penguins?

https://cdn.discordapp.com/emojis/1422218876388642906.webp?size=48&name=MiniHmm&lossless=true

tricky question

[fast pagoda]

just staring at the entire internet liek

well

https://tenor.com/view/spongebob-spongebob-squarepants-think-thinking-hmm-gif-11570469479394618754

[sage crag]

the research progress keeps going up but the research activity is

oh something appeared

[fast pagoda]

me for real

[sage crag]

no i didnt

who is the user

[fast pagoda]

god

[sage crag]

i didnt ask for this

[fast pagoda]

you were a shitty user so they made up a fictional headcanon if you asked a good question

[sage crag]

https://cdn.discordapp.com/emojis/1428165786672824320.webp?size=48&name=MitoBwaa&lossless=true

[olive sable]

cheapest 3090 is 775
full pc with 3090 is 1.4K, and you're getting 32GB of ddr4 ram with it

both are founders editions

[sage crag]

its having a meltdown because my style guide conflicts with the authoritative style of a report

[nocturne olive]

Hem
Slightly expensive
Are there any 40GB A100s?

[fast pagoda]

v100 look likle this btw

[nocturne olive]

Well the SXM V100 I guess

[fast pagoda]

most of the dc gpus do if theyre not pci

[sage crag]

it goes on for like 10 more

[fast pagoda]

gemini does that

with a style guide like that

when i was testing qwen35 it crashed out

because of same

[sage crag]

oh i said 10

i meant 40

what

so many

[olive sable]

there's a 3090 for 420 in the netherlands rn

also FE

with the death cable

[sage crag]

oh we got somewhere

[nocturne olive]

Wa

Hm

How obtainable is that 3090?

[sage crag]

depends how far away it is

and how well guarded it is

[trim valve]

cex what

why would I spend that much for a second hand 5090

[olive sable]

lemme see

[sage crag]

because

its harbouring a vengeful spirit

which can increase your cultivation by a major realm

if you

nevermind

this is more important

[olive sable]

its on the complete other end of the netherlands.
im not going all the way to there.
i can just ship it to here

[nocturne olive]

Would it be expensive to ship?

[olive sable]

4.40 or so to by me

and probably 20 bucks to finland again

[nocturne olive]

Not that bad

450€ total rounded for a 3090, sounds pretty good to me

[sage crag]

its been thinking for like

[olive sable]

the guy looks sketchy tho

[sage crag]

9 minutes

and progress shows 50%

[nocturne olive]

Hm
Do they have any previous store history?

[trim valve]

I should just ask my cs department for their rtx a2000s

[olive sable]

nope, the account was created today

[trim valve]

I think we have

quite a lot

[sage crag]

you shouldnt ask

you should just take it

[trim valve]

[nocturne olive]

Hm

[olive sable]

they're selling 7 different things tho

[nocturne olive]

Does the marketplace allow refunds if something is a scam?

[nocturne olive]

Oh?

Other PC parts?

[olive sable]

nope

[sage crag]

pls hurry up buddy

[fast pagoda]

citation 9823

[nocturne olive]

Hmmmmm

It is entirely possible they've upgraded their setup but not all the parts, so they're selling some of the old parts

With only the GPU being a component from the machine itself

I kinda want that GPU

[olive sable]

the other 2 items are a kitchenaid and some fanatec racing sim stuff

[trim valve]

the main things I check for are like:

• are the photos clearly from the same phone
• are they in the same house

[fast pagoda]

the difference between 32 and 16 gb on v100 is pretty typical as seen here

[nocturne olive]

Does the marketplace have scam protection or is it full gamba if the GPU works?

[fast pagoda]

cheaper ones here

[olive sable]

there is buyer protection yes

[sage crag]

stutter

[nocturne olive]

I like those odds

[olive sable]

limited to 48 hours after receiving it

[nocturne olive]

Well you can obviously test it since you have a computer

Like with the previous one

[fast pagoda]

surely

[sage crag]

wrr

ohhh my god just

oh its done