charmOS

[eager yew]

do u hab scheduler

[stuck nexus]

the trick to having good ideas is to just be well read and trick people into thinking you have good ideas by learning about and implementing other ppls good ideas

[eager yew]

I never saw one

[placid vessel]

yes

[eager yew]

do it work and yield and stuff

[placid vessel]

yes

[eager yew]

neat

[severe ermine]

you can search "rusty os" on github and you'll get dozens of kernels with the same name lmfao

such original name

[stuck nexus]

rewrite mintia2 scheduler in C and just put my name somewhere in the credits file

[placid vessel]

yikes

i really wanted to avoid "OS" in the name

[severe ermine]

menix == minix

[placid vessel]

yikes

OH WAIT

duh

[eager yew]

nah my scheduler is complete aside from the lack of manual yield so i just need to fix it to do that

[severe ermine]

great marketing

[placid vessel]

why did i never look at minix vfs

[severe ermine]

people will confuse the names and star your repo

[placid vessel]

well menix actually stood for something

[severe ermine]

mervin unix

[eager yew]

so rsp doesnt matter? oh ok lol

[severe ermine]

you won't guess what Ilobilix stands for

[eager yew]

oh wait i see

new_rsp

nvm

[placid vessel]

"minimal expandable unix like system"

[eager yew]

manix is marvin unix

[placid vessel]

this makes use of the C calling convention

[stuck nexus]

counterpoint you dont have priorities

[placid vessel]

he has an AFS

[eager yew]

Soon™

[placid vessel]

actually fair scheduler

[eager yew]

i was socialismmaxxing

the cores even try to be equal to each other in load

Shocked

[severe ermine]

is actually fair scheduler more or less fair than completely fair scheduler?

[placid vessel]

our timeslice

[eager yew]

actually fair just makes sure you cant have things like kernel events with higher priorities or something i think

so it's actually fair whereas CFS still has those priorities

i think

[placid vessel]

i just made the term up

idk if that's actually an algo

[severe ermine]

lmfao

I believed you

you betrayed me

[eager yew]

NFS isn't Network FileSystem, it's Not Fair Scheduler

[placid vessel]

i never say things that are true

[severe ermine]

paradox

[stuck nexus]

@eager yew hows your implementation strategy are you more of a "completely implement each component before moving on to the next" guy or a "do each thing simply and integrate them and then build upon them later" guy

[eager yew]

the second one

my scheduler used to be RR

no work stealing

then i came back and made it MLFQ with work stealing

and i am fixing it for manual yield

[stuck nexus]

good the first is unbelievably boring

[eager yew]

"We have menix code at home"

[stuck nexus]

id also argue its only correct once you know exactly what everything needs to do which is unlikely for your first 5 or so rewrites

[placid vessel]

who the fuck does the former

[eager yew]

i havent had to rewrite things like drivers which are just an implementation of a specification but i have indeed had 4 total allocator rewrites 3 scheduler rewrites 1 vfs rewrite and some other small things here and there

[stuck nexus]

me rn

[placid vessel]

"yes please never give me any results i don't want anything working"

[eager yew]

"we have -> at home"

[placid vessel]

.deref()

[stuck nexus]

i completely implemented like 3 subsystems in sequence and they each turned out like 10k lines and then i got so bored cuz there needs to be another like 2-3 of them before it looks like its doing anything that i stopped working for like 4 months

[eager yew]

whole function call is super necessary for what is just a [ptr] in asm 💥

[placid vessel]

i feel sorry for u

[eager yew]

is mintia cooked

[placid vessel]

i keep staring at my VFS code and my mind goes numb when i look at my todo!() statements

[stuck nexus]

what challenges u so

about vfs

[eager yew]

i can helps a bit my vfs works fine

[placid vessel]

i know how it works, i just don't know how to design an OKish interface

[stuck nexus]

interface between what and what

[eager yew]

where is the pain and struggle

[placid vessel]

like an API to interact with it, as in: File::open needs what arguments, returns what, etc

[stuck nexus]

why cant you just start with something and then change it as your needs change

[placid vessel]

i wish i could tell you

i don't know where to start 😢

i may be on the spectrum

[eager yew]

shocker

[eager yew]

write the function prototypes

[stuck nexus]

i bet i have a suggestion for how you should do your vfs that doesnt sound like anything you read before and might be simpler to you

and is also better

[eager yew]

@placid vessel come up with these

and then implement them for each filesystem

it's that shrimple (for me at least)

[placid vessel]

oh?

[eager yew]

marketing

[placid vessel]

exactly

i need something like that

i just need to know what i have to implement

not even how

just what

[eager yew]

look at the linux fs syscalls 💥

you'll need most of those

that's what i did

[stuck nexus]

yeah i wont tell u though its a trade secret

(im too tired)

[placid vessel]

[stuck nexus]

diagramming it might take less energy than verbally explaining it and then ill have something i can spam whenever people say the word vfs

[placid vessel]

i can't listen to verbal instructions anyways

[eager yew]

so you know all your commands you have like mkdir cd chmod and things, just implement interfaces necessary for the basics of that and later on you can add if you need

that was my logic

it worked fine

ohhh shoot i just remembered i have to implement the functions necessary to respect file flags like secure delete on ext2

ts gonna be so painful

https://github.com/BlueGummi/charmos/blob/main/include/fs/vfs.h here is vfs

not very good but it's ok

sigh i am not a fan of intels specs

yes thank you intel i was absolutely looking forward to writing all your documentation as code comments because i dont want to constantly have to be referencing your manual this is just what i wanted

[stuck nexus]

basically involves completely eliminating the concept of the vnode cache

when a vnode's refcount drops to 0 you immediately delete it

sounds bad right

[placid vessel]

can that be performant

[stuck nexus]

well you also have a dirent cache and the entries therein keep a reference to the vnode

what you reclaim is entries in the dirent cache with a refcount of 0 when there are too many of them

what this lets you do in the fs drivers is they can each create, keep track of, lookup, delete vnodes in their own way

which is most appropriate imo

a unix-like filesystem will want to keep a hash table keyed by inode number that yields the vnode

(or something)

whereas FAT can keep a tree of vnodes looked up by name as children from their parent

this is superior to the centralized vnode caching you sometimes see which enforces a particular lookup scheme, for example the haiku one makes all vnodes have a unique "vnid" which is simply equivalent to the inode number for unixlike filesystems, but then the FAT driver needs like 5000 lines of code to maintain fake vnids for its vnodes

it also makes vnode reclamation easier

[eager yew]

yea the linux kernel on line 853 of fs/fat/inode.c has fake inode calls

it's interesting

they have it elsewhere too

I use unique identifiers in my vfs for faster lookups and to compare two vnodes as being the same thing on the filesystem

[gentle gazelle]

iunique is a bit silly, i wonder why they moved away from using the dentry position as the inode number

[eager yew]

would collisions happen there or somethin

[gentle gazelle]

inode numbers are only unique within a file system

[eager yew]

BUGS

[gentle gazelle]

and you can't have 2 dentries in the same position

[stuck nexus]

the dirent position can change from the file being renamed or whatever

but you still need to keep it unique

so if another file is created in the same position you need to tell that you cant give it the same id

and give it some other id

and it gets messy

[eager yew]

is there a better way to generate unique inodes

like linear search cant possibly be the only method

probably a hash function but then that collides

[stuck nexus]

anyway my way i just described is way better and avoids this whole thing

[gentle gazelle]

well you still need inode numbers to satisfy userspace

userspace tools will use (st_dev, st_ino) pairs to track files

[stuck nexus]

just give it incrementing numbers

no need to even check for uniqueness

roll over could cause minor chaos but only in userspace lol

[eager yew]

u64 solves that too

[gentle gazelle]

tbf on modern lunix ino_t is 64-bit

so it just won't overflow

™

[eager yew]

oh deary me i have 18 quintillion inodes whatever shall i do 😨

[stuck nexus]

also the approach i took with my asid sequence number

[gentle gazelle]

i love 64-bit monotonic counters

[eager yew]

should i do this for pids or no because i really dont want to have a system with a long uptime and see pid 2 million or something

would be a little bit silly

[gentle gazelle]

eh i used to get that on linux since i had a script that'd run a bunch of commands every second

[stuck nexus]

i just use a free list within the pid table to allocate pids and keep a 12 bit sequence number for each pid table entry so that pids arent reused too quickly

ez and allows me to have O(1) pid lookup

[eager yew]

Shocked

smarto

speaking of O(1) time i think my cryptography in the kernel might be screwed because the CSPRNG isnt O(1)

chacha20 😔

sidechannel 💥 😔

[stuck nexus]

sequence number is necessary because if you delete and create processes repeatedly the same pid table entry will be reused each time since its a singly linked free list

so if the seq number wasnt there itd reuse way too fast

[eager yew]

isnt linux kernel crypto kinda cooked

they also use chacha20

oh wel

r/linuxsucks

[placid vessel]

p sure they also have real crypto sources

[eager yew]

but chacha20 isnt linear time so you can try to guess if you know the timing

[gentle gazelle]

chacha salsa

[placid vessel]

crypto is fake anyway

[eager yew]

lore

[severe ermine]

incredible

[placid vessel]

crazy

[eager yew]

lmao this is shockingly common in hobby oses

i think my "scheduler issue" was a printf bug where printf's lock would enable interrupts at the end

i changed that and it's gone

and when i ran my scheduler with more printf's before this it would fault more

now it all goen

craxy

[severe ermine]

ipi ftw

[placid vessel]

0x79 is such an odd number

0x20 is just the first ISR that's free

just use that or the highest vector possible

[eager yew]

it's a perfect square

[gentle gazelle]

0x20 is usually where irq 0 ends up after people remap the pic

[placid vessel]

using the pic

[eager yew]

I was hard coping with making a non interrupt context switch so I'ma just do the sane thing for now and just generate schedule yields with the interrupt handler and when my brain starts working later I'll refactor it into a proper non interrupt yield but right now I want to work on other things 💥💥💥💥

[stuck nexus]

That's crazy because I feel like the interrupt thing is over complicated and the other way should be easier

[eager yew]

I was having stack corruption bugs and I'm also getting bored of scheduler stuff anyway

it's better to get things done and then fix it when you can think of a fix naturally than force yourself to think of a fix and fixate on something

at least for me

this is why I go on walks when I get really problematic bugs

[short lichen]

Am I tweaking or does this not switch to the IST defined for the ISR?

[eager yew]

huh

why would it not

[short lichen]

I had an issue where upon forcing a reschedule the stack would get corrupted

And turns out the way i rescheduled

int 48

Does not switch to the stack for the isr

I literally had a similar issue back in March

[eager yew]

whar 😭

[short lichen]

Yeah

[eager yew]

is the timer interrupt calling the isr

is your asm syntax incorrect

[short lichen]

No the syntax was correct

[eager yew]

how does this occur

[short lichen]

int 0x30

[eager yew]

you need the $

that treats it as an address

[short lichen]

Brother

[eager yew]

💥

I'm not joking

[short lichen]

Im using intel

[eager yew]

try it now

oh

[eager yew]

is it inline assembly

[short lichen]

Yes

Intel inline

[eager yew]

did you do the .intel_syntax noprefix

[short lichen]

-masm=intel

[eager yew]

and then .att_syntax prefix at the end of the segment

[eager yew]

wait that lets you do intel inline asm?

oh neat

[short lichen]

Yes

[eager yew]

that's a wild bug then

[short lichen]

Hold on let me pull the Intel sdm

I'll show you

[eager yew]

oh boy

oh this one's gonna be a doozy I bet

it's always something wild when you gotta search the docs for 10 minutes

[eager yew]

did you ever find it

it's been a little bit

[short lichen]

I'm finding bruh

But I swear I read in the manual somewhere

That hardware exceptions/interrupts trigger an IST switch

[eager yew]

yeah software interrupts don't use the ist

was that the bug

[short lichen]

Ok where did you find it

[eager yew]

oh this is just information I knew lol

[short lichen]

I swear it was chapter 7 volume 3

[eager yew]

you just brought it up

but was that the issue

[short lichen]

Yes

[eager yew]

absolutely wild stuff

and it never got fixed?

[short lichen]

It did lol

[eager yew]

what was the fix

[short lichen]

I fixed it by sending an ipi to the cpu itself

[eager yew]

just rewrite the scheduler logic to not require an interrupt?

[eager yew]

crazy lol

[eager yew]

this is actually kinda silly but yeah understandable lol

I don't think my scheduler needs that

it works fine without that thingy mabob I can just manually call a software interrupt

hilarious bug tho

[short lichen]

I know 😭

[eager yew]

oh yeah here's the pseudocode for my scheduler if anyone wants to just get the jist of it without reading that codebase

function schedule():
    save_current_thread_state()

    if not should_attempt_steal():
        goto regular_schedule

    if too_many_stealers():
        goto regular_schedule

    victim_core = find_victim_core()
    if victim_core is NULL:
        goto regular_schedule

    victim_thread = pick_thread_from(victim_core)
    if victim_thread is NULL:
        goto regular_schedule

    switch_to(victim_thread)
    return

regular_schedule:
    next_thread = pick_from_own_mlfq()
    switch_to(next_thread)

[eager yew]

ah I know what I can do next for my scheduler

[short lichen]

No

[eager yew]

so basically, once I get to having thread priority levels, I want to avoid wasting time trying to work steal because that is an expensive operation

if it detects high system load and a currently high priority thread, rather than checking constantly if we can steal work, we can instead add batch stealing, and attempt a batch steal when we have a lower priority thread currently being scheduled because who cares about those goobers

that seems like a reasonable place to do that

this way I can avoid constantly checking for a steal opportunity and instead do one large steal to mass rebalance many threads when the opportunity presents itself

this will probably also help with steal success rates because this will have less cores that are able to perform a steal (but similar impacts when I do steal work) because I have a hard cap that says only half the cores can be work stealing at once

[eager yew]

how does work stealing work in your os

[short lichen]

There isn't any

[eager yew]

😲

[eager yew]

do you plan to add it

[short lichen]

Nah

[eager yew]

so if a core just becomes severely overloaded as time goes on it's just "oh well... that's too bad..."? lol

understandable

this isn't difficult to implement jsyk

[short lichen]

It shouldn't happen

[short lichen]

I know

[short lichen]

Again I doubt

I never thought a lot about the scheduler

Get a thread run it and repeat

[eager yew]

how about in a case where one core (purely by chance) has many interactive (constantly running tasks like DE threads and whatnot) assigned to it, while other cores get threads that quickly die

then over time that core that may be running all those interactive threads would have a much higher load

💥

[short lichen]

Yeah

[eager yew]

so that just won't happen?

how are you preventing this

[short lichen]

Well it can happen

[eager yew]

💥

and it's just "oh well, too bad"?

understandable

[short lichen]

Yeah lmao

Rn it's just fix and implement shit
I'll think about different scheduling algs after that

[eager yew]

ok well I mean if you ever get bored just remember that this is pretty fun to think about and implement because it's nice to change things up

[short lichen]

Yeah

[eager yew]

I'll go back to filesystems and come back later tomorrow because I don't wanna think about threads and tasks and cores and locks and mutexes and yields and async and all that ANY MORE

idk how people can put up with this for so long

[eager yew]

next april fools I'm gonna try to put my entire OS in one file and compile it like that

hopefully it will be a relatively more solid project by then so that it will either take 5 minutes or crash the computer

will be very silly

[eager yew]

back to implementing support for microsoft's bum ass filesystem

noooooogodwhy

or maybe xhci

that seems more pleasant

[eager yew]

it seems like steals are multiplefold more inefficient than regular schedule events

might wanna do batch stealing

[twin escarp]

wut the heck is a steal?

[eager yew]

let's say you have cores 0-3

core 0 is heavily overloaded, having 40 threads on it

cores 1-3 are bing chilling, with almost no workload

cores 1-3 realize this and go "Hey, let's remove threads from 0's queue, and give that work to ourselves"

now, core 0-3 all have 10 threads each

mucho faster 😁

now if ur wondering "why would i need this in the first place?"

let's say you have the following scenario

you have a desktop come alive at the start of your user login, as well as a bunch of small scripts

purely by chance (as the scheduler does not know how long a thread will run for when you first load it), the desktop threads all go on core 0 and the script runner threads all go on the rest

after the scripts end, now you have an imbalanced system, and it is work stealing that allows it to balance out

[twin escarp]

Ahhh, that makes sense!

[eager yew]

switched to a bitmap, regular scheduling events are sucking mildly less

i think i need more infrequent steals

ok yes regular schedules are noticeably faster once the steals stop

the problem is that the steals don't stop at the start, but i think that is OK

I think i might introduce something where after enough non-steal scheduling events (failed steals due to none available), i will stop checking for steals

10 microsecond scheduling overhead 😬

I think i should avoid load recalculating and just take a heuristic based on thread count

COOKING

i think this is good enough

the load computation heuristic was wasting too much time so i just use thread counts as loads and it's good enough

2.3 microseconds 🤩

i dont really need it too much faster but i might take a peek around and see if i can get it faster

work steals are still less than ideal so i might do something where they give up after waiting on a lock for too long

this is what happens when u dont prematurely optimize 💥

optimization is now pretty trivial

i think i'm gonna do a spin_trylock instead of waiting turns

ok how did that work so well

we are down from a few million clock cycles so whilst this is still hot garbo, it is better

the fact that this simple optimization cut the time by 2-4 times is wild

load balancing is slower but i think that this is overall better because more work is done and this test case is pretty extreme

it's testing one core with 50 more active threads than the others

and it still balances in <25s even with very very slow reschedule events

ok i will probably go and figure out batch stealing once i optimize a few more things, regular schedule events take 7,000 clock cycles which is "Good enough for me", and steals take about 100 times that much sometimes which is "Holy moly this is bad"

but steals happening don't really affect regular schedule events too much so i should be ok

[eager yew]

ok because constantly stealing sucks, this is the plan

I'm gonna do a periodic steal attempt in the main schedule event, with budgeted batch stealing, as well as a full rescan if a very large influx of threads is detected

should be fine

[placid vessel]

do you always steal

why not just wait until it's actually necessary

[eager yew]

I always attempt to see if a steal is viable on every scheduling attempt

[eager yew]

see here

[placid vessel]

yeah but how do you determine if something should steal

i would only do it if the load difference is too high

if the load is equal on all cores then there is 0 point in stealing

[eager yew]

yes this is what I do

but steals use lots of locks and do lots of waiting

[placid vessel]

so increase the threshold

[eager yew]

no because that still causes singular steals

I will steal once every X scheduling opportunities and steal in batches as well as have a mass forced steal if a large influx of threads is detected

it's just the locks that cause the slowdown

batch stealing should help

[placid vessel]

what i do is during thread creation it tries to find the optimal cpu to place it on from the beginning

[eager yew]

but see this would still cause stealing scenarios

you don't know a thread's runtime

I wonder if I should name my scheduler lol

CASS the charmOS Adaptive Scheduling System idk

[stuck nexus]

You never migrate after that?

[placid vessel]

not yet

i think rebalancing on thread creation should suffice for now

[eager yew]

the workstealing only appeared slow because i was PRINTING TO THE TERMINAL and that takes AGES

this is actually perfectly fine

[molten night]

The measurements are under kvm/real hw right?

[eager yew]

yep

[eager yew]

oh yeah i tested that tmpfs thing today and i apparently did it right (I think) first try

time to stress test

[eager yew]

lsp crashout

ok fixed it lol

i am a little bored so i will try to enter userspace for fun

ok time to jump to it now

[eager yew]

snooooooozefest

man...

syscalls time

asldkfjal;sdfjaweo;ijha;or

Shocked

how have i not page faulted yet

sus

ah i finally got one nice

i wonder if i will have 100 lines of includes by next year

oops i wasnt marking the pages as executable

rookie mistake

[eager yew]

lol this guy keeps page faulting 💥

funy

iretq here is faulting for some reason

weird 💥

[eager yew]

i am dumb it is not a page fault

my GPF handler was faulting

💥

i got to userspace or something

[eager yew]

we have now returned to userspace after a syscall

i forgot that nasm wants you to write

  o64 sysret

instead of

  sysretq```

[eager yew]

ok now that I got this bs working I shall integrate it with my scheduler later and spend 20 years writing syscalls

yippee 😁

[eager yew]

i had the funniest bug that ended up being a file read bug where i wrote an extra line

elf loading is just fine

ran a program

i was not supposed to get this far

whatever

i will keep going for the funny

[eager yew]

i fogort to allocate rsp0 so my fault handlers didnt trigger from userland

https://tenor.com/view/bloodcartier-gif-25860033

0 minutes since last idiotic bug

longest streak was 7 hours lets see if i can beat it

[eager yew]

feeling bored gonna do proper logging

current logging lowk goofy

the . is saying that the filesystem was mounted and the 'yabadabadoo' is saying the tmpfs mounted

wild stuff

[eager yew]

wait I need to start doing unit tests

shouldn't be too bad

__attribute__((constructor)) my beloved

[eager yew]

test

[severe ermine]

ping!

[eager yew]

should i do connectivity and networking tests later on

sounds fun tbh

[severe ermine]

pong :(

[eager yew]

this is starting to feel like macro abuse tbh

[twin escarp]

Hey! I like pong! XD

[eager yew]

low budget testing

"We have cargo testing infrastructure at home" ahh

[eager yew]

reminds me of rust's cargo test

[severe ermine]

[TEST]: RUNNING load gdt ...  PASS
[TEST]: RUNNING load idt ...  PASS

[eager yew]

i wish i knew how to check those things

i would absolutely add them

hmmm i wonder if i should keep all tests in one file

ykw this is funny

it'll probably grow to many thousands of lines but who cares

[eager yew]

got bored and added more to testing infra and more tests

it should not say all ok

oops

might add like timing and stuff tho

idk

ok probably should actually write tests instead of wasting time on this

less yappy now

[eager yew]

ext2 driver has decided it wants to find deleted directories

hmmmmm

[eager yew]

fixd it

nice 💥

the really nice thing about tests and testing NOW is that I find bugs that I would've otherwise missed that would've otherwise caused huge damage

like I forgot to check if a file was actually a file and not a directory before writing to it in ext2 and that caused insane filesystem corruption

but I found that bug with unit tests

yippee

it tends to be really stupid stuff like that that I screw up and not large issues

just forgetting an if statement and whatever

first degree skill isuse

[eager yew]

probably gonna go and refactor a buncha things now that i can be confident i wont just make the system implode with my unit tests

80 line functions 😬

not very good 👎

[eager yew]

shoutout to the lemonlime bootloader

[eager yew]

wow these are already starting too look so much more aesthetically pleasing

small functions my beloved

[eager yew]

i feel like i made this look worse

[steel herald]

i really hope this is under a mutex

[eager yew]

not yet

i'm almost around to implementing mutexes

cant say i'm a fan of this but it's fine

biggest spaghetti code ever

[stuck nexus]

You are way out of order

[steel herald]

Yah lol

[steel herald]

Also you're leaking memory on error

Because of try

[eager yew]

?

try will just attempt that operation and goto cleanup if it fails

[eager yew]

wdym

[steel herald]

So you cant use try in functions that don't have the cleanup label?

[eager yew]

it's specifically for this function

I really didn't like how repetitive it looked without it

but I'll probably go back since this might be super dumb

[steel herald]

Usually try is just like the rust question mark operator

[eager yew]

ah ok understandable

[stuck nexus]

You were supposed to do that loooooong before an fs driver

[steel herald]

also i wanted to see bread/brelse

[eager yew]

whoopsie

I'll do that as soon as I finish test writing then probably

[eager yew]

oh yeah I don't have buffer caches yet either thanks for the reminder

or block caches or whatever it's called

[stuck nexus]

You will be in severe pain for this

[gentle gazelle]

sounds like you're gonna skin him for this

[eager yew]

it's only about 3 thousand lines of filesystem code I'm not too worried

[stuck nexus]

I would claim that locking strategy determines kernel design

If you write code without worrying about locks you're making a blob

[eager yew]

I'll probably be able to fix this just fine over the next few days

this kernel is really small rn

[stuck nexus]

Generally speaking the real structure of a kernel is the structure of its locks

#1 design constraint maybe

[steel herald]

A what

[stuck nexus]

A dlob

[steel herald]

Who needs locks anyway

Either have big process lock or idk

Consult serenity code

[eager yew]

lock the whole computer so you technically are SMP but only one thing can ever run at once

🧀 biggest lock

oh yeah speakin of that scheduler stuff i made it properly go to idle and stop the timeslice stuff if it doesnt need it

i manually yield to tell it to go back to scheduling stuff

mild improvement

[eager yew]

mutexes from the dollar store

ok i guess i'll test these and then add em everywhere

[stuck nexus]

ok so that doesn't work like at all

unless you are doing a nonpreemptible uniprocessor-only kernel

Are u?

[eager yew]

no

what's wrong here

get_curr_thread gets the thread that the current core is running

probably a bad name here

[gentle gazelle]

what if you get preempted between the load of m->owner and the null check?

[eager yew]

oops yeah i forgot about spin locks

should probably do that

spin lock

probably still problematic

i'll go take a look at linux or freebsd's mutexes before i do anything even dumber

linux mutex docs are pretty good

[eager yew]

probably still broken

i'll test it

[stuck nexus]

I disagree with many things s about this

[eager yew]

how do i fix it

it worked in a test

[stuck nexus]

Why do you remove yourself from a queue

[eager yew]

I mean, i could have the scheduler just skip over blocked threads but i figured that might add overhead to my main scheduling loop when picking threads to run

so i figured removal and then re-queueing would be better

[stuck nexus]

Why are you still on a queue while running??

[eager yew]

oh wait i forgot about that

oops

yeah it isnt on a queue nvm

ok i fixed the problems the mutex works

probably still work to do

i know linux's mutex has 3 paths it can take or something

i believe it makes an attempt to busy wait and acquire

[stuck nexus]

mines 1200 lines so yeah

[eager yew]

yep 👍

[eager yew]

first mutex prototype done

time to read lots and lots about this

seems cool

lock convoys ⁉️

nah that's quite complicated

works ok on a few more cores

should be good

[stuck nexus]

its broken

[eager yew]

how come

every thread here gets a turn at the mutex

i think i'm not properly printing the core number of the thread i wake

something odd happening there probably

but every thread comes around and has its turn

[stuck nexus]

what happens when you scheduler_enqueue the thread before it scheduler_yields

[eager yew]

wdym

[eager yew]

I sure can't wait to directly copy the hashmap implementation from menix to my kernel to avoid reinventing the wheel so I can have a filesystem cache

thanks for being OSS menix

really appreciate it

[eager yew]

bread brelse

[steel herald]

[eager yew]

real

[eager yew]

where was that code from

I'm adjusting this for a hashmap now lol

[steel herald]

someone from this server designed this fs

[eager yew]

filesystem cache design

[eager yew]

truly peak

[steel herald]

yes

[eager yew]

you get to see your beloved bread and brelse soon

isn't that fun 🤩

[steel herald]

yea

[robust sorrel]

i need to implement this

[eager yew]

no more TRY 😁

fs cache going well

[gentle gazelle]

rip try

why not do a page cache for inodes?

[eager yew]

meh it's not that necessary imo and i dont want to build a page cache too early

[eager yew]

it works tho, the cache has sped things up a lot

integration test is down ~40ms

pretty decent i'd say

> git commit -a
[main 484983a] [ext2 + fs]: block cache
 19 files changed, 310 insertions(+), 228 deletions(-)
``` love to see it

now i gotta go through and use the cache entry locks everywhere

one pleasant side effect of this is that I no longer need a bunch of memory management code in the filesystem code because the generic cache essentially serves as the "memory manager" for all the buffers and frees and allocates them as necessary

I think I got down from about 30 calls to malloc/free to about 3-6 with the fs cache

nice

pretty good cache hit rates

I might turn my cache hashmap into a generic hashmap but idk, it's got some fs-cache specific features so it might not be "trivial" to do that, I'll probably just use a generic hashmap elsewhere and use the custom one for the fs block cache

[eager yew]

I think i might make batch evictions in the cache possible by introducing something like a maybe_evict flag

this way, when you call eviction the first time (probably before the batch eviction as it will only decide to batch evict under higher pressure), it will go and mark a bunch of entries as maybe_evict

this would probably allow the batch eviction to very easily just go and throw out all of the entries marked as maybe_evict instead of constantly finding the next-last-recently-used one

void fs_cache_decay(struct fs_cache *cache, uint64_t now) {
    for (uint64_t i = 0; i < cache->capacity; i++) {
        struct fs_cache_entry *entry = &cache->entries[i];
        if (!entry->occupied || !entry->value)
            continue;

        if ((now - entry->access_time) > STALE_THRESHOLD)
            entry->maybe_evict = true;
    }
}
``` maybe i could have something like this get called every now and then to re-check the cache

@steel herald we have dollar store bread

gonna put locks everywhere soon

[steel herald]

this looks like filesystem cache, not block cache

[eager yew]

it is block cache

[steel herald]

those are separate things

[eager yew]

wait what

[eager yew]

cache of filesystem blocks isn't a block cache?

o

[steel herald]

Welll

[eager yew]

now I'm a little perplexed

[steel herald]

at least on linux those are separate things

[eager yew]

oh ok well they're the same here

[eager yew]

what is it on linux

are blocks there just regions of disk or what

now we have low budget bread

[steel herald]

so how would this cache work for multiple filesystems with different block sizes on the same disk

[eager yew]

each filesystem has its own block cache

it's that shrimple

the block size is dynamic and generic

[steel herald]

yeah thats called a filesystem cache

[eager yew]

wait what

[steel herald]

infy@DESKTOP-IGT40G0:~/linux$ cat /proc/meminfo | grep Cached
Cached:          2175728 kB
infy@DESKTOP-IGT40G0:~/linux$ cat /proc/meminfo | grep Buffers
Buffers:           24312 kB

on linux

[eager yew]

what are the blocks in the context of linux

[steel herald]

Cached is filesystem page cache

Buffers is raw disk blocks

[eager yew]

ohhhh ok well this is already called a filesystem cache anyway so I guess that's ok in naming

[eager yew]

will I also need a block cache alongside my filesystem cache so that it goes and makes an entry in the block cache for a particular disk when it does a filesystem read/write

[steel herald]

so basically the idea is actual file data is stored in page cache

filesystem metadata is stored in buffers

The former is "Cached" the latter is "Buffers"

[eager yew]

wait so this would be like copying the file data to swap, right?

since that's kinda what page cache is?

[steel herald]

well a per-filesystem page cache is a pretty bad idea ofc, u want this to be a generic system that all filesystems get for free

[steel herald]

swap is exactly like a normal file, but it covers anonymous memory instead of file mappings

[eager yew]

it's not a per filesystem "page cache" it's just a cache for per filesystem block data (the data inside of blocks)

the cache is just a hashmap in memory

[steel herald]

u literally call ext2_*

will your fat driver call ext2_bcache_create?

[eager yew]

no it will have its own way of making caches

the underlying cache itself is generic but each filesystem uses it differentlu

[steel herald]

hyena would literally skin u

[eager yew]

huh

what am I supposed to do here

I don't get it then

[steel herald]

@stuck nexus can u explain pls

[eager yew]

basically, there is this structure

struct fs_cache_entry {
    /* allocated upon new reads */
    uint8_t *buffer;
    uint64_t size;

    /* number of this entry */
    uint64_t number;

    /* used with a counter - not a real 'timestamp' */
    uint64_t access_time;
    struct mutex lock;
    bool dirty;
    bool no_evict;
};

struct fs_cache_wrapper {
    uint64_t key;                 // block number
    struct fs_cache_entry *value; // pointer to cache entry
    bool occupied;
};

struct fs_cache {
    struct fs_cache_wrapper *entries;
    atomic_uint_fast64_t ticks;
    uint64_t capacity;
    uint64_t count;
    struct spinlock lock;
};```

each filesystem makes its own instance of this

and uses the fields as it needs

[eager yew]

why would I rather have a per-disk cache in memory?

[steel herald]

thats not what i said

so what does this cache store exactly

[eager yew]

the data of whatever is inside of filesystem inodes/blocks or whatever it needs

[steel herald]

so filesystem metadata

[eager yew]

kinda sorta, it's not metadata it's the actual data (for now) but yea

it sits in memory

[eager yew]

so what's the issue here

[steel herald]

https://elixir.bootlin.com/linux/v6.15.3/source/fs/ext2/inode.c#L1327 example

TLDR you want to have a centralized thing that works for everyone

instead of forcing every filesystem to do their own bolierplate and their own private cache

[eager yew]

but how does this benefit the system performance wise

[steel herald]

wdym

why are we talking about performance

[eager yew]

that's kinda why I did this

I wanted it to go a bit faster

[steel herald]

a centralized block cache is exactly the same in terms of performance

[eager yew]

so why would this be better

different filesystems handle things differently so I wanted to implement independent ones for each

[steel herald]

why is reusing code a good thing in your opinion

literally all filesystems have metadata stored in blocks

thats kind of how they work

they dont need to implement private caches every time

[eager yew]

hmmm ok well this should be pretty trivial to refactor into that

[eager yew]

so would this cache just store the LBAs of the blocks or what

[steel herald]

a centralized system also makes it easy to implement eviction on memory pressure

[eager yew]

ohhh yeah makes sense

thanks infster

[steel herald]

what linux calls page reclaim

[steel herald]

yeah

by default most ram (depending on the watermarks) gets eaten by page/buffer cache

then when memory is suddenly needed all the blocks are evicted and pages are freed

and linux for example has a thread that runs every second and writes out dirty blocks to disk

[eager yew]

is this cache per-disk or global global (as in all disks use the same cache in memory)

[steel herald]

the question doesnt make much sense

what are you even asking lol

different disks have unique data at the same lba

[eager yew]

oh wait yes it should be one per disk nvm

thanks for the help infster i shall go fix this

[steel herald]

https://elixir.bootlin.com/linux/v6.15.3/source/fs/buffer.c#L1453

if you're interested

this function retrieves a block from a disk device

basically this entire file is the buffer cache implementation

[eager yew]

hmmmm i kinda wanna call it bread

nah whatever

[steel herald]

thats the classic thing they use in unix

i kinda hate it because brelse sounds like "else branch"

[eager yew]

bcache_get, bread, same thing 💥

[eager yew]

so am i su pposed to call thiss a block cache

[steel herald]

well it is a block cache right

[eager yew]

hmm i guess it is now

yea ok thanks

[steel herald]

an anonymous LRU cache of disk blocks referenced at some point

not belonging to any file

[eager yew]

all done, thanks for the suggestion infster

[steel herald]

cool

[eager yew]

bcache

no bread 🍞

[steel herald]

why cant bcache_get handle this transparently lol

[eager yew]

I thought i might have cases where I need to handle things differently and not evict, idk tho, i will probably fix it up to do that

i just thought it would be confusing to have a get function also decide to call evict when it's named get

no big deal

[steel herald]

what

im confused

thats kind of the entire idea of an lru cache

[eager yew]

nvm yea i can trivially change this

[steel herald]

it drops stuff transparently

u can take the entire code from line 18 to 26 and paste it into bcache_get

thats literally how its supposed to work

[eager yew]

👍

[steel herald]

u can even have a generic superblock struct like linux, then u dont even need ext2_block_read

because u can just do bcache_get_fs_block(fs->superblock, block_idx)

and superblock contains all that math in lines 8 to 10

because all filesystems work like that anyway

[eager yew]

yea i was planning on doing that too, might do that once i finish the FAT VFS function implementation

[steel herald]

at least like a struct fs_coordinates idk lol

[eager yew]

what are coordinates here

like where the fs is?

i mean i have a partition struct

generic_partition

will probably do generic_superblock

[steel herald]

base block, number of disk blocks per fs block

so that generic block cache can do disk requests with correct offsets without you handcrafting the math in every filesystem

[eager yew]

bread fix

[steel herald]

thats better

[eager yew]

thanks infy for all ur help with my exceptional stupidity 👍 i honestly really appreciate this community for that

[steel herald]

static inline struct buffer_head *
sb_bread(struct super_block *sb, sector_t block)
{
    return __bread_gfp(sb->s_bdev, block, sb->s_blocksize, __GFP_MOVABLE);
}

Its funny because thats literally how sb_bread works

[eager yew]

i copied thems...

[steel herald]

it takes s_blocksize from the superblock

[eager yew]

gpl 3..

[steel herald]

nah i just mean its definitely the correct thing to do

[eager yew]

is __GFP_MOVABLE saying whether or not it can be evicted?

not sure what movable would mean here

[steel herald]

this is for memory compaction

movable means it can be transparently replaced by a different page

because there's nothing referencing this address via the HHDM for example

all userspace allocations are movable by default

if you know C# it has stable pointers and all others are movable by default

same thing here

[eager yew]

now the real trouble is dynamically resizing the silly block caches

[steel herald]

you should probably store them at least in pages, with some amount of readahead

[eager yew]

i mean the actual expanding isnt the issue, it's knowing when and how much to expand/shrink

idk that will be figured out as i make this project

[steel herald]

u expand forever

[eager yew]

limitless hashmap?

oh ok

i thought i needed some set max size

[steel herald]

well look

u can set a watermark to e.g. 80% of ram

but u dont have to

[eager yew]

ah ok that makes sense 👍

I buy all RAM I use all RAM

got it

[steel herald]

the key part is that your PMM can go and evict everything with refcount == 0 if it needs the memory

[eager yew]

so each cache entry would get its own page?

[steel herald]

yeah i would store them as a collection of aligned blocks

so e.g. if someone requests block 18 you go read 8 blocks starting at block 16

[eager yew]

so during caching, i could mark a bit or something on the PTE saying "hey these pages are available for eviction"

then, the page evictor will remove the block cache pieces, and the next time I try to access that thing in cache I can just load it from disk again?

sounds alright

[steel herald]

nah

pmm_alloc():
   pfn = alloc_fastpath();
   if (!pfn) {
        block_cache_lru_evict();
        pfn = alloc_fastpath();
   }

   return pfn

you block cache should have an lru as well as a hash table

so u can walk the lru and discard least used blocks

[eager yew]

yes it does have that

ok neat

[steel herald]

linux stores the lru as part of the struct page i think

so they're linked to each other

[eager yew]

sounds awesome sauce thanks infster

[eager yew]

good luck with your OS I hope you get to "gdt init... ok" soon

[steel herald]

thanks

im not really trying to speedrun so idc

[eager yew]

[TEST]: 164 block cache hits, 11 block cache misses decent ig

[eager yew]

forgot that it would be smart to just page align these block cache buffers so that the drivers dont need to allocate their own and memcpy a bunch, ez fix tho, noticeable speedup too

hmmm the maestro page cache thing looks kewl

might take a further gander at it

[eager yew]

forgot that CDROMs can mount via AHCI

fun, i get to write more drivers for that 💥

[twin escarp]

Looks fun! I'll get to do disk stuff.... eventually

Also I didn't know that lol

[eager yew]

ya i think these are the fancy writable ones, indeed, there are Writable devices that are referred to Compact DIsk Read Only Memory devices

truly a misnomer of all time

wait i think the standard ATAPI ones are also writable too

yea nvm lol

writable read only memory 👍

[eager yew]

if u want disk advice, you should probably start off with virtio and do NVMe next

buuuuuut you'll need that fancy microkernel driver interface first

[twin escarp]

I was debating whether or not I'd do virtio or just jump straight to NVMe (since I can't use virtio on hw)

[eager yew]

going from no block device driver experience to straight to NVMe is doable but you might wannna consider starting off on something easier to understand basic concepts shared across devices

logical block addresses, sector sizes, stuff like that

[twin escarp]

ahh okay

[stuck nexus]

you are also wrong so u have made a grave error in pinging me

but ur lucky this time i feel like SHIT and dont have the energy to type a wall of text about this topic

[eager yew]

is this more correct

page cache - file contents, for r/w operations
buffer cache - block level file metadata

or what

whatever

i think i mightve just said what infster said

ohh bcache allows faster storage to act as cache for slower storage it seems

bcache (abbreviated from block cache) is a cache mechanism in the Linux kernel's block layer, which is used for accessing secondary storage devices. It allows one or more fast storage devices, such as flash-based solid-state drives (SSDs), to act as a cache for one or more slower storage devices, such as hard disk drives (HDDs); this effectively creates hybrid volumes and provides performance improvements.

from https://en.wikipedia.org/wiki/Bcache

https://tldp.org/LDP/sag/html/buffer-cache.html

By reading the information from disk only once and then keeping it in memory until no longer needed, one can speed up all but the first read. This is called disk buffering, and the memory used for the purpose is called the buffer cache.

Most operating systems have buffer caches (although they might be called something else), but not all of them work according to the above principles. Some are write-through: the data is written to disk at once (it is kept in the cache as well, of course). The cache is called write-back if the writes are done at a later time. Write-back is more efficient than write-through, but also a bit more prone to errors.

oh i think i've created a buffer cache then

block cache seems to be

take thing from slow disk -> put it on fast disk

[steel herald]

no thats exactly why i pinged u

i only have a vague idea

[stuck nexus]

you dont need to have both

the NT way is to only use a page cache to cache both file data and filesystem metadata

however this has complications

if you have a page of disk data

a 4kb page

with 512 byte sectors

there are 8 sectors on disk within that page

if you use the page cache to cache the disk as just like a linear bag of bytes

and theres mixed sectors of metadata and file data within a page

then you might have that file data in two places: as the raw sectors within that page of disk cache and also within the file's own page cache

this is called "cache aliasing" and is bad

imagine a scenario where you write to the file and dirty a page in its cache

and theres also a dirtied page of metadata that happens to contain the sector of file data you modified

if the former is written out, and then later the latter is written out, then the result is that the old file data is still on disk and is different from whats in the file's page cache

that kind of thing

[steel herald]

whats the solution?

[stuck nexus]

NT avoids this by having disk metadata writes go through the filesystem driver which gives it a chance to "mask off" file data sectors from the write before passing it on to the disk driver

some filesystems dont freely mix metadata and file data like FAT and theres just a hard cut-off between the two and then its easier because you can check if a metadata write straddles that boundary and truncate it

[steel herald]

i wonder how linux solves it tho

[stuck nexus]

theres another fancy solution that im starting to prefer which is to actually just use a buffer cache for disk metadata, but the buffer cache is a management layer atop a bag-of-bytes page cache for the disk

which keeps a bit mask of which sectors within each page were modified

and does the masking-off of the writes for you

so its basically a page cache with a special writeback mechanism; it doesnt go through the same writeback that other pages do

i think this is (essentially) what the modern kernels aside from NT do

[steel herald]

ah

[stuck nexus]

writeback is specialized for these pages, but reclamation isnt

[eager yew]

I currently only cache LBAs in the per disk cache and the cache entry is just a buffer of whatever is at that LBA

writing is write through for now

[stuck nexus]

other than you might have some special buffer cache metadata to clean up and free and whatever when you reclaim a buffer cache page

[eager yew]

I'll probably also mark these cache buffer pages with some special bit among the available pte bits so eviction is easier

later on I will probably implement write back

how does linux do write back? I swear i saw something somewhere about a per-second sync event to write back all dirty cache entries

that seems a little bit strange to me though

[stuck nexus]

from what ive seen the NT filesystem drivers accomplish this by reusing a structure they use to cache the mapping of file offsets to disk offsets

which is like an AVL tree of on-disk extents

this happens to also be perfect for the purpose of representing which sectors on disk are metadata or not

they add sectors to this "mask" (really an avl tree of extents) whenever they modify metadata on-disk

[eager yew]

is this reusable for block based filesystems that don't have extents

[stuck nexus]

this has nothing to do with whether the on disk fs uses extents

extents are just used because its memory efficient

if you add sector 5, 9, 6, 8 in that order

you only need to track 2 structures, not 4

because 5,6 8,9 can be represented as extents

[eager yew]

oh so they're just used to track things on the disk rather than in the filesystem ok sounds great

I thought "NT filesystem drivers" was talking about the NTFS filesystem

ok great

[eager yew]

holy smokes some of my driver code is spaghetti

in the true spirit of osdev driver code :^)

gotta fix it up for sure 💀

[stuck nexus]

look up netbsd's unified buffer cache

[eager yew]

optimized the cache a little more, filesystem tests are 7% faster in the VM, dunno if this translates to real hardware or not tho, i'll probably benchmark these eventually

i think i've cut the time in half, so it only takes about 40% as much time compared to before i added the cache, so that's alright i guess

i wonder if i can change device driver waiting event loops to use microseconds instead of milliseconds

that's probs a big one

hmmm i'll see lol

wow ok that brought it down by an unreal amount

[1   ]: ext2_integration_test...  ok  (10 ms)``` lmao this was taking 50 ms prior to that

nice 👍

[steel herald]

Or ya know, use goddamn interrupts

[eager yew]

i'll do that later

i was gonna do that too

[steel herald]

Poll mode is only used for super fast hardware when the cpu is relatively free to do shit faster

When each request takes microseconds usually

[eager yew]

ok i'm tryna figure out how to refactor things to do IO and stuff with interrupts now, I think i have everything in place i just need to track a few contexts so i wake up the correct guy when the interrupt fires

struct nvme_wait_context {
    struct thread *waiting_thread;
    bool completed;
    uint16_t status;
};
``` might do something like this and then match the CID to figure out which thread should be woken up

this way, interrupt fires -> i check thte CID of the completed request -> i look in table to find thread to wake up -> i wake up thread -> i tell scheduler "hey bud, please run this thread ASAP"

something like that....

i think a bit of a trouble i might have is if people decide to suddenly unplug the drive because all threads that are waiting on that drive would just never wake up

[molten night]

You'd want to set up a timeout anyways no?

[eager yew]

welll.... if i go the route of having interrupts (which is what I'll be doing), and the interrupt never wakes the thread that is blocking on the device, the thread just kind of stops running again

I dont know if it's viable to just check all blocking threads with a timeout, that seems a little bit unnecessary? I guess I could somehow figure out a way to keep the timestamp of when the thread started blocking and then occasionally run through and see if a thread has been blocking for something absurd like 10 seconds and then just force it to die

i dunno....

maybe a potential way to do this is

record timestamp of most recent blocking event start -> occasionally check threads -> see if what they are blocking on still exists -> end the thread if what they're blocking on suddenly disappears

idk

[molten night]

Devices are my absolute weakpoint, so I don't speak with confidence, but to me it sounds like you'd want a timeout to be able to gaurentee forward progress. You don't have to be accurate too, just every once in forever run a timeout check

[eager yew]

so maybe i could once-per-second or longer check all blocking threads and make a ping to whatever they're blocking on to see if it exists?

i dont really know how to go about this

i know people will probably never just randomly unplug a SATA disk from their computer tho lol

but if they do i dont want the machine to just have processes that never wake up again

[molten night]

An easy solution is to have your forever long enough that any wait for this period is unreasonable, unless you cant gaurentee such time

[eager yew]

but the trouble is, if for some reason there is a comically slow device i dont want to all of a sudden go "this thread has been waiting for too long, i feel like killing it" because then that could cause unexpected behavior too

I just need to try and figure out some way to ping and see if a device still exists and only remove a thread if the device suddenly vanishes

[hard talon]

if its slow enough that "forever" isn't long enough aren't there bigger issues at hand at that point?

[eager yew]

sounds like a problem.... for future me 💥

i'll have a think about this

[eager yew]

made nvme raise an irq thingy

tryna make my polling things go away (it kinda dum)

[eager yew]

i got it to catch the interrupt

now to implement the fun stuff

nice

[eager yew]

apparently the "spawning a worker thread to go do work every second" thing that I thought was strange is common and reasonable

I might get that up and running to evict things from caches and stuff

[stuck nexus]

wdym

[eager yew]

well, I thought it would be strange to have the kernel spawn a worker thread at intervals like that, but I looked around in actual OSes and found they have something similar

it's not necessarily fixed to one second, but for example linux has kworker, kswapd, etc.

so I figured I might spend a little bit to just write short functions that can run periodically to clean things up or make the system run a little better

also how come you follow this thread but not others lol, is this a random choice

thanks for all the help tho mister sky 👍

[twin escarp]

I follow it 🥺

[eager yew]

snowos ftw

[twin escarp]

...uh, wut does ftw mean XD

[stuck nexus]

ik my irl last name is public on my github profile but dont call me by it

scary

[eager yew]

oopsies

[stuck nexus]

normally you dont SPAWN IT each second, you spawn it once and it sleeps until theres work to do

[stuck nexus]

someone probably pinged me here at some point and now i see things here

[eager yew]

ohh ok, my concern was how I would implement heuristics to tell the threads when they should go do work, but that should be trivial

sometimes I see my laptop suddenly get an influx of a dozen or so kworkers and the fans spin up real bad out of nowhere, I think that happened when kde decides to index the filesystem

[stuck nexus]

its virus

get neew laptop

[placid vessel]

btrfs moment

[eager yew]

as a fellow btrfs user, do you use btrfs outside of VMs and your personal computer

I don't use btrfs on my NAS

xfs over there

[placid vessel]

marvin@aquaduct ~> mount | grep btrfs
/dev/nvme0n1p2 on / type btrfs (rw,relatime,nodatasum,nodatacow,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
marvin@aquaduct ~> 

[eager yew]

yep 👍

[placid vessel]

i use btrfs on everything

[eager yew]

has it ever been a problem

[placid vessel]

no

[eager yew]

I was a little concerned that data loss might happen with it or other shenanigans

oh ok

[placid vessel]

quite the opposite

[eager yew]

neat 😎

cow snapshots and compression

[eager yew]

we have lock_buffer at home 🥀 💔

I get to have the very Fun, Pleasant Time of implementing writeback in the buffer cache 😁

[eager yew]

oh yeah i'm doing the silly interrupt thing

going good 👍

[eager yew]

going good

[eager yew]

LMAO the code behind this is INSANELY cursed but the tests now work with the scheduler and in multithreaded mode

this is horrendous

downright diabolical

but it works 👍

NVMe with ISRs works fine but extremely extremely slowly

now i just gotta find the source of the slow and we're all good

ok the slowness was literally just because the idle task existed lol

all gooood

sweet

ok time to fixed the cursed horrendous register saving

i fixed it

ferrisButGrimacing

the tests are marginally slower now but this is probably because this is a VM and on real hardware i am saving time and pain

CI is failing

their cooked ubuntu version's assembler doesn't support a thing I used

[eager yew]

also courtesy of the most gracious infy for reminding me of this

giving credit where credit's due fr

[eager yew]

CI is passing

I rewrote the att asm to intel asm and it's fine

intel syntax wins again

muahahahaahahahaha

[placid vessel]

imagine not having intel syntax by default

[short lichen]

This

[eager yew]

a kitten named lizzie gave me macro tips using att syntax so I used that and then switched back to the saner syntax

[eager yew]

probably gonna finish xhci soon now that I have proper interrupt driven io handling

[eager yew]

nice

[severe ermine]

nice

[severe ermine]

C++ includes module interface files too

[eager yew]

fancy language

[severe ermine]

Indeed

[eager yew]

ahci interrupt driven IO

gonna do IDE and ATAPI later prob

[steel herald]

an ahci driver should really not know anything about the scheduler

let alone switch the state of the current task

you should have a block io packet struct

that is submitted to a block device

and that can be waited on

so that the submitter has a chance to do some extra work and not even get blocked if the request completes fast enough

[eager yew]

wait wdym by having a block io packet that can be waited on

ohhh I understand

ok thanks infster

[eager yew]

ok i have mostly figured this out, I'm gonna introduce async read and writes into my generic_disk abstraction but still make the default reads and writes fully blocking, so if i need async or blocking i can pick

[steel herald]

Blocking is just async that waits right after submission

[eager yew]

yes

block device gameplay

[eager yew]

💥

[eager yew]

something something async io something something

🧀

this will be great help in my block cache for prefetching perhaps

time to implement bio stuff for the rest of my block devices 🧀

[steel herald]

but like

why cant this be a generic function

there's 0 ahci specific code here

other than that well done

[eager yew]

i only have this on ahci right now lol

once i do it for other drivers i'll make it generic + introduce abstractions for that

[eager yew]

thanks infster you've got the most goated suggestions 🙏

truly a saint

[placid vessel]

i read finnster

[steel herald]

np

how do u unit test btw

[eager yew]

it's really silly, I just have two symbols in my linker script .kernel_test_start and .kernel_test_end, and all i do is I insert struct test s into that region and have a runner that goes through that region and executes the functions for every test

i couldnt figure out any better way and this was simple

void tests_run(void) {
    struct kernel_test *start = __skernel_tests;
    struct kernel_test *end = __ekernel_tests;

    uint64_t unit_test_count = 0;
    uint64_t integration_test_count = 0;
    for (struct kernel_test *t = start; t < end; t++) {
        if (!t->is_integration)
            unit_test_count++;
        else
            integration_test_count++;
    }

    k_info("TEST", K_TEST,
           "running %llu " ANSI_CYAN "unit" ANSI_RESET " tests...\n",
           unit_test_count);

    run(true, start, end);

    k_info("TEST", K_TEST,
           "running %llu " ANSI_MAGENTA "integration" ANSI_RESET " tests...\n",
           integration_test_count);

    run(false, start, end);
}```

literally

and yeah yeah i know double underscore is undefined behavior because compilers reserve it

[steel herald]

makes sense

ill probably have test modules for that

[steel herald]

so your kernel logs it?

[eager yew]

ya I log it in the run function by having each test have a success field and a should_fail field and a skipped field

so once the test function returns it just checks those statuses

struct kernel_test {
    const char *name;
    test_fn_t func;
    bool is_integration;

    /* TODO: fancy state machine with enum? */
    bool should_fail;
    bool success;
    bool skipped;

    uint64_t message_count;
    char **messages;
} __attribute__((aligned(64)));```

you can also report back messages by adding them to the messages array and incrementing the message count

[placid vessel]

wow ok