#chaos's lilkernel (no LLM use) - a reference implementation for how a kernel should NOT be written

1 messages · Page 2 of 1

charred surge
#

that heap arena will have the 2mb bootmem
theeeen
after initialising everything there i would make pmm rely on slabs that gets backed by the heap arena

the heap will later expand itself with another span that covers the actual kernel heap region

and I'll have the pf handler back it with memory as needed
afterwards the pmm will get initialised

now the pmm itself doesnt rely on slabs for managing its free tags storage
instead it'll do it like vmem
aka
have a callback that snatches a page, maps it then initialise it as tags
and uses few of these to record everything

now all this might introduce an infinite recursion

say
vmem on heap has to allocate a region or maybe a span -> uses its last tag, bc a perfect fit wasnt available or to commit that span to the tags list, next time it'll have to allocate and back with memory -> pf on that region, so pmm is called to back it with memory -> pmm is out of tags, allocates a page -> calls the heap vmem to request a new page -> they dont have tags to record it -> pmm ...

sooo
to solve this I'll sort of have tag blocks on vmem leave few tags for like
emergency
which only the pmm can signal that they have to be used, on its own call to vmem

im still not fan of this solution tbh
but at least I wont pollute the vmem layer itself with it and Instead have this case local to a tag block creation callback function or whatever that applies to only the heap region
other vmem arenas will have callbacks that doesnt interact with the pmm and instead will use the heap or idk

#

I'll rethink that again ig

charred kraken
#

pmm rely on slabs that relies on heap?

#

what does the heap rely on

#

the pmm?

charred surge
#

it doesnt rely on slabs

charred kraken
#

u just said it does...?

charred surge
#

.-.

okay
pmm uses tags
and it manages them internally
it just relies on the heap arena to provide it with pages
as if its a slab allocator
so it kinda is
but it grows its blocks/slabs differently
not through the slab interface
whichs severely suboptimal but eh

so
in order for it to grow its tags it will carve out some page, get it mapped and now poof new block of tags
it can then record everything and life is good
but
in order for it to get that page mapped it should request a region from vmm
which could have ran out of its own tags too and for it to replenish them it'd need to go through pmm
bad
so instead
each vmem arena will save 2 of its tags for expanding its own free tags list
so that it doesnt need to go through the pmm just to expand them

charred kraken
#

ur pmm isn't a page frame allocator?

charred surge
#

it is

charred kraken
#

wouldn't the pmm normally provide the pages to back the heap/slab?

charred surge
#

it does

charred kraken
#

but it also gets memory from the heap/slab?

charred surge
#

for its own internal structures it needs some mapped region to hold them?

#

and that mapped region would need to grow

charred kraken
#

and does pmm alloc/free possibly call to allocate these structs from the heap/slab?

charred surge
#

calling to the pmm again to back it with pages

charred surge
charred kraken
#

this just sounds like a recipe for deadlocks

charred surge
#

how else should that work :P

charred kraken
#

generally u allocate page structs for every page in the system at boot

#

that's normally all the pmm needs

charred surge
#

mmmmmmmmmmmmmmmm
okaaay
but i thought this'd be wastefull..

charred kraken
#

well

#

linux does it

#

NT does

charred surge
#

unless i do something wild like
not back THAT region with memory unless it becomes in use

#

but then again thats the same issue

#

which i still think it might be done without resorting to the whole "set it all up and forget about it"

#

idk

#

ill try my wierd thing then if it failed miserably which im sure it will i might fall back to do this

charred kraken
#

page structs for every page makes the most sense

charred surge
#

i thought about it a while back
i was told it will have something like
~8% overhead for describing the whole memory
which shouldnt be bad taking into account the advantages
but idk i wasnt convinced :P

#

i will still look into that again

#

what i'd do
in the context of the vmem paper
would just move the responsibility of the "tags thats used to allocates more tags" from the new block (they say that new block can use few of its own tags to record/link itself iirc) to the existing block

#

actually
ig it'd still fail 💀

#

hmm

charred kraken
#

it doesn't have to be 8%

#

besides if you don't need to store flags for pages in-use

#

an intrusive freelist is 100% free

#

and O(1) on both ops

#

and like 15 lines of code

charred surge
#

i opted for a buddy based pmm for 0 reason really lmao
and i kinda regret it

charred kraken
#

I have a buddy

charred surge
charred kraken
#

it doesn't

#

period

#

well

#

not exactly

charred surge
# charred kraken I have a buddy

ooooo
so u have that page frame db thing and use them for as entries in the buddy lists? o_O
how do u keep track of the used pages then

#

wait ig dumb question :P

charred surge
# charred kraken not exactly

mmmmmmmmmmmm
i think they can but through some bitmap layer on top of that eh
so u can relatively cheaply keep track of whether the 512 pages starting at some address are all free or not
well actually thats a silly approach
like
u can still do some magic to support hugepages still but idk how

charred kraken
#

my buddy map just uses a list_node (two pointer, 16bytes) + 1 byte for some flags

#

per page

#

would be possible with just one pointer too

#

that's 0.4% of mem

charred surge
#

ooo

#

btw i didnt know ur kernel was so impressive
i just checked there and :O

charred kraken
#

it runs nothing 😭

#

lol

#

tbh I could prolly impl lot's of syscalls atp but there are still subsystems so flimsy that I wanna fix more

#

last time I rushed syscalls and shit ended in disaster

#

but thanks 🙏

charred surge
charred surge
charred kraken
#

I spent the last week implementing an init system that I am currently tearing down

charred kraken
charred surge
charred surge
#

at least u get work done lol

#

me here be yapping on discord instead of writing code chad

#

(i also got another exam in 2 days so i shouldnt probably write code for chaos rn either)

charred kraken
#

here's an example

#

this was generated from my kernel elf

charred surge
#

i can barely understand the problem 🙏
but like
an init system in this context is like the decision of which parts are supposed to wait on the initialization of which
right?

#

.-.

charred kraken
#

my impl was these target hooks that declared a name, the function to run, what objectives it provides and what it depends on

#

ik for example astral has a similar thing

#

but it's used a bit differently afaict

#

in my case the targets were very small and just became a mess

#

I an resorting to a normal init function and some hooks that I run at specific times

charred surge
#

i didnt know any of that was a thing 🙏
i live in the blessed valley of ignorance fr

#

i kinda only knew the primitive way of having main calls init functions one by one and u just figure out whichs supposed to be placed before which

charred surge
charred surge
#

introducing..
the fucking mess™

image.png
#

i mean there's more into vmem_alloc than that ofc
this is just about the tags_grow callback hell
whichs 100% contrived (and absolutly unnecessary)

#

i wouldnt make the memory backing these tag blobs pageable (if i ever reached a point where i have swapping or whatever)
so hopefully there will be no need to touch it again
or idk if that would make any difference anyway

#

in the vmem model this part can theoretically be repeated 43258435934573 levels down
which will never be the case since the heap arena will directly use the high memory as a source..
but thats why "we add to a list of pages to be backed" anyway
and since this situation is the cold(est) path and should only very rarely happen its not that big of a deal (?)
also it should simply just lock down the pmm globally until we are back again
and the pmm tags dont really have to grow one page each time
it can grow more bc they are used a lot
so i thinkkk this is solid enough..

image.png
#

i swear i always come up with the most abhorrent solutions ever chad

charred surge
#

chaos is still alive

#

its just the uhh
exams
I might get some work done over the course of the next 2 days

charred surge
#

bit sized spinlock trl

260115_19-48-04-10665.png 260115_19-47-45-14428.png
charred surge
#

with suboptimal solutions ofc ✨

#

wait lemme review whats been achieved bc i forgot 💀

#

tmrw we are free chat

#

only for 3 weeks tho

#

but today we arent nooo

charred surge
charred surge
heavy forum
#

bit sized spinlocks are not really useful unless they are in a wider bitmask of some sort

charred surge
#

mmm im not sure that I get what u mean here

heavy forum
heavy forum
#

otherwise they're just gonna be slower for no reason

lavish onyx
heavy forum
#

no that was an example

charred surge
# heavy forum otherwise they're just gonna be slower for no reason

oooo so i understood it correctly
I wanted to use them for a struct page unironically yea
bc i have like
an address
buddy node metadata
refcount
and that lock bit all smashed into one 64bit field
then few other fields for intrusive tree node
just so that it all fits under 32 bytes 😭

heavy forum
#

ah yeah thats fine then

charred surge
#

but rn i just have them there
havent used them yet but that was the plan

lavish onyx
#

You’re actually good at kernel internals

#

Good shit

#

🔥

charred surge
charred surge
lavish onyx
#

That’s why I read so much

#

Because I forget shit

#

I’ve read the SDM countless times

heavy forum
#

at some point I considered adding a bitmap of spinlocks for fine-grained locking over a multi level data structure but that ended up too slow and not better than just having a single top-level spinlock and having multiple instances of that structure, which is why im telling you bit-sized spinlocks arent magic

heavy forum
charred surge
lavish onyx
lavish onyx
#

Will is goated

heavy forum
#

will is hyenasky

#

😭

charred surge
heavy forum
#

well the intent was to have granular locking but a bitmap of locks like that doesnt really work

charred surge
lavish onyx
#

I was just saying his project

heavy forum
#

so i ended up with the solution of having like 16 buckets with tree nodes and locks that are accessed by hash

lavish onyx
#

I use hash tables to store my slab-like objects

#

Hash tables my beloved

charred surge
#

ot maybe im misunderstanding shit here

heavy forum
#

yeah

charred surge
heavy forum
#

and also you have to do a cmpxchg on the whole byte anyway

#

im pretty sure

charred surge
charred surge
charred surge
heavy forum
#

the obvious solution is to have bit sized locks

#

but spaced by 64 bytes

charred surge
#

(edit: i first read it as "bits".. what...)

true.png
charred surge
sour raven
charred surge
#

okay so I had at least 2 things on my today's todo list and I did absolutely none of them

#

instead I kept bikeshedding on discord ✨

#

truly fascinating

charred surge
#

but I dont think it shouldnt work tho

#

ive went through some issue tho where I figured out mapping a page would itself require at most n pf allocations
all would in theory take at most a couple of dozen tags 💀
so i came up with another ugly solution where the pmm will have some side path for stealing shitton of pages without actually committing any of that into the pmm structures, then later on it can commit everythijg
since it will do all that with the pmm locked out anyway it shouldnt be an issue
it just defers the cost of registering shit after it starts to use it kekw
it would incur almost the same overhead anyway it just batches it

#

so thats one more very rare worst case path

#

everything else is supposed to follow the fucking mess™ model

#

another point is that for pmm tags I belive i should allocate tag blocks in more than a page sized units

#

since they are used extensively

#

whichs unironically one step closer to having pfdb pain

#

but whatever

#

just to minimize the need to go through the aforementioned code path

#

it should be easy anyway
and doesnt seem that necessary
so i'd do it later

charred surge
#

okay chat we made it out alive this time

#

not vmem but academic-wise trl

#

now I can get back to vmem (waiting for the grade pinned the fuck out of my attention even though i knew beforehand the hour on which It was gonna get released, silly autistic brain momento)

charred surge
#

eh yk what
fuck it im taking today off trl 🙏

upbeat stirrup
#

Im taking that title from you

charred surge
#

nuh uh

Screenshot_20260122_005131_Chrome.jpg
charred surge
#

okay soo update on the fucking mess™

#

i reached a point where it should work

#

in theory

#

ofc none of it is tested and the vmem part is WIP

#

but

#

it has kinda proven me that it would work

#

BUT

#

its just ugly asf

#

so its more like

#

and by ugly here i mean really ugly

#

slab code shoved into pmm kinda ugly

#

vmem having pmm code and vice versa-ugly
and violations of the locking scheme through dumb workarounds

#

which's yk

#

unacceptable

#

wrong

#

harmful to society

#

and frankly un christian

#

like

#

if our father god almighty descended from heaven right now

#

and looked down the humble repo

#

he will just be like

#

so

#

with that being said

#

i decided to give up on it aaaaand succumb to the pfndb propaganda

#

i will still only back it with wired physical frames as they get referenced

#

this will like shrink pmm code 25% unironically

#

and get the rest of the affected components back to a sane state again

#

so that i can finally be done with memory management and fucking move on

#

i will also be giving the rest of the code a quick cleanup if i have enough time

charred surge
#

okay this's basically a complete rewrite

charred surge
lavish onyx
charred surge
#

its funny cuz im basically going in circles right now

#

i just wanna get over memory management stuff dawg 🥀

#

but naaaah my dumass must keep running into the pmm -> vmm -> pmm again -> vmm again -> pmm again -> ... cycle

#

also we are working on total fire chat

#

a pmm system with 0 tradeoffs

#

bc it performs equally terrible for all testable metrics

#

✅ wastes memory
✅ slow asf
✅ no constant time allocation
✅ single threaded with Big Beautiful Lock™

charred surge
#

to put this into perspective btw
it will still be a buddy allocator
but the fuckery lies within the pfndb part
im gonna make the pfndb virtually compact (and perhaps position independent lmfao), to some extent regardless of the gaps in the memory map
and so far i still have plans on backing it with memory on demand actually (havent given up on that yet, despite the self proclaimed rumors :P)
and at init time
i will construct an array of segments # -> segment base (virtual and physical)
with some read only radix tree of depth that depends on the number of segments and the ranges they describe for mapping pfn ranges to array indices
i might store the index thingy into page_t's as well for fast mapping, would actually be better than storing the pfn itself there i guess o_O
and therefore a page_t from a pfn will follow a path similar to how virtual -> physical address calculation works, tho with far less hops
while a pfn from page_t will be just segment # -> index into an array -> segment physical base + page_t offset from the virtual base

#

now pfndb segments are gonna be virtually aligned by their largest order block to page alignment

#

with sizeof(page_t) being a power of two

charred surge
#

and consequently a block fitting into a page will have all its broken page_t's fitting into the same page

charred surge
#

as a result of how buddies work (original blocks being power of 2-aligned)
and how random memory gaps are
a worst-case memory map segment will first get its original buddy blocks initialized that way

image.png
#

when the page_t of the largest block is page aligned, any break down of it to an order less than the ratio of page_size/sizeof(page_t) will mean that the resulting 2 blocks will reside in a different page

#

this will also ensure that
any smaller block on either of its sides
if its of an order >= page_size/sizeof(page_t) it will also be page aligned too

#

with the same conclusions appling to it

#

that conclusion btw, that will get us to know whether a pair resulting from a break down will reside on a different page or not

#

will allow us to know whether we need to map a page there or not

#

and similarly any merge upwards to an order larger than page_size/sizeof(page_t) will mean that we can get rid of the physical pages that used to back the mapping of the other half

#

now this can only go wrong if i didnt figure out a way where this doesnt loop back into itself since
we will need to allocate frames to back the frame that we need in order to back the structures that we will use to allocate the frames trl

#

but i think i can figure out some way similar to the page stealing clownness i had previously

#

since..
there is an upper boundary for pages we will use to map a page anyway
where that page will be enough for us to back page_size/sizeof(page_t) pages already

#

as for the page_size/sizeof(page_t) ratio btw
using normal pages
should be 128 for 32 bytes page_t's
but i believe i can bring that down to 16 bytes somehow
resulting in 256 page_t's per page
making us required to allocate and map a new page when breaking down segments of an order of 8 (+12) instead of 7 (+12)

#

as for the "✅ wastes memory" bullet point
its due to the fact that virtual segments will be aligned by their maximum block
not their start address
so for the worst case, we will be wasting 2x 4kB for a frame on both sides instead of one at the partially used far end

#

also the fact that we will use normal pages to back the mappings is wasteful too
but i might rethink that part later, optimizing this POS of a system is futile anyway chad

#

locking can be improved too

#

but ig i should first write the thing and see how it turns out first

#

Plan B here would be just to fall back to a boring statically mapped pfndb with virtual gaps mirroring the physical gaps and all

charred surge
charred surge
#

behold
The Pathetic page_t™

image.png
#

16 bytes including an intrusive doubly linked list node btw

#

sad that the locks cant be at least one cacheline apart but perhaps i can (in the far future) optimize the buddy algorithm to take into account some sort of cache coloring of page_t structures somehow

#

but with these 16 bytes we get:

  • per page_t lock
  • 16-bits reference count
  • 2 flag bits (used and wired)
  • double links for Fast AF™ list ops
  • and only adding one extra memory hop into a ro array as a tradeoff for the virtual compactness of the whole thing
#

not a bad deal

#

wait a sec i missed the wired flag pain

#

better

image.png
#

i believe that if it turned out to be necessary, allocating a non intrusive node for linking used pages on some list for a swapping daemon shouldnt lead to circular reference into pmm, since it wouldnt be done by it, perhaps even much later
but i dont really have to do any of that
i can even carve out more bits out of that 36 bit segment index since like
if someone's physical memory map was mode of 2^36 fucking pieces each is exactly 1 page sized i guess they have more to worry about than to get my humble kernel running pain

#

sooo i can move the 8 bits taken from a free page's refcount to a permanent offset so that the double linkage would work regardless of the state of the used bit

#

but i wouldnt need to worry about any of that now anyway

#

not in few years either

#

i might need more flags too
but again there are many ways i can find some
that 16 bytes struct still has a lot of spare bits chad

lavish onyx
sour raven
charred surge
#

kekw

charred surge
# lavish onyx Jesus

so far it seems to be going well but im kinda struggling with the demand backing mechanism
I still refuse to just
back it all with pages then forget about it LiPerformDelisquesceUponObjectW

sour raven
charred surge
#

only commonality is that its as yappy as hardware structures' descriptions usually are ✨

#

so real

#

I luv yapping

#

and overcomplicating stuff uwu

lavish onyx
#

where did chaos go

#

(its only been 2 days i know)

charred surge
#

lol well ty for asking xD
I finished rewriting the pmm yesterday and its much cleaner but i only attempted to run it earlier today
aaand unsurprisingly it was buggy af sooo I've been fixing bugs there today

#

also i figured that for the past 4 months or so qemu willingly allowed me to mov to %cs somehow lmfao

#

I still have no idea how i didnt notice that ive had that in my init code
nor how qemu kept simply ghosting a #ud fault like its nothing pain

charred surge
#

aight chat pmm rewrite werks
I basically scrapped the whole thing to the trash and started from a clean slate uh
so 600loc + headers were turned into ~800loc + headers and inlined helper functions
and its not even functionally complete lel (I still dont place constraints on buddy list lengths and I dont merge up when that overflows)
but idk i might write few tests first then maybe implement that part

#

next imma go back to clean up the vmm from the previous mess and then i can move on with memory stuff
well after I make an interface for memory objects and stuff but ive layed down everything needed so it shouldnt take much

charred surge
#

wow im alive

#

almost forgot :o

#

vmem should be done by tonight

#

~~I only started writing it yesterday ~~

#

then i will do the vm abstraction layer (which will take a while) and vmm will be done with
finally..

#

been reading into threads and processes recently too

#

for some reason I broke that reading block of mine for 2 days and could read 50 something pages which is absolutely crazy by my crippled brain' standards

charred surge
#

my logger is soooooo ugly and over engineered and I hate it
but idk i dont really wanna rewrite it anytime soon

#

if i could reach a successful fireworks test with an smp+smt aware scheduler by chaos' initial commit anniversary i would be very proud kekw

#

thats still on August so I have plentiful time for my slow ass

#

:D

charred surge
#

e

#

well i just figured out few simple points
1- im alive
2- I havent made a commit to the repo in well over a month
3- I got nowhere better mentally wise during that past month so apparently the lack of osdev doesnt improve ur health, and thus conversely its presence shouldnt harm it either trl

#

this being said I would get back in 2-3 weeks maximum

#

last thing ive had done was vmem which was was working fine for the most part
but then I got bored reading on vmm object layer design stuff before switching gears and playing around with GTK
which DOES have adverse health effects

#

anyways just a short yap
for now

sour raven
charred surge
#

how's bto doing

sour raven
sour raven
charred surge
charred surge
#

and
how's Greg trl

charred kraken
sour raven
sour raven
sour raven
#

you been dead so long kvass exists now trl

charred surge
charred surge
sour raven
sour raven
#

im doing powerpc interrupts now depresse

charred surge
#

okay thats actually super cool
ive missed too much 🥀 😔

sour raven
charred surge
sour raven
#

windows 10 post 2020 lore

charred surge
#

realest

sour raven
#

later releases of w10 were really stable ngl

charred surge
charred surge
#

chaos's lilkernel (no LLM use) - a reference implementation for how a kernel should NOT be written

vestal tundra
#

kernle

charred surge
#

dead kernle 😔

#

not for long tho :>

vestal tundra
charred surge
#

soon™

charred surge
#

aight fellas its time i guess
so
development of chaos was interrupted by some side quest back in late February that spanned 2 months until late April
during that time i was planning to get back to it once im done
well exams were approaching too so i had to extend that time even further (just finished today, really) but back then i kept thinking that the best case scenario this can go by is expanding way too much for me to be able to handle it with a little ROI time-wise
let alone the 99% chance of just failing miserably
and so i thought i can switch gears to a more limited environment
some low powered platform with more simple system architecture that allows for more basic implementation and perhaps developing more employable usable skillset
with that being said i guess this means the work on this bootable-memory-management-subsystem-with-barely-anything-else of a project is now suspended indefinitely

#

i may repurpose this thread once i get something materialized out of the new thing or i may start a new one

#

probably the former
currently im just reading into the platform so it will take some time until i start implementing shit :P

#

also i wanna try writing it in cpp and use cmake instead of make but that would be too much for me to experiment with at once trl

simple girder
charred surge
simple girder
charred surge
#

happens

charred surge
#

okay soo
we're doing embedded os now trl
the platform im targeting is pico 2 here (32bit arm)

as a starter i will build the image to run from ram as non-secure :P
until I get myself familiar with the security extension (whether or not i can utilise it) and how it affects the other parts of the system, the flash partitioning, the QMI stuff (xip and the available address translation ability) etc etc which I dont know how to fully control yet....

#

project codenamed colonthree btw ralsei_weed

Screenshot_20260618_232839_Chrome.jpg
#

I've had the repo with some skeleton source tree for a month now lol
but soon™ i'll start writing stuff for it

charred surge