Maestro - Linux-compatible-ish OS in Rust

[tulip ember]

Cause i tried but i found the global process really strange, based on a custom bin they made to compile and package the os

[frail junco]

you can use it

but don't rely on all those crates

make your stuff

only use the bootloader crate basically

(limine)

[tulip ember]

Okay, i see

I'm just really starting in os dev, and it's a bit difficult to find good infos, and precise instructions lol

[frail junco]

that is a good guide tho, it explains stuff somewhat good

but it is incomplete

[tulip ember]

Okai, it was my opinion too, thanks

[frail junco]

the multithread stuff is not there

and that's basically the most important one

especially in rust

bc it has its own apis

[late oar]

I mostly used osdev.org but I also relied on phil-opp.com for Rust-specific stuff like booting, Rust setup and unit testing

Thanks!

[frail junco]

how do you integrate the scheduler with rust's async api?

[late oar]

I don't use the async API

maybe I should. I tried to wrap my head around this but I just couldn't figure out how to do it

[frail junco]

Phill said he wanted to put the guide out lol

Still waiting for it

[inner pilot]

bad idea

imo the async api is not really suited for kernel dev

[balmy plume]

Managarm now has an async runtime in Rust

on top of it's kernel level API

[late oar]

My goal for Maestro for the end of 2025 is to have enough stuff ported to be able to dev on it (not necessarily Maestro itself because that would require QEMU for testing I guess)
For the end of 2026 it would be having a desktop environment
And for the end of 2027 I have no idea (running Minecraft or Firefox?)

[late oar]

I am now wondering if porting QEMU (without KVM) would be diffcult or not

[young ermine]

not really afaik

[late oar]

random fun fact: after I posted my first blog article, I had a guy from huawei contacting me to propose me a job in the same team as Philipp Oppermann (he works for huawei)

I said no because that looked boring as fuck

[frail junco]

lol

[late oar]

well I guess I could attempt to port it this year then if I have enough time left

[young ermine]

idk how fucky networking will be but for the most part it's probably just gtk

[late oar]

do I even need it if I just run QEMU in the terminal?

[young ermine]

if you don't need user networking then it's probably gonna work out of the box

i don't think you do

but it's boring if it can't do anything, no?

[late oar]

well maybe I could run Maestro inside of itself

[young ermine]

that's true, but at a certain point you'll realize you need to go past that

[late oar]

you can run QEMU without a window, just redirecting the serial port in the terminal

[young ermine]

and a net stack + web browser is imminent

[late oar]

having a GUI will require a desktop environment, which is 2026's goal

[young ermine]

after all you wanted a desktop environment by 2026, which i don't think is a reasonable goal

that's quite the other way around

[late oar]

by the end of 2026, for the desktop env

[young ermine]

you can have a gui without a full blown DE

twm/jwm/dwm is piss easy to port and you can have a gui that way

when you said DE i was thinking of a full suite of apps from like KDE, GNOME or other desktop environment

[young ermine]

along with the respective compositor + window manager + shell

[late oar]

that's also what I was thinking about

not a full suite though

like if I have a compositor, window manager, shell, a file explorer and a settings panel I would be happy

[young ermine]

unless you like torturing yourself with uncrosscompilable mess, your only option is kde or xfce

or whatever else you can find that has no relation to gnome :^)

[late oar]

I want to build my own DE c:

I think Linux stuff in general is ugly

[young ermine]

all i can say is good luck

[late oar]

thanks

[frail junco]

just have wayland apis

[balmy plume]

@young ermine instruction emulator, when?

[young ermine]

i'll see what actually is needed for kvm

in terms of instruction emulation

i am pretty sure that whatever i encountered was basically a bug

it looked like it was executing zeroes

[late oar]

Remaining things I have to do this year (roughly):

• Implement SMP (currently in progress 🔨)
• Rework the scheduler to use SMP (I have to do that thing that balances running processes across cores)
• Port stuff I need to code (autoconf, automake, make, cargo, rustc, vim, git, etc... with their dependencies)
• Fix the terminal so that vim can work correctly
• Add network support so that I can use cargo and git without it being annoying (that's gonna be what requires the most work I think)
• Maybe port QEMU

I guess I would also need openssl or similar for cargo and git

[balmy plume]

Wasn't the next missing piece the page table walk in the kernel?

[young ermine]

i don't know exactly, but qemu was mapping in memory and presumably copying the bios there so something must have gone wrong

[late oar]

implementing KVM sounds fun

[young ermine]

it really isn't

[late oar]

sad

[young ermine]

it's kind oif a shitty interface

[late oar]

what's so bad with it?

I use it at work and it's not so bad but there's a lib between me and KVM so I guess that's why

[young ermine]

it's easy enough to use, but the kernel (or server, in the case of managarm) side needs to implement a bunch of stuff

it's just a lot of stuff you need to implement to even get qemu to consider using kvm

of course i stubbed it all to hell which it was happy with but i'm sure the missing pieces are not optional :^)

[balmy plume]

Well, we did get to the point where it was executing instructions in vmx mode

Iirc we got to an mmio exit

[young ermine]

it executed a bunch of zeroes though

like i said

so i don't think it was exactly correct

i need to get back to it at some point

[late oar]

maybe you'll get it to execute ones instead

[balmy plume]

if you can execute zeros and ones you only need to make it execute mixed zeros and ones, how hard can it be?

[inner pilot]

[late oar]

okay I fell into the cities skylines 2 rabbithole. see you guys in 3 months

[last shoal]

another one bites the dust

[late oar]

I've been working on this for so long, don't worry the project is far from over

It has become my reason to live at this point

[shell tusk]

Thats what they all say before not being seen for the next 3 years

[late oar]

I've already interrupted this project for 6 months in 2020 because of a crappy internship, and here I am still working on it

also I've mispoke, I cannot play on my laptop and I will have only that next week so I will work on Maestro anyways

[late oar]

okay guys I am back at working on Maestro

[late oar]

next blog article ready. I will post it when I figure out why I can't upload the video to my website

[inner pilot]

upload directly?

not sure if you want that ngl

you just upload to yt and embed in the site

because if you're selfhosting it might kill your bandwidth

[late oar]

I've already had a video in an article and it was fine

[inner pilot]

where do you host?

[late oar]

My company's service (clever cloud)
I think I have unlimited bandwidth

But even when it was on a simple OVH VPS that was fine

[inner pilot]

then good

you might have unlimited bandwidth with a fair use clause

which doesn't hold when you e.g. get ddos'd

obviously just being paranoid

[late oar]

new blog article: https://blog.lenot.re/a/compiler

[plucky fiber]

nice work Luc

[fast halo]

that's awesome!

[errant venture]

Nice, btw are you sure there's an interpreter in the statically linked programs? Id assume since they're static they don't need relocations

[late oar]

I think so?

You're making me doubt

I'll recheck when I have access to my computer

[little escarp]

there is a statically linked ld.so (ld.a )

static PIEs still have R_RELATIVE relocs that the kernel does not handle for you

[balmy plume]

yeah that's also how mlibc does it

with static linking, we link the ld.so code into libc.a

musl only have libc.so (and ld.so is a symlink to libc.so) so it's similar

[errant venture]

didnt know static relocatable stuff is a thing

[little escarp]

well we do produce an ld.a lib

[balmy plume]

ah

[balmy plume]

even w/o static relocations a statically program can just dlopen() a DSO

mlibc does support that

[inner pilot]

dlopen("libc.so")

[late oar]

I have yet to figure out how to remap interrupts with the APIC

Although it's probably too hot for me to think properly

[errant venture]

fair i guess, i thought that didnt need to relocate anything because u can only call stuff via function pointers

[inner pilot]

i dont think the apic does remapping

that's ioapic iirc

[late oar]

Yes

[late oar]

2208 unique visitors on this article since I posted it. Not bad

[late oar]

I/O APIC interrupts redirection: ✅

now, let's try to start the other cores

[desert hearth]

smh, just use limine

but nice - is this a rewrite or did you not have smp and ioapic stuff before?

[inner pilot]

the latter

[inner pilot]

you'd have to use limine to boot in the first place

[desert hearth]

shh I'm trying a conversion

[inner pilot]

actually, yeah he'd have to switch bootloaders for !x86

bc mb2

[tough onyx]

Do you guys use limine SMP?

[inner pilot]

i dont

[late oar]

yeah but if the user doesn't want to use limine, that's a problem

[inner pilot]

not because i think it's bad, just because i want to NIH it

[desert hearth]

Have in the past, it's convinient.

[late oar]

I use multiboot2 for booting

[inner pilot]

how are you going to boot arm

[late oar]

idk yet

[desert hearth]

The Linux protocol for non x86 is actually quite nice

That's my plan

[inner pilot]

hm

i was going to do limine + uefi

[desert hearth]

Again limine is nice, but I want to venture to non-uefi machines

[inner pilot]

i'm sticking to uefi 100%

i do support u-boot efi emulation though

[desert hearth]

Yeah makes sense to use limine then

[inner pilot]

powerpc would be funny to do but there's basically no 64-bit hardware that's cheap and easy to get

[late oar]

not many things are running powerpc nowadays, isn't it?

[inner pilot]

don't summon binder

many consoles run ppc32 but that's last last gen

[late oar]

summoning @static moat

[static moat]

You should assume that at least half the commenters on HN didn't even read the article

[static moat]

ppc?

do you actually mean powerpc or power? (it's a little complex)

[late oar]

I don't know anything about powerpc or power aside that it was used in the Wii

[static moat]

so, if you are looking at oldschool powerpc
MACs are a good source. Power Mac G5 for instance

PowerPC and Power were merged together later... so technically there are no powerpc machines anymore

for power you can get used Power machines cheap off of ebay.. like $800

[solid rose]

email?

oh you mean ebay

[static moat]

yea

https://store.vibrant.com/IBM-8202-E4B-4-Core-30Ghz-NO-PVM.html

there are places which sell refurbished as well

[coral wing]

"cheap"

[inner pilot]

that's pretty cheap

[static moat]

you likely spend more on coffee...

[coral wing]

Thats like, almost 2 nintendo switch 2's

[inner pilot]

a switch 2 is also not a powerpc mac

[coral wing]

Nah, H2O gang

[static moat]

oh... thats even more expensive

[coral wing]

I dont even pay for the (tap) water, my parrents do

[static moat]

ah... so you have no idea what things cost.

Just for future reference... that is cheap

[coral wing]

$800 does not seem cheap

[static moat]

I know... but when you actually start looking at things, and general expensive... that is a small cost.

example:
large number of people buy a starbucks 5 days a week
this is roughly $25/week * 52 weeks a year = $1300

[coral wing]

Water is like 1.26€ per m³ on avrage (where i live)

800€ would be like 635m³ of water

thats a lot of water

[static moat]

cost of bottle of water, $1.44

[late oar]

800 euros is 800 baguettes 🥖🥖🥖

[coral wing]

Imagine drinking bottled water

[static moat]

yea... don't do that if you can avoid it

ok, think of it another way...
an $800 machines is ~ $3/day for a year

[coral wing]

isnt it closer to 2.1

[static moat]

yup

I just rounded to $900 and 300 days / year 😄

[coral wing]

ok boomer /s

[static moat]

In 1987 I raised money to buy a commmodore, I was 12/13
In todays dollars it would be around $2300

[coral wing]

I'd be more concerned about the cost of the power it requires

[static moat]

my dual xeon server costs around $230 / year in electricy if I run it 24/7

[coral wing]

my home sever setup would be like

umm, let me calc rq

[coral wing]

my would consome about the same: 236.52€ / year if it runs 24/7 at full load

[coral wing]

thats a lot of money, how long did you spend raising it?

[static moat]

around 6 months

remeber that was about $800 back then

[late oar]

I've managed to get the other cores to boot (it does a triple fault right after because I didn't finish the trampoline code but that's a minor detail )

[solid rose]

Lol

Just keep them in real mode

[soft depot]

indeed, GameCube, Wii, and Wii U are all PPC750-derived
Xbox 360 and PS3 are 64-bit PPC970-derived
many consoles are PPC

[fast halo]

how is water more expensive than coffee what

[fast halo]

then dont do ppc64

[late oar]

I'm going to move drivers in the same tree as the kernel. I currently have a repository for each driver (only PS/2 and e1000 so far) and I think that was a bad design choice

[late oar]

I've moved in my e1000 driver that I didn't touch in a while and that's the worst code I have seen for a long time

[late oar]

I have to write the SMP trampoline but I don't feel like writing assembly code right now

[late oar]

okay guys, I have recovered the will to live (do osdev)

[inner pilot]

https://tenor.com/view/pearto-kasane-teto-pear-spinning-fruit-gif-17080334588594587046

[late oar]

I've got all the CPU cores to start, print something on the terminal and then go in an idle loop

(and then the kernel crashes because of an unaligned pointer )

gotta fix this, and then I'll try to refactor the scheduler to handle multiple cores

[nimble cobalt]

https://tenor.com/view/safety-for-safety-mouthwashing-curly-anya-gif-6281022657649740567

[late oar]

update: now it's a null pointer

[late oar]

now fixed, it was a swapgs issue in interrupt handlers

bash starts correctly on the first core. While the others cores are up, but they are not yet able to run anything

I think I need to have a per-core TSS btw. Imma do that right now

[late oar]

✅

[late oar]

The kernel now triple faults when booting in x86 (32 bit) and I don't feel like checking why so instead I am implementing SMP support on the process scheduler (I test with x86_64)

[late oar]

I don't feel like touching the HPET right now so I guess I'll use the PIT to calibrate the APIC timer

[errant venture]

its like one reg read and divide by frequency

[late oar]

okay guys i am doing the hpet

[late oar]

okay guys, I spent the evening not doing the hpet

(distracted)

[late oar]

currently removing over-engineered code I wrote yesterday

[late oar]

I'm getting sick of my kernel's logging which is just white text on a background with no color, time nor logging level

I will be fixing this at some point

also most logging messages during boot are actually pretty useless

[young ermine]

gdt init OK ✅

[shell tusk]

gdt init MAYBE ❓

[errant venture]

Idt init FAIL

[keen turtle]

Schrodinger's GDT

[late oar]

(funny thing is that I actually had issues with my GDT today)

it wasn't mapped anymore and I didn't realize it for several hours

[solid rose]

😭

[late oar]

should I drop support for 32 bit CPUs? It's annoying to maintain sometimes

(not 32 bit userspace though)

[fast halo]

no

what if you want to port your OS to the wii

[balmy plume]

🔥 blazing fast 🚀 gdt 🗃️ init fail ❌

[late oar]

I would have to support PowerPC for that

[fast halo]

GDT init fail is probably disastrous

[fast halo]

Yes, and supporting 32-bit is the first step

Also how annoying is it to maintain really

I haven't really tried porting boron to 32-bit but I feel like I could do it anytime because I already use uintptr_t everywhere I deal with addresses, and everything

[late oar]

APIC registers too high. Need to move them. Lazy

[solid rose]

cant you map them to another address

[late oar]

that would require efforts

[fast halo]

?

Theyre at 0xfee00000

[late oar]

they are in the 32 bit range but I only map the first GiB for the kernel in 32 bit

the rest is reserved for userspace

[solid rose]

where do you load your kernel

[fast halo]

Oh yes, you would need to remap them

HHDM is impractical on 32-bit anyway

[late oar]

from 0xc0000000 to the end of the memory space (for 32 bit)

[solid rose]

0xfee00000 is in that range

[late oar]

it is mapped to the first physical GiB

[solid rose]

oh i see

[late oar]

no because that's the physical address

[fast halo]

that's kinda bad

[late oar]

If I remove the whole kernel, maybe I won't have any effort to do anymore

[fast halo]

i'd map the first 8m and then map the rest to whatever other pages

that way your kernel heap doesn't depend on the physical layout of the pages

and you can also map things like the apic into it

but i understand if thats not something you're willing to do (anymore)

[late oar]

oh yeah physical memory holes are a major flaw in my OS

will have to handle them at some point

[late oar]

I am refactoring the processes queue in the scheduler and this is the annoying moment where I would need an intrusive linked list but Rust cannot have those while being safe

[inner pilot]

yeah it's things like this where i want C back, but oh well

[late oar]

well you kinda have intrusive linked list but you need reference counters on every node

I do this already for the page cache's LRU

but the function to unlink a node that isn't at the beginning or the end of the list is unsafe because I cannot ensure the node the caller gives me is actually in that list

[late oar]

basically I have two structure: ListNode which is an element in the list and List which represents a whole list (with one or several ListNode in it).

ListNode has to be stored in a field of another arbitrary structure which itself has to be in an Arc to force interior mutability and have the reference counter.
When an element is added to the list, I increment the Arc's counter and when it is removed from the list, I decrement it (which has a cost btw. The alternative is to be unsafe)

The point is to have operations done on List so that it is borrowed, and then I don't have to worry about concurrency.
But I have a function defined as follows to remove an element in the middle of a list:

    pub unsafe fn remove(&mut self, val: &Arc<T>) {

It has to be unsafe because if val isn't actually a part of that list and is inserted in another list, I get a concurrency issues (because the other list is not borrowed so there can be concurrent accesses on it)

[civic copper]

does it run doom 🔥

[late oar]

nop

will do someday

[civic copper]

what about a doom ELF port

[late oar]

I don't have anything to display graphics yet

[civic copper]

what if i would make it like fully console based

possible but hard

[late oar]

then it should work

I guess

[fringe pike]

just package a kernel exploit to get graphics output :^)

[late oar]

lmao

[late oar]

okay guys, kernel module loading doesn't work anymore for some reason

[late oar]

I need to add a testcase where I attempt to load and unload a module

[late oar]

I suspect it might be memory corruption, and I really don't have enough energy to debug it

[late oar]

It works on my computer but not in the CI. help

[charred kernel]

that usually confirms the memory corruption theory

[inner pilot]

also on real hw?

[late oar]

I am on holiday so I don't have real hardware to test on

[late oar]

seems to work when running in release mode
increasing the kernel's stack size does not fix it so I guess this is not a stackoverflow

[late oar]

the issue arises after I remap a portion of the virtual memory for MMIO. Still investigating

The issue I have from the beginning is that I have a hashmap with all the kernel's symbols in it, and the symbols that are needed to load the kernel module are disappearing from it at some point

although the portion of memory that is supposed to be remapped does not overlap with this hashmap

[late oar]

I think I have solved the memory corruption issue with duct tape, however the issue remains on the CI.
It turns out the module, when compiled in the CI, looks for symbols that aren't actually in the kernel. I am starting to believe I fucked up my CI and the module is built against the wrong kernel

[late oar]

there was indeed a CI issue, where I was building the kernel with a feature enabled (the one that prints system calls like strace for debug), then it was building a test kernel module against that, and then it was rebuilding and running the kernel without that feature and trying the load the previously built module
When you enable/disable features on a crate, it changes the hash at the end of symbols

[late oar]

Also, turns out I was doing module relocation wrong all along. For relocations that don't have an addend (Rel, not Rela), you have to read an implicit addend which is located at the address where you are going to write the new value

I was using zero instead

[inner pilot]

well, aren't you just supposed to do
*location += data

[late oar]

there's some relocations where the addend is not used

well I guess you have a 0 at these positions

[inner pilot]

it depends on the type

[late oar]

swapgs shit broke again

[late oar]

I think I fixed it

(I was setting gs to zero somewhere, which zeros the hidden gs base on Intel processors, but not AMD)

so now I have the scheduler that stops scheduling for some reason. Gotta debug this

[late oar]

that's an intrusive linked list issue. Removing an element seems to remove the whole list

[solid rose]

man i totally hate it when that happens

[late oar]

yeah

I tend to struggle with linked lists, they are tricky to visualize without paper

[solid rose]

I see

It's funny how it just went like "want to delete an item? I'll delete the whole list."

It did the job... Horrendously

[late oar]

lmao

well after a bit of research, it doesn't seem to be the removal function that is the issue but rather the one to move an element at the end of the list

(which I am using to push a process at the end of the run queue after it has been selected to run)

I've commented this line and then it was working (although I have to fix it anyways) and I've realized the OS has become a lot faster since I rewrote the scheduler

my puter is running out of battery, I'll continue debugging tomorrow

[placid vine]

holy shit is this actually linux compatible?

can it run a gui?

[inner pilot]

no

[uncut crown]

5 YEARS? PROJECT FOR FUNN?? PROJECT FOR SCHOOL??

AHHH

[placid vine]

disappointing but understandable

[late oar]

Not yet

[late oar]

Yeah

[solid rose]

bentobox ftw :^)

/s

[placid vine]

sad

[inner pilot]

@late oar how do you keep track of mapped objects in an address space

[late oar]

I don't
I have a BTreeMap with MemMapping elements in it, which has the starting address, size in pages, protection, flags, etc...

[inner pilot]

I'm starting to do the same now

you probably saw, before i literally stored every single page

do you handle mappings that shadow other mappings?

[late oar]

I used to do that too

[inner pilot]

like 0x1000-0x4000, and you mmap a single object on 0x2000

[late oar]

If you map over another mapping, the old mapping is truncated/split or removed

[inner pilot]

yea if you map something in the middle, it should get split up, no?

[late oar]

Yes

[late oar]

https://github.com/maestro-os/maestro/blob/608a1dc950f407579b6590975c5350fa0ae3fd04/kernel/src/process/mem_space/mapping.rs#L295

[inner pilot]

thx

yoinking that

jk

iretq and i were trying to get it working all night lmao

[late oar]

I have a MemGap structure to represent free memory but it is not necessary. That will go away at some point

[inner pilot]

but why? just remove the mapping?

[late oar]

Yes, and you can use gaps in the tree to find free space

I just didn't implement that thing to find holes

I think my previous messages don't make sense

In short: my implementation is more complicated than necessary

[late oar]

I now have critical sections support

(thanks @silk wren for explaining those to me here ❤️ #schedulers message)

[silk wren]

np

[late oar]

I used to only disable interrupts to prevent preemption but it turns out that's crap when you start preempting everywhere (I ended up with race conditions)

[silk wren]

yeah

and it's also useful to be able to take an interrupt while running in a preemption critical section

there might be stuff like TLB shootdowns that you want to do via IPI while running with preemption disabled

if preemption disabled == interrupts disabled, you can easily die from this

[silk wren]

(yes this is really stupid to do, and you should instead do whatever page table modifications you do under a blocking (RW)Mutex where you don't need to disable preemption)

[late oar]

I didn't even think about tlb shootdown with smp yet

[late oar]

*orgasm*

(integration tests finally passed for the first time in a few month)

[inner pilot]

interrupt test

[placid vine]

irq 1

[latent anchor]

i mean there's a crate for that I've been using for quite some time

but yea you're right unlinking stuff is hell

[late oar]

how is it called?

[latent anchor]

intrusive_collections or smth

you provide your own cell type to it so like

you can use Rc or anything

i think I just use UnsafeCell or smth i forgot

it's been a couple months since I last touched that

[inner pilot]

yea i also use intrusive_collections

[late oar]

I have to find the energy to write the code to relocate I/O APIC registers in case they are out of reachable memory in 32 bit

And also I have to write the code to rebalance processes across cores when there's a process that stops running

[errant venture]

isnt it pretty much hardcoded at fec or whatever

[desert hearth]

yeah the spec says they start at that address and then have a fixed increment

I forget the exact number, but iirc it should fit under the hpet(?) at 0xFED00000

[late oar]

The osdev wiki says:

To increase system flexibility when assigning memory space usage, the I/O APIC's two-register memory space is relocatable, but defaults to 0xFEC00000

[desert hearth]

btw relocating them is a chipset specific thing

and I dont recall seeing it mentioned in the amd bios/kernel PRMs anywhere

so it might just be an intel thing

[late oar]

huh

[errant venture]

you're probably thinking of lapics

[late oar]

I know I can relocate lapics, but the osdev wiki says I/O APIC

[desert hearth]

here's something from a 1996 spec lol

[late oar]

unless that's an error

[desert hearth]

you can tell by it referencing PIIX3 (lol), but I cant any documentation on that particular register

[late oar]

well I guess I'll just remap the virtual memory

[late oar]

If there's several I/O APIC, their registers are all next to the other?

[desert hearth]

that wasnt covered 😔

I dont know where I (or if I indeed did) read about the spacing of registers for multiple ioapics

maybe its worth seeing how linux handles them?

there would be multiple entries in the madt though, one for each ioapic

[errant venture]

i think just remapping makes the most sense

do u support PAE?

[late oar]

nop

[errant venture]

u really should

or just long mode

[late oar]

I have long mode

My stuff works for x86_64. I am doing the 32 bit support right now

[errant venture]

oh ok

i wouldnt support 32 bit without pae

or tbh just compat mode support is good enough

[late oar]

I should probaby implement it, yeah

I do have compat mode with 64 bit

[errant venture]

oh

thats cool

any reason why u want to add 32-bit support?

[late oar]

idk I begun with it and since it's here I just keep maintaining it

also I tell myself that would allow the OS to run on old hardware

[late oar]

okay, remapping is done for the I/O APIC, now I have to the same for the HPET

[late oar]

god i love when the things i do work first try

[inner pilot]

does thing correctly
works
mfw

[late oar]

literally my last commit:

• code is crap and does not work
• remove crap and rewrite correctly
• works

[silk wren]

does thing incorrectly

doesn't work

[inner pilot]

many such cases

[silk wren]

a worse situation is

does thing incorrectly

works

because then you will end up with

does thing correctly

doesn't work because of bug somewhere else

a week/month/year later

[late oar]

oh yeah, that happened a few times

[late oar]

wait, if the kernel panics, how do I make sure the other cores stop?

IPI?

[inner pilot]

halt ipi

or you can queue a stop message

[late oar]

is it a hardware thing or do I have to implement it?

[inner pilot]

it's just an idt entry dedicated to that

[inner pilot]

like you could have a generic "action" ipi where you have a message queue and you just perform an IPI which tells it to do something

like

void enqueue_msg(uint32_t lapic, msg_t message, void* data);
...
enqueue_msg(42, IPI_MSG_HALT, NULL); // lapic 42
do_ipi(42);

then cpu 42 would look at the queue and pick the first element off of it

[late oar]

makes sense

do I have to make IPIs for TLB shootdown sometimes?

I am trying to determine if I should implement that just for halting or if there's other usecases I should consider right now

[inner pilot]

of course

[late oar]

when

[inner pilot]

multi threaded programs come to mind

if you have the same page table active on 2 cpus

you need to somehow notify the other cores of a invlpg

[late oar]

right

[late oar]

so I guess I need to pass the address to invalidate too?
but I also probably want to batch those instead of sending an interrupt for each page

[inner pilot]

that's what the data here could be used for

[late oar]

and that's the moment I realize it's going to be annoying because I init paging before the APIC

[solid rose]

that's how you should do it...?

[inner pilot]

?

[solid rose]

nvm

[late oar]

yeah but then I have to add a condition in the paging code to check if the APIC has been init to know if I can send the IPIs

[inner pilot]

why?

that's unrelated

you only need tlb shootdown for unmapping and remapping

[silk wren]

A good way is to, when you have acquired the panic context, first send CPUs a normal IPI on some vector. Next wait some time, if CPUs don't respond (by marking themselves offline) send a NMI.

[silk wren]

if there are no CPUs to send a TLB shootdown to, you don't need IPIs

if there are active processors that need to have their TLBs shot down, you have APIC

[late oar]

yeah

[late oar]

I've got processes running on different cores, and then I have a triple fault

[silk wren]

Smells like percpu issue.

[late oar]

yup

it crashes when switching to a process, which apparently has a NULL stack kernel address?

[late oar]

okay I'll continue debugging this tomorrow

[desert hearth]

I find it userful to have per-cpu queues for work that other cpus can add to, then you send ipi to draw that cpu's attention to the cpu. Stuff like halting I do with a flag that gets checked before running any work from the queue. You can build tlb shootdowns on top of that, and I'm finding it useful for other things (migrating threads, checking timers are in sync).

[late oar]

okay I found it.
Each process has a pointer to save its kernel stack when switching context.
I have an "idle task" which just does a sti-hlt loop to wait for events to happen on the system when no other process need to run. However I was sharing this task across all cores, so when the idle task was used by one core, it was overwriting the saved kernel pointer stack, thus corrupting it for other cores
My fix was just to have a per-CPU idle task

I have processes running across several cores. Now imma check if gcc is still working

[late oar]

nevermind it's not fixed. I have just been (un)lucky (there's a data race or race condition somewhere)

[silk wren]

you fix one issue, it unmasks the next

[late oar]

okay I think it's actually an issue of how I implemented fork. I manually switch the current running process to be the new child, but I do so on the current core's scheduler, while the child may be assigned to another core

I will rework this tomorrow I think

my fork implementation feels like it has been fixed with duct tape

[late oar]

lmao

[solid rose]

Yes.

[late oar]

I really should build a website for Maestro. Then I could have a page to try the OS with https://github.com/copy/v86

[late oar]

when I'll have SMP working correctly, I think I will port make to make sure its working

and have a nice screenshot for my blog

[late oar]

Do you think I should check the queue at each interruption or only when receiving that specific IPI?

[desert hearth]

hmm good question, I suppose checking it at more points would improve responsiveness.

I only check it for that specific vector

you might save an interrupt entry + exit here and there, I suppose it would depend on a lot of things.

[late oar]

also, should I print the kernel panic message only once I know every core has halted? I guess?

[desert hearth]

I like to print something to tell the user a panic is happening, and then only continue once I know every core has halted

so it something blows up while halting you've got some record of what was supposed to happen

[late oar]

right

[desert hearth]

but I dont know if thats the best way

its just what I settled on

[late oar]

when sending an action on a CPU's queue, should I directly send an IPI or should I do something like wait for the queue to be full, or a timeout to be reached, before sending the actual IPI?

[inner pilot]

depends on the urgency

a tlb flush should be urgent

so should be a panic

[late oar]

okay, since those are the only two things I need right now (I think?), I guess I won't bother and just send an IPI directly

@desert hearth you mentioned thread migration too. Why do you need an IPI for that?

I was thinking about just locking the other scheduler's queue to steal a process from it

[desert hearth]

you might need an ipi if the thread is the one currently executing on another cpu

[late oar]

mmmmmh

[desert hearth]

-me eating dinner

so it might go something like: you want to migrate the thread, its currently running on some other cpu. You set that cpu's 'should reschedule' flag and then send it an ipi. That cpu handles the ipi, goes to exit from the interrupt (this is one of those points you check that flag) and then clears the flag and switches away from its current thread.

I mean ideally you wouldnt migrate a thread that is already running somewhere else, the only place I can think of is if you're setting a fixed cpu affinity for a thread.

[late oar]

I don't support cpu affinity for now

[desert hearth]

if you're looking to balance workqueues, then yeah you'd just take threads from queues

[desert hearth]

that makes things simpler then

[inner pilot]

cant cpu aff be stubbed anyways?

[late oar]

The load balancing I plan is the following (for now):

• when a process transitions to the Running state, I select the scheduler with the least processes on its run queue and I insert the process on it
• when a process transitions away from the Running state, I remove it from its schedule run queue. Then I look at all the core with the most running processes, and if it has more than 1 more process than me, then I steal a process from it

I just want it to be simple and to work, not be optimal (that will be work for later)

[inner pilot]

afaict it's just a hint

[desert hearth]

you might be able to, but I want to support that way.

[shell tusk]

could take processor topology into account, too

[late oar]

like, NUMA stuff?

[desert hearth]

also hyperthreading

[late oar]

I didn't have a look into that. I don't know how it works

[desert hearth]

its pretty easy, just some cpuid stuff.

[desert hearth]

I would suggest only stealing work when you run out of things to do, and then have a load balancing task/thread that runs occasionally.

in order to minimize inter-cpu traffic.

[late oar]

btw if I wanted to make a more elaborate load balancing, should I rebalance every time a process exits the run queue or should I have a task doing that periodically?

[late oar]

well you answered my question

[desert hearth]

lol

[late oar]

all of this stuff shall make a good blogpost

[desert hearth]

https://tenor.com/view/emperor-star-wars-good-goooooood-evil-gif-12447951

[late oar]

I want to work on Maestro so bad

but I am at work

[inner pilot]

same

[late oar]

I am reading about hyperthreading and if I understand correctly, logical processors that are on the same physical processor share the same TLB.
So I guess I should attempt to keep processes on the same physical processor when possible?

[late oar]

maybe I should better focus on my fork issue and the per-CPU queue before thinking about that

[solid rose]

Yes

[late oar]

current number of loc in Maestro

note that this number has been going down since the beginning of 2024 (when I started cleaning up)

[late oar]

I made a fix yesterday and it still doesn't work but the issue is different. So I don't know if my fix is bad or if I just discovered the next issue

okay apparently gs is zeroed in the child process after forking, so my fix is likely shit

okay I know why. When forking I was taking fs and gs from the parent process's structure, instead of taking them from the current context directly. However, those fields are init only when switching from a process to another (which couldn't be done already since no fork had been made before)

now it works when the kernel is build for 64 bit, but not when built for 32 bit. Need further debugging

I have to go back to work (my job that gives me an income, not Maestro)

[late oar]

once I have SMP working, I will try to port make and vim (and maybe autoconf and stuff)
I'll have to fix the terminal for vim. I am 100% sure there's missing stuff for displaying it

After that, idk if I will first port the Rust compiler, or git.
If I do git, I will implement the network stack since I will need it for pushing/pulling

once I have those ported with network support, I have pretty much fullfilled my goal for this year (in which case I will begin next year's goal earlier, which is to have a desktop environment)

I kinda want to write userspace code. It's been a long time (even at work I write kernelspace code)

[late oar]

turns out it crashes when I do the INIT/STARTUP IPIs for the other cores (when in 32 bit)

[late oar]

I think I have memory allocator issues (help)
and I have several cores triggering a panic at the same time

[late oar]

okay I think I know where the memory allocator issue is
turns out, writing this in Rust is useless:

let _ = MUTEX.lock();

because it drops the guards instantly. To avoid this, you have to give the variable a name other than _, such as:

let _guard = MUTEX.lock();

[errant venture]

[tough onyx]

omg this is stupid

does that mean naming variables is undefined behavior

[errant venture]

what were they thinking

[young ermine]

let _ = x; is a common way to discard expressions in rust

[tough onyx]

lock() isnt an expression

its a statement

[young ermine]

MUTEX.lock() is an expression

[tough onyx]

unless it returns a spinlock

[young ermine]

it's a function call

[tough onyx]

sorry more like

its procedural

unless it returns something

like a LockGuard

[young ermine]

it does, hence why dropping it unlocks the mutex

but in rust even if a function returns nothing it still has a return type

() is of unit type, something like void in C

and you can use it as a value

let _ = {
  println!("Hello world!");
};```

this is valid rust, and the block that prints hello world will have a return value `()`, which is then assigned to _ and hence discarded

[late oar]

okay guys, Maestro is now running Minecraft and Firefox, but I cannot show any screenshot nor source code because of copyright reason

trust me though 💯

[late oar]

I currently have a global mutex around memory allocators. I guess that's bad when you have multiple cores because it causes contention.
How could I refactor my memory allocator so that I don't have to lock everyone?

[inner pilot]

i'm not sure you can express this safely in rust

[late oar]

I have unsafe code in the memory allocator anyways

[silk wren]

the number one way to make multiprocessing fast is to not share data

many allocators have a "big Allocator lock" which protects a shared allocator datastructure but then cache allocations per-thread (per-cpu in the kernel case)

[late oar]

I won't work on Maestro for the next two days since I am invited at a wedding

[tough onyx]

Cringe. Work on maestro at the reception

[shell tusk]

invite the people getting married to work on maestro with you

[errant venture]

cant believe you didnt decline saying you have to work on maestro

[late oar]

lmao

[shy trail]

lol

[late oar]

Okay guys. Back to Maestro in a few hours

[late oar]

time to write that thing to stop other cores when there is a panic

I am also going to make better panic messages

[shy trail]

same with me ngl

[late oar]

new panic messages now look like that

important detail: I am now printing register states when the panic was triggered by a hardware exception (I wasn't doing that before)

[inner pilot]

demangle pls

[late oar]

too much work

[inner pilot]

no

[late oar]

I am not using a crate for this

[inner pilot]

https://github.com/menix-os/menix/blob/main/kernel/src/generic/panic.rs#L97

why not lol

it's by the rust-lang people iirc

https://github.com/rust-lang/rustc-demangle

[late oar]

Now imma do that message queue between CPU cores for TLB shootdowns

[late oar]

do you guys think it's okay if the per-CPU queue has a fixed size and each message also has a fixed size? (messages are an enum)

[late oar]

I am not mentally ready to add a dependency on my kernel

[late oar]

I'll use lambdas instead of enums (like Linux does) so that I can execute whatever I want without having to modify the enum

that's how Linux does it: https://elixir.bootlin.com/linux/v6.16/source/kernel/smp.c#L630

[late oar]

I have that thing in the kernel CI where I build a disk image with a program that performs tests on the kernel to check the behaviour is correct.

I think at some point I will use my package manager to install pre-built programs in that disk image, so that I can test the kernel with them (like, install gcc and try to build a hello world with it, to make sure it works. Or try to clone a git repo when I'll have git working)

When I have the network stack I could probably attempt to make a HTTP call from the kernel, in the CI too

[balmy plume]

We have a nightly CI job that runs xbps-install into an empty sysroot, puts the resulting sysroot onto an image and runs the image on qemu to check if it boots

[late oar]

nice

pretty much what I would like to do

[mystic kestrel]

yh i use that too

[late oar]

I am struggling to implement the queue to defer function calls to other cores because I use dyn 'static + FnOnce, which I have to store in the queue, and then move to the other CPU's stack, which is annoying to do because it's unsized

I don't want to use a Box because that's allocating memory, and I don't want to use Fn instead because that means I can only capture 'static immutable references with the function (while FnOnce allows to move non-reference stuff in it)

[nimble cobalt]

what??? 😭 is this a real problem or made up by rust

it sounds like the kind of fake rust issue

hat wouldn't happen in C

[late oar]

that's because in C you would pass a function pointer + another pointer for the data that would be passed as an argument to the function (which isn't very convenient)
Doing that in Rust is using Fn, but I also want to be able to pass other stuff than references
The main issue with using FnOnce is that it has to be moved to be called (because once it's called, it is dropping whatever captured stuff remains). While Fn can be called by reference

[nimble cobalt]

Hmm I see but I think he C solution is fine tbh

[late oar]

but then the ownership of the additonal data is ambigous. You can probably allocate memory for it (which I don't like) and then it has to be freed by the function when called

or if the call is synchronous (you wait for it to be done on the other core before continuing) then you can keep it on your stack and free it yourself

But Rust needs this to be handled clearly

maybe I could have a struct wrapping my pointer to dyn 'static + FnOnce and make it act like a Box but without needing an allocation

I kinda feel what you mean right now @inner pilot

but it's clearly not 90% of time

[inner pilot]

yeah that was a bit of an overstatement

but it is annoying

[late oar]

it is annoying indeed

[balmy plume]

well, what would you do in C instead?

you also have to allocate in C to make this work in the generic case

[late oar]

indeed, but not when you are doing this synchronously. I want to handle both

and I don't really see myself using a function pointer + a pointer for data in Rust when closures exist (and are way easier to use)

[balmy plume]

you don't really need to do that anyway because &dyn Fn() is already a function pointer + pointer to data

[late oar]

yes

[balmy plume]

i wouldn't use the same mechanism for sync and async calls, that just sounds like a mess (especially in rust)

[late oar]

the difference being, when you use Fn you take the data by reference, while with FnOnce you are moving it

[balmy plume]

just send a &dyn FnOnce() for the sync case and a Box<dyn FnOnce() + 'static> in the async case

[late oar]

mmmmmh

[balmy plume]

• Send of course

ah i guess FnOnce doesn't work for the sync case

because you can't call it on a const ref

[late oar]

indeed

but, I may use Fn or FnMut since I know the reference won't die before returning

most likely FnMut since it gives more possibilities

so I either need to have two queues, or a single queue able to take both FnMut and FnOnce

[balmy plume]

you can also wrap it in a ManuallyDrop and move it using unsafe code or similar

if you need FnOnce

[late oar]

I've been trying to do that, but since it's Unsized, moving it is very annoying

last issue: my queue has an AtomicPtr for each slot (that is null when unused). I need to somehow turn the reference, or the pointer in the Box, to something I can pass to an AtomicPtr (which must not be unsized, that is, not a fat pointer, since a fat pointer means there is a ptr + size, and they cannot be written both atomically)

• AtomicPtr<dyn FnOnce()> is invalid because dyn ... is unsized
• AtomicPtr<Box<dyn FnOnce()>> is annoying because I have to make two allocations (one for the Box and one just to hold the pointer to the Box)

well, I think I need only sync stuff for now so I'll just do that with FnMut and I'll see later for async

Wait I may be stupid.
I don't have to make everything fit in a AtomicPtr. I can just make my queue hold structures like:

struct DeferredCall {
    // Set to `true` after `inner` has been written
    ready: AtomicBool,
    inner: UnsafeCell<DeferredCallInner>,
}

enum DeferredCallInner {
    Sync(NonNull<dyn FnMut() + Send>),
    Async(Box<dyn 'static + FnOnce() + Send>),
}

I think that would work

[balmy plume]

well, if you hack it hard enough you can also use FnOnce in the sync case

[balmy plume]

like, you can send a (unsafe fn(*mut ()), *mut ()) (wrapped in a type that is Send) where the function casts gets the *mut () and casts it back to the concrete type of the FnOnce (wrapped in a ManuallyDrop such that you can move it out of a mutable ref)

[late oar]

maybe I could do that, yeah

[balmy plume]

use std::sync::mpsc;
use std::mem::ManuallyDrop;

struct RemoteCall(unsafe fn(*mut ()), *mut ());

unsafe impl Send for RemoteCall {}

unsafe fn do_call<F: FnOnce() + Send>(erased: *mut ()) {
    let ptr = erased as *mut ManuallyDrop<F>;
    let f : F = unsafe { ManuallyDrop::take(&mut *ptr) };
    f();
}

fn do_receive(receiver: mpsc::Receiver<RemoteCall>) {
    while let Ok(rcall) = receiver.recv() {
        unsafe { rcall.0(rcall.1); }
    }
}

fn do_send<F: FnOnce() + Send>(sender: mpsc::Sender<RemoteCall>, f: F) {
    let mut inner = ManuallyDrop::new(f);
    let erased = &mut inner as *mut ManuallyDrop<F> as *mut ();
    sender.send(RemoteCall(do_call::<F>, erased));
    // TODO: Block to make sure that inner actually lives long enough.
}

it's incredibly ugly though

[late oar]

is it even sound? I think when you are casting to *mut () you are turning a fat pointer into a thin pointer

I think casting it back to a fat pointer afterwards is UB

[balmy plume]

no, it's a thin pointer all the way

[late oar]

oh, right, I didn't see you were casting RemoteCall

nvm

[late oar]

okay so I've spent the last few hours on this and I am starting to get fed up

Turns out I can't even use Box for this.
I have my own version of it (I don't use the one in the standard library because it crashes when there's a memory allocation failure) because to be able to call the FnOnce on it, I have to implement the trait on the Box to forward the call to the inner value

The implementation in the std looks like this:

impl<Args: Tuple, F: FnOnce<Args> + ?Sized, A: Allocator> FnOnce<Args> for Box<F, A> {
    type Output = <F as FnOnce<Args>>::Output;

    extern "rust-call" fn call_once(self, args: Args) -> Self::Output {
        <F as FnOnce<Args>>::call_once(*self, args)
    }
}

in the implementation of call_once, you can see there's a * before self. That's dereferencing the unsized value in the Box.
This is not supposed to be authorized (moving an element out of a reference that does not implement Copy) but it seems they added a special case to the compiler so that if you add #[lang = "owned_box"] to the structure, then it somehow becomes fine
I tried adding it and then I get an error because there's a conflict with the Box in the alloc crate when I build my code for testing in userspace

So right now I am gating the lang item behind a feature to avoid this issue

okay it doesn't work, fuck this

I'll have only sync calls for now

[tough onyx]

certified rust moment

[late oar]

@shell tusk I don't even have a slab allocator since I switched to Rust. That's an issue I'll have to fix one day

(The reason for not reimplementing it at the time being: laziness)

[shell tusk]

I have slabs but Im not sure I actually implemented them right and they could use some speed ups as wells (astral's uacpi score is atrocious) so I will probably rework if not rewrite it

[late oar]

when I implemented it I based myself only on the english wikipedia page

[nimble grove]

If you did it right it should just be pushing and popping an item to a singly linked list

You could inline it even

[shell tusk]

Yeah thats what it does

Did not do magazines but will do that soon

[shell tusk]

I think I implemented mine based on an old book about unix

Unix internals by vahalia

Its from the mid 90s and I have a physical copy

[balmy plume]

isn't there a nightly Box api to handle alloc failure?

yeah, there is Box::try_new()

behind nightly feature allocator_api

[late oar]

but then I have to add the alloc crate as a whole to the kernel

which makes it heavier

[dim frost]

I am pretty sure unused funcs will not appear in the final binary

[tough onyx]

depends if u have lto or not

and functions in every section

[late oar]

all pub functions (and the functions they call) will be in the kernel because I've setup it to do it. That's necessary for my kernel modules

[late oar]

last time I checked, LTO broke my modules

I've just discovered I can divide the size of the kernel binary by approx 2 if I disable fmt::Debug

[late oar]

You can do that by adding -Zfmt-debug=none in the RUSTFLAGS environment variable. You are also supposed to be able to add it in Cargo.toml to make it more convenient. I tried it and it doesn't work

today is the day absolutely nothing works

[late oar]

I've managed to implement the async call defer
Although both sync and async are using Fn instead of FnOnce

[late oar]

for TLB shootdown, how should I check the CPUs that share the same virtual memory space?
Should I:

• Defer a call to all CPUs, in which I have a condition to check cr3 and if matches, I invalidate and else I nothing?
• Have the CPUs that are binding the same memory space in a linked list and defer a call on each?
• Something else?

[inner pilot]

first point sounds sane?

[late oar]

okay it seems Linux does something smart (as usual)
The mm_struct structure has a bitfield with a bit set for all CPUs that are currently binding it

If I understand correctly, when you bind it, the bit is set atomically (and likewise, when you unbind it, it is atomically cleared)

That's a lot simpler than the linkedlist thing I imagined

[inner pilot]

linux also has mm_ranges

[late oar]

what's that?

[inner pilot]

see flush_tlb_mm_range

[late oar]

is it just for flushing a range of virtual memory?

[inner pilot]

yes

[late oar]

I am also making such a function, so that I am not spamming the queue with each page

[inner pilot]

so you don't do it page by page

okay

[late oar]

turns out reading Linux's code puts me in a good mood for some reason

I forgot to do the IPI after inserting in the queue. I'll do that tomorrow

[fierce orbit]

(first time this sentence has ever been said btw) /j /lh

[late oar]

is there a better way to invalidate a range of memory on the current core?

#[inline]
pub fn invalidate_page(addr: VirtAddr) {
    #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
    x86::paging::invlpg(addr);
}

pub fn invalidate_range(addr: VirtAddr, count: usize) {
    for i in 0..count {
        invalidate_page(addr + i * PAGE_SIZE);
    }
}

[ancient galleon]

after a configurable count you should just invalidate everything by reloading the pagetable

[late oar]

sounds like something I am skilled enough to do

let's do that

any recommendation for the default threshold?

[ancient galleon]

linux uses 32 at least on my system

/sys/kernel/debug/x86/tlb_single_page_flush_ceiling

i think this is the right one but im not sure since linux also does batching and lazy unmap

can't be more than the total size of the tlb that's for sure

[late oar]

thanks!

[little escarp]

make sure to not try that for invalidating kernel pages though :^) (assuming they are mapped global)

[late oar]

yeah

[late oar]

I am having a good time writing my kernel

(usually impossible)

[late oar]

I think I am done with TLB shootdown

Remaining tasks before I merge SMP (I think):

• CPU topology detection (aka translating @shell tusk's code to Rust )
• Rewriting the function that assigns processes to CPU to make it take more stuff into account (maybe I can implement nice, getpriority and setpriority at the same time)
• Write the kernel thread that periodically re-balances processes across cores

[shell tusk]

Oh yeah sorry for not upstreaming the updated code I am spending time with the bf

Its nothing major tho its mostly in the last part which is unrelated to the cpuid stuff

[late oar]

how dare you

[late oar]

@shell tusk where does __get_cpuid_count come from? I can't find its definition in your code

okay nvm I search discord and that's a gcc builtin

[desert hearth]

__get_cpuid_count() is for accessing cpuid subleaves (leaf 7 comes to mind)

ooh you're doing topology stuff

[late oar]

yeah

trying to minimize the frequency at which I destroy the TLB

[desert hearth]

ha yeah fair

[late oar]

@shell tusk I'm looking at the builtin's sources and it seems to be doing stuff you have already implemented
https://github.com/gcc-mirror/gcc/blob/master/gcc/config/i386/cpuid.h

[shell tusk]

The builtins cant search the hypervisor leaves, so for things like kvm clock it will not work

[late oar]

i've never wrote so many calls to cpuid in a single day

I've just realized I am not handling logical destinations when I do IPIs. That's gonna be an issue

should I also add x2apic support before I merge? Is it difficult to do?

[inland mirage]

Its very easy

There are just a few registers that are different

[late oar]

Nice

[young ermine]

in general x2apic registers are at msr 0x800 + (reg >> 4)

so to read the id register with xapic, you would do read32(apic_base + 0x20) and for x2apic you would do rdmsr(0x802)

they are mostly the same (minus the memory- vs msr-mapped thing), there's a few differences like: the first 24 bits of the id register for xapic are reserved but for x2apic all 32 bits are used

and for the icr you use the single msr instead of low+high

[late oar]

thanks!

[late oar]

maybe I understand something wron. Are logical destinations required for IPIs to reach logical cores?

or am I mixing up two things?

[late oar]

is xapic something else than the apic? are there 3 things? (apic, xapic and x2apic?)

[young ermine]

xapic = apic in this case

apic has 2 modes, xapic and x2apic

x2apic being msr mapped

[late oar]

alright

[balmy plume]

the retroactively renamed the local apic to the local xapic

[late oar]

thanks

I think I'm going to rewrite the function that chooses which CPU to run a thread on, and I'll do the x2apic later

[late oar]

I've been osdeving for the whole day without making long pauses

and I'll probably do the same tomorrow

[late oar]

I've just realized I didn't implement tlb shootdown for kernelspace mappings

sigh

[fierce orbit]

may I ask, sorry if this is a dumb question, what is the state of networking with Maestro?

[desert hearth]

Logical destinations are another way of addressing lapics, you don't need them unless you want them.

[late oar]

Inexistent

[late oar]

I have the CPU topology tree but I didn't check if it is correct (I'll print logs to verify I guess) and now I have to figure out how to use it

I have started to rewrite the function that selects the CPU to enqueue a process on.
Now it works like that:

• If the last CPU the process ran on (if any) is currently running a process whose priority is lower than the process to run, select it
• (not implemented yet) else, go up the CPU topology tree (starting from the last CPU), looking for a CPU whose currently running process has a lower priority
• Check a global bitmap for idle CPUs and select one if any
• (not implemented yet) Select the process with the lower amount of enqueued processes among those whose currently running process has a lower priority
• Select the process with the lowest amount of enqueued processes

Note that I'll have to replace the "amount of enqueued processes" metric by something else at some point

[late oar]

I'll have to write a task to re-balance processes across cores periodically. I guess I could reuse this logic somehow?

[shell tusk]

ULE just looks at the most loaded and least loaded cores and balances them out

In number of threads enqueued

[late oar]

nothing more? it doesn't care about CPU topology?

nor process priority?

tomorrow I'll start writing my blogpost about SMP

maybe I should implement sched_getaffinity so I can get nproc working (and do sched_setaffinity at the same time)

@shell tusk how do you explore you topology tree? do you walk up once, do a depth first search on the subtree and repeat?

I think I don't feel like touching the topology tree. Let's write the rebalancing task instead

[ancient galleon]

it just balances across nodes of the same level

so first across smt twins, then cores sharing L2 etc

[late oar]

I wrote the rebalancing task and it doesn't work (deadlock). Trying to figure out why

[late oar]

I think I'll do the x2apic another time. I want to move on and port stuff to the OS

once I merge SMP, I'm trying to port vim

[late oar]

I have random bugs I should have a look at sometimes. Like, I can't type |. When I do it types ?

(which is obviously annoying for testing pipes in bash)

I've just realized I didn't try to launch a fork bomb with SMP

[shell tusk]

Inb4 keyboard layout diff

[late oar]

yes

probably

[late oar]

maybe I should make a kanban so I stop forgetting stuff

[late oar]

@shell tusk @silk wren would you guys like to be cited in my next blog post (for explaining scheduler load balancing and critical sections to me) and if yes, under which name/nickname and would you like a link to a website and/or github?

[shell tusk]

Sure, Mathewnd and avalios.dev or my github

[silk wren]

dbstream, no website/github link

[late oar]

what packages do I need to port vim? autoconf, automake, make and ncurses?

sed? grep?

[ancient galleon]

libtool and libintl too i think

[late oar]

so, the only thing that remains to do for SMP is using the topology tree

or maybe I could do this later and move on?

[late oar]

I am starting to refactor my package manager. I currently have a rust http server to serve packages but I am going to remove it and use a s3 instead (not hosted on AWS though)

[deft sonnet]

wtf is wrong with Gnu code

Looks like rain fall

And the cloud is the code comment at the top

[brazen sierra]

what on God's green earth is that

who decided that was an acceptable coding style

[inner pilot]

8 space tabs interpreted as 4 or the other way around

[deft sonnet]

Why though. Just why.

[inner pilot]

hm?

that's an issue on your end

[deft sonnet]

Are you telling me that that monstrosity of a macro is remotely okay

Like it’s so easy to align that shit

[inner pilot]

????

it is literally aligned

[inner pilot]

like i said

[deft sonnet]

So I realized I’m stupid

💀

Maybe I shouldn’t make an os after all

[inner pilot]

you need to set your tab display width to the actual width of the project

[deft sonnet]

No I was on mobile

[inner pilot]

so am i lol

if you couldn't tell

[deft sonnet]

Yeah but your phone was rotated lol

I’m a smooth brain

Im about as useless as internet explorer is a browser

[late oar]

I did not work much on Maestro this week end. I was busy having friends

[late oar]

I am going to attempt to build make on my OS

not today though, I am gonna go to sleep first

[late oar]

I think I'll use it later. I want to move on

[late oar]

I wanted to use a screenshot of the Intel SDM for my blogpost but I think I can't actually do it because of intellectual property

[late oar]

well turns out I just implemented it

but now I think I have a concurrency issue somewhere

[charred kernel]

im pretty sure u can quote it tho

like I must have seen quotes everywhere

[late oar]

yeah but I need at least a cover image for the blogpost

and no, I ain't generating one with AI, I would rather not make a blogpost

[shell tusk]

What is the post about, smp scheduling?

[late oar]

yeah

also shitting on ACPI

[shell tusk]

Have it be a pile of cpus

And on fire because of acpi

Im so good at design you should hire me

[late oar]

okay I think I know where it comes from

I have core 0 handling keyboard interrupts, and a cat running on core 1
cat is woken up to handle a keyboard input. It handles it, and then goes back to sleep, dequeuing it from the core 1 (that's what it should be doing)
but a next keyboard input comes in (on core 0), in between the moment cat atomically switches from state Running to Sleeping and the moment we dequeue it from the run queue (on core 1)
this keyboard input atomically switches cat back to the Running state (on core 0), then attempts to enqueue it in a run queue, but it does nothing since it notices it is already in a queue (since the core 1 didn't dequeue the process from the previous transition yet)
then, the core 1 dequeues the process, ending up with a cat in Running state without being in a run queue

that was non-trivial to find

and I am not even sure this is really the issue. I'll attempt to find a fix this evening

btw the Mutex in my kernel's code is actually a spinlock and the thing I've called WaitQueue is actually a badly implemented mutex (it does memory allocations to queue processes instead of having an internal linked list in the process's structure)

[late oar]

I have spent the last few days trying to figure out how to expose a QEMU VM on the local network with a static IP and I failed 👍

I am fed up about this, so it's time to get back to work on Maestro

[deft sonnet]

So fuckin real

Turnstile_Acquire is just spinlock_acquire

We can’t afford real locks these days 😔

[late oar]

okay so, I think it works

It may be time for me to review the PR for SMP and if everything is good, I guess I'll merge it

insertions/deletions in the PR:

@silent lava idk if you know, but I also have a blog about Maestro btw. The last article has a video of a rotating ASCII doughnut
https://blog.lenot.re/a/compiler

[silent lava]

Niiiice

One day I'll get there with BadgerOS... This gave me some renewed drive to actually get me there.

[late oar]

‼️

[silent lava]

I'd guess I'm only about 5-10% as far as you are though :P

[late oar]

I've taken way too long to get there though. The project started in 2018

I am so slow

[silent lava]

Faster than my average speed so far :P

I started in like 2020 and have way less progress

[late oar]

meanwhile, @shell tusk is going super fast

and then you have @wintry wolf who's going at light speed

[silent lava]

You only have so much time

Especially if, like me, you're full-time studying CS at uni

[shell tusk]

I am studying at uni and working full time I am just mentally insane

[silent lava]

CS is mostly easy asf though, most of the things I encounter is stuff I've already learned from, among other places, here

[late oar]

turns out my OS started as a uni project

I am just continuing this project even though I left uni

[silent lava]

My OS started before I started uni

[late oar]

I also have a fulltime job, but it's full remote and it's very flexible. I get to choose the time at which I start and stop working, and the amount of time I work each day

[silent lava]

That is an awesome privilege

[late oar]

indeed

I got this job thanks to my OS btw

[silent lava]

HAHA NIIICe

[elder sonnet]

Are they searching for remote package maintainers?? I’m available and I have plenty of experience with that

[late oar]

the company noticed it, contacted me on linkedin, offered me lunch at a restaurant and a job

[shell tusk]

insane

[late oar]

I think they actually do, I can check tomorrow if you really want it

[silent lava]

I'm probably going to end up looking for some kernel dev job or smth after uni. IDK. That'd be my preference.

[elder sonnet]

Good OSses mentioned but no Managarm. It’s time for another big ticket item to drop I guess. Time to get back to work

[late oar]

oh yeah of course Managarm is going very fast too

[elder sonnet]

Wait what unironically? That’s insane if true. If I may ask, which company do you work at?

[elder sonnet]

It comes in waves usually

[shell tusk]

I would love to work with kernel dev but unfortunately I live over 30° south of the equator so thats a no go

[late oar]

yes, and it's https://www.clever-cloud.com/

[inner pilot]

so did i

[elder sonnet]

Just move lol

[elder sonnet]

I’ll keep it in mind, I don’t think they have a jobs page do they?

[silent lava]

Where in the world are the kernel dev jobs anyway? Because moving to the USA is never gonna happen for me.

[late oar]

they have a job page but there's none on it at the moment

[late oar]

either US or EU, mainly

[silent lava]

Good thing I live in the EU

[late oar]

There's Suse in Germany who has kernel jobs

Bootlin in France

probably other companies in other countries

[elder sonnet]

Marvin works there

Join him

[shell tusk]

send remote eu job that will accept random south american with double citizenship and fucked up schedule thansk to uni

[late oar]

I think my company doesn't hire outside of the EU because GDPR

[silent lava]

I'm gonna do that after Uni if they'll let me

[elder sonnet]

I’ve looked at it more than once

But no package maintainers job open and I’m too smooth brained for actual development

Be advised that you’ll likely go to Germany for that tho

[shell tusk]

do package maintaing for astral I will pay you in paper clips and a beer every month

[elder sonnet]

I mean I like beer

[silent lava]

I live 200m on the NL side of the border already

[elder sonnet]

Which is bad cuz we all know Netherlands superior

[silent lava]

GEKOLONISEERD

[elder sonnet]

Oh damn I thought my dad was good with 700m outside of the border

[late oar]

I think Redhat is hiring pretty much everywhere in the EU
There's also Canonical in the UK

[elder sonnet]

Precies makker!

[inner pilot]

lol no

[elder sonnet]

Netherlands only has some sales office

[inner pilot]

0 jobs here

literally

[late oar]

Terrible

[elder sonnet]

But I know at least one senior kernel engineer that lives close to me and works at red hat

[silent lava]

Actually it's 400m but still

[elder sonnet]

I have had the privilege of working with him for an uni project

[silent lava]

Über close to ze border

[elder sonnet]

That shit even got a FOSDEM presentation out of it and is now upstream in libcamera

So that was fun

[late oar]

I want FOSDEM

[silent lava]

God damn this discord server is so fucking cool

[late oar]

can we have FOSDEM every 3 months please 👉👈

[silent lava]

Where else do I get to meet people like you all

[elder sonnet]

Real

[late oar]

FOSDEM

[elder sonnet]

FOSDEM FUCK YEAH

[silent lava]

I should go there at some point but have been reluctant because the density of people is very high and autism no likey

[late oar]

last fosdem was the first time I came and I am definitely coming back

[silent lava]

When we go there I should also introduce y'all to some of my badge.team friends.

[elder sonnet]

As a fellow dutchie with the tism, it’s so worth it. I get to meet up with friends from here, meet likeminded people, see cool talks (and talks about Managarm sometimes!) and in general have a good time. The Managarm gang usually has a solid presence there so I start with that, but I’ve seen loads of stuff. Idk if you know lore but sortix is a well known hobby OS. We (Managarm gang) met Sortie, the developer behind sortix, last FOSDEM. I looked up to those kinds of projects (and still do!). That’s just amazing. Talks itself are also interesting as hell to go to. I did miss the meet with RFC, I was in another talk so we’ll have to fix that next year @late oar

[late oar]

I think we met real quick but we didn't talk much

[elder sonnet]

And depending on your wants and needs, you can just go to your place of stay during FOSDEM in the evenings and enjoy a chill evening alone. Managarm gang usually makes either Brussels unsafe (bar go brrr) or we have enough beer stocked at our place to enjoy there

[elder sonnet]

Nah we missed each other. You were already gone when I got to the cafeteria with Leo

[late oar]

I was eating fries next to you and @cursive lintel who was fixing his GPS receiver 2 hours before his talk iirc

[elder sonnet]

Look, just a good excuse to fix that next year

[elder sonnet]

That wasn’t me I’m afraid

[late oar]

okay I may be mixing you up with someone else

[elder sonnet]

Solution: we try again in 2026. How unfortunate

[late oar]

‼️

[silent lava]

I'm going in 2026

[elder sonnet]

But in general, FOSDEM I can highly recommend. It’s the biggest open source conference in the world nowadays so it’s the thing to visit. The first edition was kinda like what am I doing here the first few hours. Then shit was lit

[elder sonnet]

Tijd om zuid Nederland weer te annexeren!

[silent lava]

Haha gekoloniseerd

[tough onyx]

bro buy me a plane ticket i will buy you a beer

[shell tusk]

fr

[elder sonnet]

Bro no money. It is still not guaranteed I can go (tho we’ll know after tomorrow or potentially end of month and it’s like 95% sure)

[elder sonnet]

Same bro. I wish. OSDev staff and regulars meetup at FOSDEM would be baller

Alas, I can’t fund that atm

[tough onyx]

nah maybe one day if im in europe during that time I'll go

but that's unlikely

i say you guys all come to north america

[shell tusk]

lets meet in the closest point to everyone, somewhere in africa

[tough onyx]

no the closest point would be somewhere in the atlantic

lets all get a boat

and meet at predetermined coordinates

[shell tusk]

yatch meetup

[late oar]

likely in the middle of an ocean, or in a war zone

let's meetup at the mali-burkina faso border

[elder sonnet]

Meet at NULL,NULL

[elder sonnet]

I want to. Got a bunch of space stuff to see.m

[shell tusk]

I want to go to america and get vanished by ICE for having an accent 😻

[elder sonnet]

Lmao

[inner pilot]

[fast halo]

Null Island

[late oar]

my TTY's code is such a pile of garbage

there's obvious deadlocks in it

[late oar]

I've put my hand on a x86 macintosh. I'll put it my Maestro's testing bench when it will exist

It's taking quite an amount of space though, and it's heavy too

I've just reminded myself I also have a Synology NAS laying around. That could be used for testing a RAID implementation I guess?

[late oar]

I have started my attempt at porting make and other stuff

[late oar]

I have attempted to cross-compile m4, ncurses, diffutils, file, findutils, gawk, grep, gzip, make, patch, sed, tar and xz for Maestro.
The following succeeded: diffutils, file, grep, gzip, make, patch, sed and tar
The rest needs their build script to be fixed

I think I need something to prepare a whole disk with a ready-to-use system on it.
Because like, the current disk I use, I have built it by hand a long time ago and I just keep using it

[late oar]

okay nvm, the ones that succeeded were linked to glibc instead of musl

FUCK

[elder sonnet]

lol

Idk if you’re using something like xbstrap or jinx but it massively helps you out with this stuff

[young ermine]

they have some NIH solution iirc

[elder sonnet]

As long as it’s a good one that works I guess

[late oar]

nop

it's invented here

https://github.com/maestro-os/blimp

but I've had a look at xbstrap a few months ago and it helped me fix my tool a bit

the bootstrap/ directory has the stuff to build a cross compilation toolchain

and https://github.com/maestro-os/blimp-packages has packages

[late oar]

okay, so now that I have the right compiler, I've managed to build diffutils, file, findutils, grep, gzip, ncurses, patch, sed and tar
So make does not build for some reason

fnmatch.c:124:14: error: conflicting types for 'getenv'; have 'char *(void)'
  124 | extern char *getenv ();
      |              ^~~~~~
In file included from ./stdlib.h:36,
                 from fnmatch.c:40:
/home/luc/Desktop/dev/blimp/bootstrap/sysroot/usr/include/stdlib.h:53:7: note: previous declaration of 'getenv' with type 'char *(const char *)'
   53 | char *getenv (const char *);
      |       ^~~~~~

fnmatch.c:299:33: error: too many arguments to function 'getenv'; expected 0, have 1
  299 |               posixly_correct = getenv ("POSIXLY_CORRECT") != NULL ? 1 : -1;
      |                                 ^~~~~~  ~~~~~~~~~~~~~~~~~
fnmatch.c:124:14: note: declared here
  124 | extern char *getenv ();
      |              ^~~~~~

well, let's go to sleep. I'll try to figure this out tomorrow

[fast halo]

do you have TTY support within the kernel? or are you just using a pipe duplex and communicating with user space

[late oar]

TTY is in kernelspace

Is it supposed to be implemented in userspace?

[inner pilot]

"supposed" is a strong word

[fast halo]

I want to implement TTYs in userspace, yes

For my own design philosophy, I don't think kernel space is a good place to put TTYs

[inner pilot]

i won't put ttys into the kernel when i get to it

i expect a user program to register itself as a DRM master

[fast halo]

maybe not like that

the init program would run the terminal host which would also run the first shell

[inner pilot]

wdym

[fast halo]

i expect a user program to register itself as a DRM master

[inner pilot]

it's not really different from what you said