Maestro - Linux-compatible-ish OS in Rust

[late oar]

I have no idea what's going on

[late oar]

okay after investigating for the whole day, I am now fairly sure this is an issue with how I determine if a physical page is used in multiple places or not

[late oar]

merged

(finally)

[late oar]

so it was that, and also when mapping the executable file, I was mapping the page in the cache even though the mapping was MAP_PRIVATE (had to make a copy of it instead), so when bash modified a static variable (for example), it was modified for all running bash

[coral wing]

agree

[late oar]

banned

[coral wing]

freedom of speach left the chat

but its still a nice kernel

[late oar]

thx

so I think the next step is to get gcc or clang running

[worn cosmos]

:O

[late oar]

yeah

been waiting a very long time for this moment 🥲

[worn cosmos]

well good luck, very exciting milestone indeed

[fierce orbit]

wait holy fuck maestro dev is here???

damn

[late oar]

I am indeed here

[fierce orbit]

based!

[late oar]

I have posted my new blog article!
https://blog.lenot.re/a/page-cache

[desert hearth]

excellent timing, I am waiting in line for something

I'll have a read

[late oar]

I think I will renovate my blog at some point. The design could be improved

like I could use stuff from there: https://www.fffuel.co/

[worn cosmos]

I quite like your blog's design tbh

although I suck at web dev and graphic design so idk what looks good 😭

[late oar]

yeah but the 2 cyan blobs on the left and in the bottom right corner feel awkward to me

I think that could be improved

[late oar]

I am currently switching the kernel to rust edition 2024 and this is rather tedious

I have new UBs appearing out of nowhere. I guess they were already present and the update is only revealing them

[inner pilot]

really? isn't there an auto upgrader

i used that

[late oar]

no I mean

that part is easy

but fixing the UBs is not

[inner pilot]

i probably have tons of ub already with no code to show for it lmfao

[late oar]

btw today I discovered multiboot2 does not care about the alignment of ELF section headers and will align them to 4 even though the kernel is a 64 bit ELF (hence requiring alignment at 8)

[inner pilot]

you use the multiboot protocol?

TIL

[late oar]

multiboot2, yes

[inner pilot]

when limine

[late oar]

idk

[inner pilot]

or uefi boot

i might make a preloader with uefi-rs

[late oar]

is that a thing?

[inner pilot]

wdym

like an EFI stub

what the linux kernel does

[late oar]

I don't know how Linux boots

[inner pilot]

oh

it's basically an ELF image masquerading as a PE executable

which has a simple entry point to call into linux' main

[late oar]

ah yes, PE executable

my hate for microsoft is resurfacing

[inner pilot]

yep

you can fake your way with linker script hacks

[late oar]

well I guess I'll have a try at that someday

[inner pilot]

i'll attempt doing that this week

feel free to copy

[late oar]

👌

also someday I will have something else than VGA text mode for display

[inner pilot]

oh yea, by doing uefi boot you get a GOP

free framebuffer

[late oar]

no thanks I don't like donald trump /j

[late oar]

It has basically the same functions as VESA, you can query the modes, set the modes. It also provides an efficient BitBlitter function, which you can't use from your OS unfortunately. GOP is an EFI Boot Time Service, meaning you can't access it after you call ExitBootServices(). However, the framebuffer provided by GOP persists, so you can continue to use it for graphics output in your OS.
Does that mean you can't use that to do modesetting later? (for example)

[inner pilot]

you wouldn't do modesetting on the GOP

it's only meant as an initial console

your actual video output should be done by a kernel driver

[late oar]

I see

[inner pilot]

tbh i don't think having modules is worth it

i'd rather just link everything statically

because that saves like 70% of binary size lol

this is the debug kernel

and this is the release kernel

over 900 symbols in the symtab

and i've already done stripping

if i also strip the .rustc section it goes down to 900kib

still a lot for basically nothing

[late oar]

my debug kernel is 17 661 096 bytes and the release one is 1 589 112 bytes

[inner pilot]

you build as a binary, right?

[late oar]

I did not do that. What's this section used for?

[late oar]

yes

[inner pilot]

linking metadata

compiler info, abi info

but it's completely useless to a dynamic linker once everything has been linked

do you have a kconfig equivalent?

or any config beyond cargo features

[late oar]

yeah I have a toml file which is read by the build script

[late oar]

how do you load modules if the kernel is stripped?

[inner pilot]

[teal dock]

holy shit rust binaries are huge

[inner pilot]

dylib

[late oar]

yeah that's one of the issues with this language

[inner pilot]

by compiling as a dylib all local symbols are discarded

but they're kept if they're used globally

[late oar]

I don't even remember if my modules are dylibs

[teal dock]

what's dylib

[inner pilot]

.so

[late oar]

turns out they are

[teal dock]

just shared object?

[inner pilot]

but in rust

there's an ABI difference

cdylib only exports C externs

dylib can do rust dynamic linking

which is scuffed tbh

but it works

[late oar]

@inner pilot how do you load your modules?

iirc what I do is that I parse the ELF, and then when doing relocations, when I need the offset of a symbol that is not in the module, I lookup in the kernel's symbols

[inner pilot]

i do that

[late oar]

wait I am using symbol names while doing that?

I'll check my code

[inner pilot]

you should

https://github.com/menix-os/menix/blob/main/kernel/src/generic/module.rs

this is my code

[late oar]

yeah I do

[inner pilot]

i need to clean up my loading code but it's basically this

[late oar]

but if you stripped the kernel, you don't have any symbol left, right?

[teal dock]

don't strip it

and don't use lto

[late oar]

yeah I tried to use lto, that breaks everything

[inner pilot]

if i compiled as a bin, yes

not if you compile as a dylib

that keeps the .dynsym .dynstr tables

[late oar]

your kernel is a dylib?

[inner pilot]

yes

[late oar]

that explains a lot

[teal dock]

can limine load it?

[inner pilot]

yes

it's PIE

[teal dock]

is it -shared or just -pie

[inner pilot]

shared

with an entry point

[teal dock]

hmmm

[late oar]

so you don't support GRUB? I think GRUB does not like relocatable ELFs

[inner pilot]

i don't

i'm going to make the UEFI boot an option anyways so you can boot the image without a bootloader

i might also support the linux boot protocol because there are some platforms (like the wiiu) that only really have linux boot loaders

again, I'm still doubting the usefulness of modules in the kernel

all it does is increase size

see what linux' modules do

they're like multiple gbs in size (even compressed)

[late oar]

In my OS I'd like to ship only the drivers that you need, so I need to not have everything in the kernel

[teal dock]

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'g_uacpi_rt_ctx'; recompile with -fPIC

bro I am compiling with fPIC

[late oar]

I am glad I don't have linker issues right now. I hate those

[inner pilot]

same

i'm happy

all compressed linux drivers on arch are 100MB

that's a lot

the nice thing is that you can really compress this stuff well

because a lot of it is redundant data

[late oar]

someday I will have something else than PATA to access the disk

I've heard NVMe is easy to implement

[inner pilot]

nvme:

it's piss easy

[late oar]

nice

[inner pilot]

not as simple as pata

but simpler than others

[teal dock]

what's -nostartfiles equivalent for lld

[inner pilot]

-nostartfiles

[teal dock]

error: ld.lld: error: unknown argument '-nostartfiles'

[inner pilot]

huh

[late oar]

how queer

[teal dock]

I must report this to osdev server post-haste

[inner pilot]

wait

-nostartfiles isn't an ld option

that's a gcc flag

[teal dock]

smh

[late oar]

I have a memory allocator bug that appeared, that seem to be fixed by doubling the size of the boot kernel stack

huuuuuh

[teal dock]

welp I don't know what could cause this

[teal dock]

to how much?

[late oar]

65536 bytes

that feels like a lot

[teal dock]

it is

but rust

[late oar]

I mean, 32768 was enough before I went to rust 2024

there must be an issue somewhere

[inner pilot]

i have 256kib to be safe

[late oar]

help

[teal dock]

I use 16 kib

[late oar]

let's use gdb to check if this is really the reason

[late oar]

I used to have that. I already doubled it once

[teal dock]

if I'm calculating correctly, I use 2648 bytes of stack

I'm not so sure though

no yeah that's correct

just fill your stack with 'A' and check how much of it isn't overwritten

[late oar]

I thought some of my unit tests were run twice because of a weird UB. Turns out I just had 2 tests with the same name

When crashing, my kernel is using 114 736 bytes of stack

wtf

[teal dock]

do you use ton of recursive functions or something?

[late oar]

I don't think so. I'll recheck to be sure (maybe something is going wrong)

there's only 6 functions deep to reach the test that crashes

[teal dock]

what's the test?

check stack usage before those function calls

[late oar]

with only 6 functions deep I have already used 35 424 bytes

what in the actual fuck

[teal dock]

[late oar]

turns out someone else pointed out that issue on reddit
https://www.reddit.com/r/rust/comments/1dl0igg/rust_stack_usage_is_bad/

[inner pilot]

lots of stack allocs?

[late oar]

well I guess I will give more boot stack when compiling in debug mode

easy solution

[late oar]

the guy on reddit says the rust compiler relies on llvm removing unnecessary allocations

which are not removed in debug mode I guess

[inner pilot]

that's so dumb

[late oar]

yeah

[inner pilot]

i guess just slap an opt level 1 on there

[teal dock]

offloading work to llvm? lol

[late oar]

isn't it going to make it annoying when using gdb?

[inner pilot]

might

[late oar]

idk I've never used any optimization level other than 0 or 3

[teal dock]

I use 03 and it works fine

as long as the symbols are there, gdb doesn't seem to care

[silk wren]

AFAIK -O1 shouldn't affect control flow structure that much. The real mindbending optimizations like certain forms of loop unrolling, vectorization, other black magic that idk tends to happen at -O2 or -O3.

But gdb should work with binaries produced at either. Just that the one-to-one mapping to source lines might go away with optimization.

[late oar]

well to be sure I will just be adding more stack when building in debug mode

but now I am curious. Maybe the issue is also present in release mode. I'll have to check that

[late oar]

okay I am done switching to Rust 2024

[balmy plume]

what the hell

what's causing this?

[late oar]

Rust I guess

[inner pilot]

-O0

we have determined that rust doesn't remove unnecessary alloca's in opt level 0

[balmy plume]

ah

[late oar]

today I am removing unused code (yesterday I removed #![allow(dead_code)] just to see and there are massive amounts of useless stuff)

[inner pilot]

fun fact this actually affects binary size

[teal dock]

how

[inner pilot]

no clue

i also recently removed the allow dead_code and it shrinked my binary

idk why

[late oar]

[late oar]

on my side it does not

exact same size

[inner pilot]

weird

ah wait it might be because i'm a library and it included unused symbols

or whatever

maybe I'm just demented

[late oar]

likely

[late oar]

I found a useless #![allow(unused_imports)] somewhere in my code for some reason. Removed it. Got 95 warnings of unused imports

[teal dock]

that would be nice in C++

(for module imports)

I guess I just have to wait

[late oar]

okay, I need to find the motivate to cross-compile stuff

first binutils, then gcc

[inner pilot]

it's not that bad

binutils is really easy

gcc not so much

[late oar]

yeah I've already done that before

but I have PTSDs of autoconf errors

[coral wing]

PTSD sounds like the name of a config bit in x86/amd64

[late oar]

Indeed

[late oar]

and so it begins

In file included from ../../../gprofng/libcollector/collector.h:29,
                 from ../../../gprofng/libcollector/iolib.c:35:
../../../gprofng/libcollector/iolib.c: In function '__collector_create_handle':
../../../gprofng/libcollector/../src/collector_module.h:123:22: error: too many arguments to function '__collector_util_funcs.write'; expected 0, have 3
  123 | #define CALL_UTIL(x) __collector_util_funcs.x
      |                      ^~~~~~~~~~~~~~~~~~~~~~
../../../gprofng/libcollector/iolib.c:231:11: note: in expansion of macro 'CALL_UTIL'
  231 |           CALL_UTIL (write)(2, errbuf, CALL_UTIL (strlen)(errbuf));
      |           ^~~~~~~~~
../../../gprofng/libcollector/../src/collector_module.h:115:13: note: declared here
  115 |   ssize_t (*write)();
      |             ^~~~~

how does this even happen?

I am not even started cross-compiling stuff. Right now I just want to build binutils for my current computer, with a target to x86_64-unknown-linux-musl

[late oar]

also bash's management of the tty is the worst thing I have ever used. The text on my screen is just a pile of shit and I have to guess whatever I am editing

okay fuck this I am going to sleep

[late oar]

so I managed to cross compile binutils. I just updated from 2.42 to 2.44 and it fixed it. But I think I just actually needed to add --enable-gprofng=no

[worn cosmos]

pretty nice. does it run?

[late oar]

Idk yet. I am trying to cross compile GCC now

[late oar]

I may give up on this for now. I am kinda fed up with cross compilation right now

I guess I'll just test binutils for now

[late oar]

so I tried running ld, objdump, readelf and nm

they all do nothing and return with at status code 0

rebuilding my kernel with the option to print system calls to figure out what's going on

[inner pilot]

just implement ptrace

[late oar]

[inner pilot]

i'm doing that at least

[late oar]

okay so it seems the program is segfaulting during execve?

I just recalled I probably didn't implement support for ELF interpreters, and this program is asking for it

yeah, so next step: implement ELF interpreter support

[late oar]

It's actually after. I said that because it crashed before execve returns but I just remembered execve does not return on success

[late oar]

so I ran ld and it printed:

musl libc (x86_64)
Version 1.2.5
Dynamic Program Loader
Usage: ld [options] [--] pathname [args]

I think I've failed something lmao

okay so musl does not give a shit about AT_EXECFD. I must map the program in memory from the kernel and pass it the phdr with AT_PHDR if I understood correctly

[late oar]

guys, I think I've got ld, readelf and objdump running (didn't try to pass any argument to ld)

screenshot of objdump -S /usr/bin/cat while running

nm works too

I tried copying a .o file to the disk to test ld with something and now the filesystem is corrupted. I guess that means it's time to sleep

well I've decided to give it a new try before and I've got this:

Tried adding crti.o, etc... and now only the last line of error remains

[worn cosmos]

amazing!! nice one

[errant venture]

yeah great work

is that running native linux apps or did u have to cross compile them

[late oar]

Just cross compiled to use musl instead of glibc

Target triple is x86_64-unknown-linux-musl

[errant venture]

nice

whats the issue with glibc?

[late oar]

It uses a syscall that I don't understand (can't remember how it's called)

[errant venture]

rseq?

or something like that

to detect preemption in userspace

its not that bad lol

[late oar]

Probably

[errant venture]

easier to cross compile GCC than to implement a syscall

[late oar]

ld produces a file with a size of zero and I have no idea why

[fierce orbit]

Dumb question, has networking been implemented yet? I remember reading about it on iirc a post on the website but can't remember when nor what the current status is

Maestro + Uutils + Cosmic + Fish shell when

[worn cosmos]

from a quick look in the repo it seems to yes, including tcp to some extent. idk how complete it is

[late oar]

No it does not work

I started the implementation but it's very incomplete

[late oar]

the default flavour of Maestro will likely ship busybox but there will be an uutils package too as an option

Also I want to have my own DE instead of Cosmic. I guess one could swap it for Cosmic anyways if they want too

[late oar]

I remember that I made the choice a long time ago that on Maestro you could not catch a SIGSYS (signal you get when you attempt to execute a syscall that does not exist). But I can't remember why

(this is not the case on linux iirc)

[inner pilot]

why does a nonexistent syscall cause a sigsys?

should that not just return enosys and be done?

sigsys is only really used for seccomp

[late oar]

huh, I wasn't aware of that

[late oar]

why does the ELF I linked with ld attempts to get an interpreter at /lib/ld64.so.1 even though I don't have any file with this name anywhere?

[little escarp]

i think that's just the default interpreter path ld uses?

[inner pilot]

that's what's defined in the ld defaults

you shouldn't really link with ld

link with gcc instead

[late oar]

so I have to cross compile gcc then

[little escarp]

yeah iirc gcc explicitly specifies --dynamic-linker ...

[late oar]

I'll try to specify myself and see what it does

[late oar]

so now I have the right path for the interpreter. When I execute the resulting program on maestro it gives me a segfault. When I execute it on linux it works. So there's something off in the kernel I have to fix

[late oar]

there's this line in musl 1.2.5:

} else if (type==REL_GOT || type==REL_PLT|| type==REL_COPY) {

which triggers me

(no space in between REL_PLT and ||)

[little escarp]

the musl code style is odd in general

like look at this check that the two ranges don't overlap in memmove ```c

if ((uintptr_t)s-(uintptr_t)d-n <= -2*n) return memcpy(d, s, n);

[late oar]

[pastel summit]

the style is "markov chain of assorted old c software"

[little escarp]

i still wonder why the left side has the -n

i you add n to both sides of the inequality you lose the -2* part on the right side

unless it's for overflow or something

[pastel summit]

no one is brave enough to use this style anymore https://github.com/v7unix/v7unix/blob/master/v7/usr/src/cmd/sh/cmd.c#L65

[soft depot]

dear god

[errant venture]

What the actual fuck

[inner pilot]

it's modular 😃

[frail junco]

windows

lol

[pastel summit]

stephen bourne came from britain where algol-68 was a very popular language at the time

so he tried to recreate it with c macros

compare to https://github.com/coolbikerdad/Algol-68RS-Source/blob/main/mm/mm_utilities.a68

[late oar]

I can't get the program I am linking with the ld on my kernel to work (musl's dynamic linker can't find the printf symbol). That makes me unhappy
(I am trying to link a .o that I compiled somewhere else (Linux). Maybe that's not something I should be doing anyways)
So I guess I will attempt again to cross compile gcc to make it run on my kernel, then I'll compile the .o from my kernel itself and see if that changes anything

[inner pilot]

@late oar what's the point of separating Nodes and Entrys? don't they kinda do the same thing?

(Noob question)

[late oar]

you can have several hard links to the same inode

Node is an inode and Entry is a hard link

[inner pilot]

so Node is an actual entity

and entry is just a cached link?

[late oar]

entity?

[inner pilot]

like a vfs primitive

[late oar]

yes

I talk about all of that here: https://blog.lenot.re/a/page-cache

[late oar]

Entry is one too?
If you mean by "entity" something that is on disk, then Entry represents a directory entry

[inner pilot]

that's not really what i meant, but whatever

[inner pilot]

actually very comprehensive read

[late oar]

🎉

[inner pilot]

i get it now

(kinda obvious but my brain shuts off when talking about graphs)

[late oar]

I hate cross compilation

[inner pilot]

i love cross compilation

[late oar]

How

[inner pilot]

how do you compile

by hand? shell scripts?

[late oar]

Shell scripts

And it never works, I always spend days on compiling a single package

[inner pilot]

tried jinx or xbstrap?

[late oar]

I have my own thing https://github.com/maestro-os/blimp

[inner pilot]

hm

[late oar]

The cross/ dir has the necessary to build a toolchain, and then I use the builder/ with the descriptors in https://github.com/maestro-os/blimp-packages

[inner pilot]

i can only say that cross compiling with jinx and xbstrap was a breeze compared to running my own builder

(I'm sure you didn't want to hear that)

[late oar]

(Yes )

[late oar]

currently each time I attempt to build a package, the builder is re-downloading the sources because I have no caching on this. I guess I should do that (that could save me both time and mobile data)

[inner pilot]

mobile data

[late oar]

yes I am not at home right now

I used like half of my monthly quota in like 4 days. I have to do something about this

[inner pilot]

how much do you get

i'm on 15GB/mo

[late oar]

140GB/mo

for ~10 euros/mo

I should change my subscription btw. I can get more for less expensive now (like I have seen 200GB for 9 euros recently)

[inner pilot]

wtf

i'm being scammed

i have 15/mo at 12€

wtf germany

[late oar]

France is kind of an outlier in this

every one in every country is actually getting scammed

[inner pilot]

bruh

cant i just get a french sim and use eu roaming

[late oar]

probably

I have 25GB roaming I think

[inner pilot]

does france have sim govt ID requirements

[late oar]

good question

[inner pilot]

i have unlimited roaming actually

or rather, as much as my main contract

[solid rose]

jesus that is cheap

[late oar]

not cheap enough imo

[solid rose]

lmao

for me its pretty cheap too but i have a contract

its like 5 euros for 30gb of data, unlimited national minutes and like 100 international minutes iirc?

[teal dock]

I can get 3 months unlimited data for 25€

[young ermine]

i get unlimited data for like 10 eur

[frail junco]

i get unlimited for 13€ / month

with 5G

[solid rose]

5G ja standard nowadays

[soft depot]

you get multiple gigabytes?

we get like 250MB + 500 texts + 500 minutes for $15/mo over here

[solid rose]

250mb????

[late oar]

united states?

[coral wing]

i get 4GB for like 7,99€/28 days 💀

[late oar]

update: I am giving up on clang + musl. I am attempting cross compilation with gcc + musl right now

[inner pilot]

😔

[errant venture]

Just support that damned syscall glibc wants lol

[late oar]

never

(I want a musl system, glibc's bloated)

[tough onyx]

@late oar https://musl.cc/

I used this

[fierce orbit]

use mlibc

[late oar]

That could indeed be more convenient

[inner pilot]

or mlibc

[late oar]

okay guys I think I've finally managed to cross compile gcc + musl

incredible

[solid rose]

damn

[late oar]

(Didn't test it on Maestro yet)

[solid rose]

makes sense

still, i got stuck last time i tried to cross compile gcc

[late oar]

bash-5.1# gcc test.c
Error relocating TODO: vsprintf: symbol not found
Error relocating TODO: memcmp: symbol not found
Error relocating TODO: remove: symbol not found
Error relocating TODO: fwrite_unlocked: symbol not found
Error relocating TODO: sscanf: symbol not found
Error relocating TODO: signal: symbol not found
Error relocating TODO: freopen: symbol not found
Error relocating TODO: memset: symbol not found
Error relocating TODO: stpcpy: symbol not found
Error relocating TODO: memmove: symbol not found
Error relocating TODO: fflush: symbol not found
Error relocating TODO: strrchr: symbol not found
Error relocating TODO: kill: symbol not found
Error relocating TODO: strncasecmp: symbol not found
Error relocating TODO: ftell: symbol not found
Error relocating TODO: printf: symbol not found
Error relocating TODO: fgetc: symbol not found
Error relocating TODO: fread_unlocked: symbol not found
Error relocating TODO: putchar: symbol not found
Error relocating TODO: exit: symbol not found
Error relocating TODO: strerror_r: symbol not found
Error relocating TODO: fileno_unlocked: symbol not found
Error relocating TODO: mkstemps: symbol not found
Error relocating TODO: putc_unlocked: symbol not found
Error relocating TODO: __fsetlocking: symbol not found
Error relocating TODO: atoi: symbol not found
Error relocating TODO: ungetc: symbol not found
Error relocating TODO: memchr: symbol not found
Error relocating TODO: fputc_unlocked: symbol not found
Error relocating TODO: fputs_unlocked: symbol not found
Error relocating TODO: strtok: symbol not found
Error relocating TODO: getc_unlocked: symbol not found
Error relocating TODO: sprintf: symbol not found
Error relocating TODO: stderr: symbol not found
Error relocating TODO: stdout: symbol not found
Error relocating TODO: stdin: symbol not found

wonderful

(I think that's an issue with the way I load programs with interpreters)

at least this is working

ld, readelf, objdump, etc... are not working anymore but I think that's because I compiled them statically at the time

[late oar]

now I have /usr/bin/gcc instead of TODO

[worn cosmos]

pretty nice progress

[fierce orbit]

I should try maestro again

pain

[late oar]

cd kernel before cargo build

[inner pilot]

imagine reading the readme

cringe

[fierce orbit]

ohh sorry, thanks

[late oar]

FINALLY

now I have to try compiling something

The issue was the memory map of the process I think. I was overwriting things with mmap

and I think something is still fucked up since gcc can't manage to run cc1. execve fails with ENOMEM

[worn cosmos]

nice!

[late oar]

I've got cc1 executed (turns out 128MB of memory is not enough, I just increased to 1GB since the executable is like ~300MB). Now I need to implement the sysinfo system call

[fierce orbit]

what was this guy's deal 💀

oh he's still here

ruh roh 💀

[young ermine]

it's @tawny drum

[late oar]

I have sysinfo, now gcc is segfaulting (I think) after cc1 returned successfully. Investigating...

[inner pilot]

bro scrolled up

[fierce orbit]

I acc pressed the jump to top button

[late oar]

well tbf the kernel's code wasn't great at the time

to say the least

(it was crap)

[teal dock]

that's mint's old account

wow 1100 stars in 1 day?

how did you achieve that? (asking for a friend )

[late oar]

a friend of mine posted my first blogpost on hackernews

[teal dock]

ah

[inner pilot]

bruh

[late oar]

https://news.ycombinator.com/item?id=38852360

Uriopass

Maestro: A Linux-compatible kernel in Rust

Hacker News · Jan 3, 2024, 9:59 AM

[silk wren]

well tbf it isn't a very recent blog post

[inner pilot]

is maestro still linux compatible?

i see that you dropped this branding

[late oar]

aims to be

well not fully, I am not going to reimplement 100% of it

you know it's like glibc and musl. Linux is glibc and Maestro aims to be musl

[inner pilot]

ah

i aim to be linux_like in terms of rust stdlib

[late oar]

huh no, it was a metaphore

I aim to be the de-bloated version of Linux

[inner pilot]

ever thought of having a linux driver compat layer?

[late oar]

that would be very complicated I think

[inner pilot]

yea, i get that part

i meant, maestro is going to target the linux OS type

while i'm targeting linux_like

https://github.com/rust-lang/libc/blob/main/src/unix/linux_like/mod.rs

[late oar]

I didn't even know this existed

[inner pilot]

lol

as you can see it's for OSes that differ from linux abi

or only have a few linux interfaces

[fierce orbit]

based

maestro + uutils + cosmic would be peak

[late oar]

default will likely be maestro + musl + busybox + custom DE

[fierce orbit]

ahh based

curious, your X implementation ("volva" I think?) is it from scratch or is it xorg-based?

oh wait

it's rust isn't it

nvm im stupid

[late oar]

xorg reimplementation in rust

[fierce orbit]

ah, noice

[late oar]

can't do much right now, will probably rewrite it

I kinda want to try wayland instead

[fierce orbit]

noice

[late oar]

jumps to address zero. broken PLT I guess?

it happens right after cc1 exits, so I guess they somehow shared the same memory space and cc1 corrupted gcc's?

[late oar]

some of the issues I have to fix right now (I'll be using this message as a reminder):

• the above issue (I still don't know where it comes from)
• waitpid returns in a loop when WCONTINUED is passed to it (I just check the status of the process and return if it's running. Instead I should add a flag to the process structure that I set when killed with SIGCONT and clear in waitpid)
• I should not read the whole ELF when executing it but rather read only the relevant structures (ehdr, phdr and interpreter path I think?), otherwise it's very slow when the ELF is big, which is the case for cc1 and that makes it very annoying to debug
• I must make sure brk and mmap cannot overlap each others (maybe that's the cause of the first issue)

[teal dock]

why not implement brk with mmap?

[late oar]

that's what I am doing but I just remembered I didn't put any limit on brk and it might just write over stuff allocated with mmap

the opposite is also true, mmap might select an unallocated region that shoud be reserved for brk (which will then be rewritten on top by later brk allocations)

[late oar]

Okay so the brk thing was not the issue but at least now I have support for MAP_FIXED_NOREPLACE

[late oar]

I have a beginning of a clue: when wait4 returns in gcc after cc1 exits, it returns to the address 0 because RCX (which contains the return address for sysretq) is set to 0 for some reason I have to figure out, when a signal is caught (which is SIGCHLD I think)

[late oar]

okay so I was not interpreting the sa_handler field in sigaction correctly. A value of zero should mean we ignore the signal

anyways so, now I have gcc calling cc1, as and collect2 successfully. Only remains ld which is failing because it can't manage to find libz.so.1, which prevents relocating /usr/lib/libctf.so.0.
I think I am getting close ‼️

On a side note, I really have to make ELF execution faster because that's really long and I think I have a memory leak preventing me from running the command several times

[late oar]

okay, I suspect libz might be shipped with the glibc and I don't have it because I use musl

I'll try building zlib myself tomorrow, time to 💤

[late oar]

compiled zlib and fixed it. Now I have ld running but it fails with this error:

/usr/lib/gcc/x86_64-unknown-linux-musl/15.1.0/libgcc.a: file not recognized: file format not recognized

COME ON, I AM GETTING CLOSE

plays epic music in background

[worn cosmos]

does a terrible little dance to epic music in background

[late oar]

I now suspect my lseek being broken might be the cause of this bug

An offset of 18446744073709550593 probably isn't normal

negative value cast to unsigned?

is lseek supposed to take signed offsets?

[little escarp]

-1023

?

[late oar]

maybe

[little escarp]

off_t is supposed to be signed

and yes you can seek backwards

[late oar]

that explains a lot

[late oar]

IT FUCKING WORKS

LETS GOOOOOOOOOOOOOO

imma make a screenshot

[little escarp]

yoo congrats

[fast halo]

Yes, in the case of SEEK_CUR at least

[little escarp]

also SEEK_END

[fast halo]

oh yeah

[trail swallow]

🥳

[fast halo]

so what did you have to go through to get here

namely, what did you do to make mlibc compile for your architecture

or whatever libc you are using

[little escarp]

maestro is doing linux abi compat so it's just glibc?

[late oar]

pain

[little escarp]

or are you using mlibc

[late oar]

I use musl for this

[little escarp]

ah

[fast halo]

oh linux compat

[little escarp]

makes sense actually

[fast halo]

oh ok

[little escarp]

i half expected that you just copied binaries from your host :^)

[fast halo]

yeah but supposing i wanted to use mlibc for a userspace distro with my boron kernel

what do i need to do

i imagine i need to create the sysdeps

implement them using my OS' syscalls

[late oar]

I mean, I could but also I want to use musl because the glibc's bloated

[little escarp]

i mean that's more or less it

[late oar]

I think I have one last fix to make before I can actually merge my PR. chmod changes do not seem to be flushed to disk so when I reboot the executable does not have the execute permission anymore

[fast halo]

yes but i mean like setup, compilation, that kinda thing

do i need to make a cross compiler? do I need to configure something?

[late oar]

aside from that, I had to rewrite all of the ELF loading thing because I was doing it wrong

[fast halo]

is it then easy to just configure something with my target's toolchain and make?

do I need linux for this or can I use wsl (1)

[little escarp]

i mean mlibc uses meson but building it is pretty standard as far as meson projects go

[fast halo]

oh for that question i was talking about things like bash and gcc

[little escarp]

you need a cross file that specifies what compiler etc to use, meson setup builddir --cross-file my-cross-file.txt <extra opts> --prefix=/usr, ninja -C builddir DESTDIR=/path/to/my/sysroot ninja -C builddir install

[late oar]

I cross compiled gcc by following: https://www.linuxfromscratch.org/lfs/view/12.3/index.html
The gcc you see running on the screenshot is the stage 2. I run the stage 1 on Linux

[little escarp]

well for most autotools stuff you'll need to at least modify config.guess/config.sub so it knows about your os

for gcc you need extra patches to add your target

likewise for binutils

this is probably better continued in a different channel and not in the maestro thread tho :^)

[fast halo]

i see

anyway i dont have plans right now to do this, i just need to mentally prepare lmao

[late oar]

glibc uses a system call that I am missing. I didn't understand what it does at the time (I still don't btw, I didn't look back at it) so I just used musl instead

I will implement it eventually

[little escarp]

which one out of curiosity

[errant venture]

rseq

[late oar]

yes

[errant venture]

all it does is userspace registers a struct with the kernel that saves the rip of abort pointer and the end of the critical section pointer

if userspace is preempted with an active struct like that, its rip is moved to abort on resume

so it can retry the per-cpu operation

[late oar]

I see

[little escarp]

ah i was thinking it was restartable sequences

[errant venture]

very good (and only) docs here https://manpages.opensuse.org/Tumbleweed/librseq-devel/rseq.2.en.html

[late oar]

btw what's needed to make a real time kernel? Is implementing rseq sufficient or is it more complicated than that?

[errant venture]

i dont think rseq is real time related

[little escarp]

yeah

[errant venture]

real time just means u dont have stuff that can disable preemption for an unbounded time

or something like that

[little escarp]

huh very harsh punishment

Restartable sequences must not perform system calls. Doing so may result in termination of the process by a segmentation fault.

[errant venture]

i mean

how would that even work

[little escarp]

i mean you could return an error code from a syscall?

[errant venture]

this is meant to be like locked store or wahtever

for per-cpu data

not heavy work etc

[little escarp]

i mean i get it's purpose

[errant venture]

idk why they forbid them

[little escarp]

and it makes sense that syscalls aren't allowed

[errant venture]

maybe too much extra code needed

[little escarp]

my guess is that a syscall can potentially cause you to be preempted?

or block

[errant venture]

from what ive seen in code

when u syscall there isnt a register frame passed to rseq for some reason

so they hard segfault u

[little escarp]

hard as in can't even handle the signal?

[errant venture]

uhh not sure

[little escarp]

lets try

[errant venture]

https://elixir.bootlin.com/linux/v6.15.1/source/kernel/rseq.c#L466

[late oar]

so maybe Maestro is real time then? I am not sure

[little escarp]

force_sig just ignores the mask but otherwise seems to be handleable

[errant venture]

guess its fine then

btw i just noticed

this is under CONFIG_DEBUG_RSEQ

[little escarp]

insane

with it unset it doesn't do anything

[errant venture]

does this mean u can syscall under rseq

[little escarp]

[errant venture]

kinda funny but ig

the main idea is probably that its super likely u do switch cpus between syscalls

becaues its a rebalance checkpoint and might block or take too long

[little escarp]

yeah

[errant venture]

so u would retry a ton of time

and it might just never work

https://github.com/torvalds/linux/commit/d7822b1e24f2df5df98c76f0e94a5416349ff759

this explains a ton as well

[inner pilot]

how so

[late oar]

mmap broken, did not support interpreter, wrong stuff in auxilary vector

[inner pilot]

damn

[late oar]

I think I have a deadlock when running g++

[late oar]

So I currently have the following issues to fix:

• Giving the machine more than 1GB of RAM in 64 bits makes the kernel crash (I fucked up virtual memory at boot apparently. PSE cannot be used on page map level 4 entries)
• Giving the machine more than 1GB of RAM in 32 bits makes the kernel crash while the first process is executing because the address in the page directory entry is too high it seems (idk why yet) ✅ (was trying to free the page pointed to by the last level of the page directory, which contains user data)
• The ELF parser reads the whole file in order to execute it (even though it requires only a few structures), which makes everything slow ✅
• The kernel thread that flushes dirty memory pages runs regardless of whether there are pages to flush or not
• The kernel thread that flushes dirty memory pages iterates on all pages regardless of whether they are dirty or not
• Executing g++ makes an infinite loop in the flush task (idk why yet). I suspect this might be due to an OOM? In which case my OOM handling is fucked up ✅ (yes it's due to an OOM, see the above issue)
• I need that kernel thread to also flush inode metadata (I guess I need to add a LRU on inodes too) ✅ (I just modify the data structure of the filesystem and it gets automatically flushed)

[errant venture]

PSE on level 4? 💀

Bro is using terabyte pages

[late oar]

that would have made init so much easier 😔

but now I wonder why this is working with 1GB or less

[late oar]

I've written the fix to avoid reading the whole ELF file. Now that's a lot faster
Since it is using a lot less memory, I can now run g++, which exits successfully.
But when running the resulting program, I get:

Error loading shared library libstdc++.so.6: No such file or directory (needed by /cpptest)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /cpptest)
Error relocating /cpptest: _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc: symbol not found
Error relocating /cpptest: _ZNSolsEPFRSoS_E: symbol not found
Error relocating /cpptest: _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_: symbol not found
Error relocating /cpptest: _ZSt4cout: symbol not found

might need to fix something in the compilation toolchain

[inner pilot]

@late oar does maestro's fs mount work reliably? I'm looking for clues as to how to implement my vfs but most implementations ive come across now are incomplete

[late oar]

yes it does

however I am not checking yet for nested mounts on unmount, which I should be doing

[inner pilot]

right

i just need a way to structure it in my head

also helps that it's in rust :3

[late oar]

the main function you would be interested in is here: https://github.com/maestro-os/maestro/blob/21118d70d0a186777d9bc7d0cc78509935c3d483/kernel/src/file/vfs/mountpoint.rs#L224

The VFS has a tree of Entry (equivalent to Linux's dentry), and when you create a mountpoint, it also creates an Entry for the root directory of the new mountpoint. The associated Node (inode) is the root of the mounted filesystem

When doing path resolution, we can just go through the Entry tree like nothing happened

[inner pilot]

that's similar to what i do

do you also do negative lookups?

i.e. path misses are cached in negative dentries

[late oar]

you also need to make sure you cannot remove the directory if it's a mountpoint

And you must make sure you cannot unmount a mountpoint if there's another one in it.
You can do this by having a reference to child mountpoint in your mountpoint structure. If there's any child, you cannot unmount

[late oar]

yes I do

[inner pilot]

neat

[late oar]

this is represented by having None in the node field of my Entry

I talk about it in my last blog article: https://blog.lenot.re/a/page-cache

[young ermine]

The next planned architecture support will be for ARM. Although this is not a priority at the moment, and will come much later in the future.
are we talking aarch64 or 32-bit arm?

[errant venture]

32 bit arm is absolute hell

[young ermine]

also, Entry::parent should be a Weak<Entry>, no?

you have Arc<Entry> which has a HashSet<Arc<Entry>> and each Entry has a parent: Option<Arc<Entry>>

you create cyclic references without using Weak

idk if that matters, but i just wanted to point that out

[late oar]

aarch64

[late oar]

you cannot remove an entry if it has at least one child (that's on purpose), and when you remove an entry from the HashSet, it drops the reference to the parent. So that's fine

I don't have Weak because that means I need to maintain a second counter even though I might not need it in a lot of places

[young ermine]

are you not using the alloc crate?

[late oar]

No because I want allocations to be fallible, so I have my own implementation of all the data structure which return an AllocResult<...> for functions that make memory allocations

[young ermine]

ahh, that's smart

that's one thing i absolutely hate about rust

it's so nice to be able to use alloc and all that

but wait.. it panics on alloc

too bad, time to implement everything yourself :^)

i bet there are crates that have fallible allocation compatible containers though

[late oar]

yeah. They justified that by saying that most people are not interested in dealing with allocation failures (which I agree with) but that makes it annoying for embedded

[late oar]

probably

[inner pilot]

not necessarily

technically you can make room for new allocations

[young ermine]

on alloc failure

[inner pilot]

or write to a pagefile

[young ermine]

yes u can recover

but like

not always

and what if no swap

and nothing to evict

no dirty pages you can writeback

no caches to flush

completely oom

you have to return null

and then alloc shits itself

[inner pilot]

does any mainstream kernel handle that?

[young ermine]

with you own implementation you can at least propagate that

[inner pilot]

i guess

[young ermine]

i guess at that point linux will start killing processes

isnt oom killer implemented in userspace tho?

[inner pilot]

dunno

probably

[young ermine]

it's messy but it's something you should probably handle

[inner pilot]

you could still have a kernel side one

[young ermine]

instead of assert!(phys_addr.is_some())

[inner pilot]

did i do that

💀

i find it funny because the allocator_api can fail

but that isn't allowed as a global allocator

[errant venture]

nope

its a kernel function

there's a userspace oomd that monitors psi counters

but its not reliable

[late oar]

I was thinking about OOM handling this morning and I was telling myself I should have a popup window on OOM which asks the user to select a process to kill.
And if that fails for some reason, then the kernel would select one itself

[errant venture]

yeah

linux calculates a score for all processes

and picks one with the worst score

[late oar]

yes

[errant venture]

that might even be the process that invoked the current allocation

so it will kill itself to satisfy its own allocation

[late oar]

or rather, it will kill itself because it cannot satisfy it

[errant venture]

no like

it literally will spin in __alloc_pages until it makes some progress killing itself

and alloc pages will succeed

[late oar]

[errant venture]

because it doesnt know whether its safer to abort the current allocation or let it succeed

it might be for some imporant kernel metadata keeping

[late oar]

makes sense

[errant venture]

so it just lets it finish

so basically actually returning NULL for kernel allocations is possible but very very unlikely

[late oar]

My allocator takes a flag which says whether it can fail on OOM or not

[errant venture]

thats basically only for high order allocations

[errant venture]

im curious which allocations u count as fatal

[late oar]

If it cannot fail, then it spins trying to shrink caches, and if that fails, I panic (I didn't implement process killing yet)

[late oar]

The first I think of is page table allocation because I don't want to handle failure here

[errant venture]

hmm really?

why not

[late oar]

failure means I have to rollback the modifications I've made and that's annoying

[errant venture]

u dont have to technically

if its a user page table

u can just reclaim it later

[late oar]

like try to find unused tables on OOM?

[errant venture]

no like, free the page tables when the process exits

and just SIGKILL it

[late oar]

but then you don't get the page back until the process dies

[errant venture]

yes but u literally sigkill it right there

[late oar]

what if that's a process that's running for a long time?

[errant venture]

so its better to panic the kernel instead?

[late oar]

having the kernel panic while playing Minecraft would be annoying

well idk

[errant venture]

well a proper solution would be to have an actual oom killer

and then it kills whatever is currently appropriate

or u could not overcommit memory

[inner pilot]

just kill the lowest pid

[errant venture]

Goodbye init

[inner pilot]

nobody needs that anyways

[inner pilot]

actually, isn't pid 0 the kernel?

[late oar]

in Maestro, pid 0 is the idle task (just a loop with sti, then hlt in it)

[inner pilot]

fair

i allocate pid 0 for the kernel

every kernel task is part of pid 0

[late oar]

I have a different pid for each kernel task

the flush task (writes dirty pages back to disk) is pid 2 in practice (I launch it after the init process)

I just realized I think this task can be killed from userspace and that's a problem

[inner pilot]

why not have it in the same process

all kernel tasks use the kernel page table after all

[late oar]

yeah but I can have several processes sharing the same memory space

I do that thing that linux does where a process and a thread are actually the same thing but with different IDs

[inner pilot]

that's a thing?

hm, my threads and process are different things, but i see what you mean

for me a process is a struct that contains a bunch of tid's

[teal dock]

that's pid -1 for me

[inner pilot]

signed pids

[teal dock]

(it's unsigned)

iirc I removed the line that set the pid for idle process

[errant venture]

tid and pid is the same namespace in linux

[inner pilot]

is there a reason why

[errant venture]

convenience

u can pass pid to functions that expect tid

[inner pilot]

ig

because imo a process carries different information than a thread

[errant venture]

a process has at least one thread

[inner pilot]

yes

[errant venture]

and the first thread's tid == pid

[inner pilot]

but that's not mandated by posix, right?

i hope not

[errant venture]

probably

(not)

[inner pilot]

also do you think just counting up ids is a good idea

on 64-bit it should be fine

[errant venture]

isnt pid a 32-bit value?

[inner pilot]

implementation defined

[errant venture]

then probably

[late oar]

on Maestro it's an unsigned 16 bit value

I guess I should increase it to 32 bit

for PID allocation I have a bitfield on which I iterate to find an unused one. Not the most efficient thing but I will into optimizing the day it becomes a problem

[inner pilot]

why 16

couldn't you just iterate all processes and look for the first free pid that you get?

[late oar]

idk

[late oar]

that's what I do

well I iterate a bitfield instead of processes so I guess not

[inner pilot]

ye

[late oar]

I guess I could do that yeah

my processes are in a binary tree. Maybe there's a more efficient way to find a hole than just iterating over the whole tree

also I still have a global lock over my process scheduler. That's gonna go away when I implement SMP

and all my locks are spinlocks. that's inefficient as hell

[inner pilot]

ah

well my processes are stored in a binary tree as well, but threads are stored per CPU

[late oar]

how should I implemented mutexes? like if the resource is not available after 100 spins, I put the process in sleeping state?

[late oar]

when I implement SMP I will do a scheduler per CPU. And then I'll need some mechanism to make a CPU move processes from others if it doesn't have enough work I guess

[inner pilot]

yea i'll have a rebalancer that runs on every thread spawn/kill

[late oar]

when a process state changes too? like when one goes to sleep or is stopped. you probably want to use the CPU for some other running process

[inner pilot]

depends on the priority

a lower priority running thread might have to give way to a higher priority sleeping thread

at least if it's blocked

[late oar]

if a process is blocked, I put it in sleeping state

and it's woken up when the thing it is waiting for becomes available

[inner pilot]

i differentiate between blocked and sleeping

[late oar]

what's the difference in your OS?

[inner pilot]

I'm not that far yet, but I'd like to make it clear that a sleeping thread is predictably halted

e.g. via usleep

i will also do per-task time slice scheduling that doesn't rely on a constant timer interrupt

[late oar]

Maestro running Minecraft when

[inner pilot]

true

you could try for minetest

[errant venture]

You will need a lot more syscalls for java

[shell tusk]

minecraf

[late oar]

this is due to those libraries being located in /usr/lib64 while the linker is looking /usr/lib. I read the LFS book earlier today and I remember it saying gcc (iirc) should be patched so that libraries are put in /usr/lib, so imma do that

[inner pilot]

you can just symlink that

or wait you have x32 stuff

nvm then

[late oar]

is it really relevant ? my archlinux has that symlink

[inner pilot]

that might cause issues?

[tawny drum]

on 64 bit systems /usr/lib and /usr/lib64 are symlinks to each other

[inner pilot]

arch linux has /usr/lib32/

[tawny drum]

and 32-bit libs are in /usr/lib32

yes

but that's not x32

i think if you have x32 stuff you'd have a /usr/libx32 dir but i am not sure

[inner pilot]

i think i meant x86_32

[tawny drum]

well x32 isn't IA-32

i am kinda surprised you didn't know that already tbh

mods, screencap this

[inner pilot]

i know that it is but idk what the right terms was

my gcc patch has 3 different lib dirs

uh oh

i think i fucked this up then

xd

it should probably be the other way around

[tawny drum]

uh

i mean

what is x86_32 supposed to be?

[inner pilot]

no idea

i don't even support that

but gcc wants it

[tawny drum]

well

you can just set it as /dev/null really

if you don't support it

[inner pilot]

oh

fair

[tawny drum]

it won't be used

it can be anything

IA-32 is i[3-6]86 is "x86_32"

x32 is an ABI variant of 64-bit x86 code where pointers are 32-bit wide

[inner pilot]

that's stupid

then yea i don't want x32 at all

[tawny drum]

no one uses it

[inner pilot]

what was the point

[tawny drum]

it's obsolete and dead

well

one sec

The x32 ABI is an application binary interface (ABI) and one of the interfaces of the Linux kernel. The x32 ABI provides 32-bit integers, long and pointers (ILP32) on Intel and AMD 64-bit hardware. The ABI allows programs to take advantage of the benefits of x86-64 instruction set (larger number of CPU registers, better floating-point performance, faster position-independent code, shared libraries, function parameters passed via registers, faster syscall instruction) while using 32-bit pointers and thus avoiding the overhead of 64-bit pointers.

from Wikipedia, the free encyclopedia

[inner pilot]

lmfao

[tawny drum]

as i said

it's dead and afaik it has never been used

like, at all

[inner pilot]

good to know

[tough onyx]

PID reuse is not something you want

[late oar]

okay but if you run out, you want to reuse those at some point

[tough onyx]

So you use a 64 bit number

[late oar]

on a side note, I now have g++ working

[tough onyx]

Also I don't see how that would ever happen as sleeping threads aren't even on queues

[spiral glade]

what if my system spawns 18,446,744,073,709,551,615 processes 😢

[tough onyx]

You will run out of memory before that

And storage for swap

[inner pilot]

also, even if you spawned a process every nanosecond you wouldn't be alive to see the day

[tough onyx]

Yeah true

[late oar]

and then you have the issue that pid_t is an int so you can't represent the PID if its higher than the max of a 32 bit value

[spiral glade]

yall dont know me tho

like I'm different

[inner pilot]

that's why you use a libc like mlibc where you can just define pid_t as a long

[late oar]

built different™

[spiral glade]

exactly, I'd live up to see all of my 18,446,744,073,709,551,615 processes be happily spawned =D

[late oar]

I just realized implementing memory swap should be really easy since my last refactor of memory management

might do that sometime this year

[late oar]

so, the next steps are:

• fix my flush task to make sure file metadata are written to disk
• make a blogpost about ELF loading, gcc and stuff
• fix the kernel crash when booting with more than 1 GB
• fix OOM handling

Then I don't know if I should try to port more stuff first, or implement SMP

[shell tusk]

Make the blog run in maestro

[late oar]

I don't have network support yet

coming by the end of the year if I have time

this year's goal is to be able to code on Maestro itself (preferably with git, so that would require network support)
next year's goal will likely be a desktop environment

[silk wren]

it's possible to do git without internet.

[late oar]

yes but lame

[silk wren]

you just distribute the repo via physical USB sticks

honestly way more fun than internet git

[late oar]

also coding in rust without internet is going to be annoying

[silk wren]

like you get a USB stick from the man himself, you know it's legit

[silk wren]

just download everything you need on a separate computer and load it via USB sticks

[silk wren]

imagine if Linux would be distributed by a network of people giving each other physical USBs with patches and tarballs and compiled kernels

[inner pilot]

why

just don't depend on so many crates

[late oar]

dependencies

toolchain update too

[late oar]

I think that would be very unsafe

[silk wren]

maybe

but certainly fun

[inner pilot]

when maestro meson#

[silk wren]

the kernel is supposed to be that thing below the random project thanklessly maintained by a single developer in nebraska since 2003.

[late oar]

lmao

[late oar]

meson is coded in python, right?

[inner pilot]

ye

[late oar]

I'd like to avoid relying on python if that's possible

[inner pilot]

why

[late oar]

although I will of course make sure it works

I don't like python

[inner pilot]

bruh

[late oar]

inefficient

and I don't like dynamic typing

[inner pilot]

but meson isn't python

smh

i'm going to try using meson instead of cargo

[silk wren]

🐍

[late oar]

yes but it's written in python, meaning I would need python to build a distro

[inner pilot]

you just need that on the host

[late oar]

bad

[inner pilot]

huh

[shy trail]

what??

tf

[inner pilot]

don't you need python to bootstrap rustc anyways

lol

x.py

[young ermine]

x.py

[shy trail]

lmao

[young ermine]

[silk wren]

the overhead from a build system written in Python in comparison with compiler optimisation passes is irrelevent, vanishingly small.

[inner pilot]

like

cargo is terribly slow for some reason

[late oar]

I hate rust

this project is over

I'm rewriting everything in C

[shy trail]

lol

[inner pilot]

👍

[young ermine]

thank god

first rust operating system rewritten in C

[late oar]

I'm starting to tell myself maybe lazy-freeing page tables might actually be a lot better since that means I don't have to maintain a counter on the number of used entries to know when to free it.
I can just let the page table there after an unmap, and it will likely get reused the next time the process does mmap anyways so I am saving a free and alloc.
Also if the system runs out of memory I can iterate on processes to look for unused page tables and free them before killing anyone

[silk wren]

you can also do something where, at munmap time, you look at where the neighbouring VMAs are and free all page tables that are in between them, that you know that no one needs.

[late oar]

‼️

smart

[silk wren]

not invented by me

but thanks anyways lol

you can look at davix mm/vmap.cc if you want an example of how this might look (however it might scare you away from wanting to do this)

[late oar]

I'll check, thanks

[late oar]

(just keeping this message around as a TODO list, don't mind me)

[late oar]

(turns out that's fake news, you can do it with a Discord bot)

thread image is back ‼️

[lunar plover]

How ??????????

[muted fox]

Nice logo

[late oar]

https://discord.com/developers/docs/resources/message#edit-message

you can just set flags to 0

[late oar]

thanks!

|| it took 6 months to make ||

[inner pilot]

@late oar did you know you can kinda do fallible allocations with ::alloc now?

with the allocator_api

[late oar]

no I did not

well, depends on what "kinda" means

I know there's some fallible functions on collections but there's almost nothing

[inner pilot]

ah

instead of e.g. Box::new you have to do Box::try_new_in

[late oar]

There's a fallible push on Vec I think but aside from that there's not much

[inner pilot]

for example i do this

and that automatically gets coerced into Errno::ENOMEM

[late oar]

btw I've tested this: https://www.a1k0n.net/2006/09/15/obfuscated-c-donut.html
and it's almost working (just missing support for some ansi escape codes I think)

[late oar]

These last few months (and all of last year), working on Maestro have been pretty exhausting and kind of annoying (I really had to force myself to work on it regularly)
But now that I have done the annoying bug fixing and refactoring part, it is getting a lot easier to work on it. I am implementing new stuff instead of fixing the old ones and I am starting to get fun working on this again 🎉

yesterday evening I have tested some small cpp programs made by a friend and they are all working as expected 👌
https://github.com/AntoineJT/cpp-things

[late oar]

I should record this btw

[late oar]

now with colours™

that was already implemented but buggy

[late oar]

this video will be in my next blogpost

[plucky fiber]

...a program like a donut showing a donut...

[errant venture]

very cool

what file system is it using

[late oar]

ext2

[errant venture]

nice

[late oar]

comes from here: https://www.a1k0n.net/2006/09/15/obfuscated-c-donut.html

[plucky fiber]

ext2 great choice

[late oar]

I haven't looked at ext3 and ext4 yet but I guess it's just ext2 with more features?

[solid rose]

Basically, ext3 adds journaling which is probably the most complicated part of those two filesystems

[plucky fiber]

obfuscated c yumyum

[late oar]

people, the time has come to start implementing SMP

[coral wing]

C mentioned, lets gooo

[late oar]

enumerating CPU cores on the system with ACPI: ✅

[solid rose]

hell yeah

[errant venture]

Did you roll your own table scan code?

[late oar]

yes

[late oar]

I now have a rwlock instead of a simple spinlock around the list of processes

[late oar]

currently trying to setup the APIC

[young ermine]

the what now?

what tf have you been using previously

the pic???

[solid rose]

seems reasonable enough for a single core system

[late oar]

Yes

[inner pilot]

skull

[late oar]

I think I'll open Maestro to external contributors. I guess I need to write contributor guidelines

[tulip ember]

Wow, Gg fantastic work !

May i ask you if you started with https://os.phil-opp.com/ as a base ?