Maestro - Linux-compatible-ish OS in Rust

[fast halo]

"a user program" could also talk about the programs you and i use (coreutils, binutils, make etc)

[inner pilot]

i'm confuzzled

init is also a user program

in my design init starts something like kmscon or gcon as the tty, then starts a shell

[fast halo]

ah yes

okay so my design is similar except the terminal server launches the shell instead

so the shell inherits the IO handles it needs from the terminal server

[inner pilot]

makes sense

i do give a console to init, but it's not a bidirectional terminal

[fast halo]

when I have a GUI, then the GUI will launch a console by launching a terminal application which will then launch the actual app

[inner pilot]

it's just output logs via the kernel logging facilities

[fast halo]

that makes sense

i think i might do that too

but just for init

[inner pilot]

yep

this is what it looks like at the moment

(ignore the panic)

once init starts the actual console, the kernel will give up the framebuffer

[late oar]

I like the idea of having a userspace program for that, but what if it dies? OOM for example?

[inner pilot]

so what

if you kernel ooms then what xd

[late oar]

No terminal anymore

Gets killed by oom killer

[inner pilot]

ideally the terminal allocates one buffer and then never needs to do it again

[late oar]

Yeah but the oom killer does not necessarily kill the process that is requesting the allocation

[inner pilot]

just give it a high prio

then when looking for processes to kill, go bottom up

[late oar]

okay so now I managed to build everything, except make

klfsdjkljksdlklsdfklfsdklklfds

[shell tusk]

what is preventing it from building

[late oar]

I use this script to build it:

#!/bin/bash

set -e
cd *

./configure --prefix=/usr \
        --host=$HOST \
        --build=$BUILD
make -j${JOBS}

make DESTDIR=$SYSROOT install

[shell tusk]

seems like make is defining it sown getenv?

I think I ran into this before

fnmatch.c:124:14: error: conflicting types for 'getenv'; have 'char *(void)'
  124 | extern char *getenv ();
      |              ^~~~~~

[late oar]

‼️

[shell tusk]

there is nothing in my patch about it, let me look at the recipe and if nothing there then the soruce

# if !defined _LIBC && !defined getenv
extern char *getenv ();
# endif

you have two options here, figure out why its not working or just patching that out

in lib/fnmatch.c

probably unsound build environment or something?

[late oar]

that sounds like something that could happen

[elder sonnet]

or maybe gnulib bs with musl

or musl memes in general

come join us at mlibc

[shell tusk]

if you doing linux compat just go all the wayt and use glibc

[late oar]

no glibc bad

musl lightweight, musl good

[elder sonnet]

mlibc better

join us

[late oar]

one of us

[late oar]

it seems alpine linux made a patch for this issue precisely
https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/make/getopt-gcc15.patch

[late oar]

‼️

[inner pilot]

compile linux on maestro

[late oar]

I'll try to build gcc first

btw gzip seems to make the kernel do an infinite loop

lmao

or maybe the executable is just really long to load and I am being impatient? In any case I think I should normally be able to Ctrl+C the process during execve?

no the file isn't big. there's definitely an issue somewhere

diff does not work
file does not work
find seems to partially work?
gawk seems to work?
grep does not work
gzip does not work
m4 does not work
idk how to test ncurses
I didn't test patch yet
sed does not work
I didn't test tar yet
xz does not work

wonderful

grep seems to use the syscall alarm which I did not implement

[shell tusk]

Do you not mmap your files

[late oar]

I do

[shell tusk]

Then it should not take that long

[late oar]

yeah there's an issue somewhere

Also I have to figure out why a process exits with code 0 when killed by a SIGSYS

okay I think I never set the exit status code except when calling exit. Let's fix that

[solid rose]

task failed successfully

[late oar]

indeed

[late oar]

I'm stupid. It's actually mincore who's missing

[errant venture]

why the hell does it need mincore

[late oar]

no idea. I'm just implementing it right now since I'll need to do it at some point anyways

[errant venture]

I think it might be so it only scans blocks which are actually present

To avoid doing no op reads of just zeroes

Basically sparse file handling

[late oar]

when I type grep --help, it scans a big portion of the address space for some reason, which is very long

it does that because /proc/self/maps does not exist (idk yet why it needs it)

I guess I'll implement /proc/self/maps 🤷

sed needs fchown apparently. I didn't implement it

[late oar]

I have implemented fchown and fchownat. Now it's rename who's broken apparently

[late oar]

I am debloating my system calls handling. I have 492 compilation errors to fix

[solid rose]

peak

[late oar]

peak osdev moment

[late oar]

New blog article (@shell tusk and @silk wren mentioned at the end)
https://blog.lenot.re/a/smp

[errant venture]

no aml support

[late oar]

yet

[silent lava]

damn

a whole ass explanation even

[shell tusk]

Oomfie famous

For a second those circles on the screen made me think I broke my phone screen gjisnfisncos

[tough onyx]

This is pretty good, but note that AML IS bytecode

ah ok you just meant it's hard to parse

[late oar]

Yes

[late oar]

lmao

[silk wren]

in your explanation of critical sections, you state that the counter has the following layout:

• bit 31: set by default. clear means preempt.
• bits 30..0: the number of nested critical sections.
  It is worth clarifying that Linux uses some other bits of the "preempt_disable counter" for additional state like in_irq, in_nmi, etc. Otherwise LGTM

[late oar]

I'll do that, thanks!

[late oar]

I'm back to work on Maestro. I have to implement newfstatat for the find command, so I guess I'll make sure all of the variants of it are implemented

[late oar]

find works (I haven't tested every corner of it though)

[late oar]

The file command is missing pread64. Let's implement it and do pwrite64 at the same time

[late oar]

file is working

[late oar]

I like when I implement stuff and I end up with less lines of code than before

In my current PR I have added 10 syscalls and I ended up removing 342 lines in total

[late oar]

I am currently implementing /proc/<pid>/maps

[late oar]

I have a deadlock inside of it and idk how to fix it
I lock the maps list to iterate onto it, and I write a line to userspace memory for each.
But, if the userspace page we are writing to isn't allocated yet, it triggers a page fault, which in turns tries to allocate the page, but that requires locking the maps list, which is already locked

[late oar]

I have a temporary fix that I don't like (swap a spinlock for a rwlock and add another spinlock somewhere else). I'll have to change it later

Now grep seems to want sigaltstack

[silk wren]

reader-writer mutex, read locks can nest?

[late oar]

that's the solution I went for and that I don't like

it feels cumbersome to me

[silk wren]

oh you use a preempt disabling rwspinlock for that instead?

[late oar]

I don't disable preemption, only interrupts (I don't even remember why)

[silk wren]

I fail to see how that would work with waiting for IO etc., because if you schedule away to block when holding the maps lock, and someone else tries grabbing the maps lock, now you can never switch back to the original task and you are deadlocked.

[late oar]

indeed, good point

[silk wren]

the maps lock really should be a sleepable mutex (spinlocks rwspinlocks and the like aren't sleepable because of this specific issue)

[late oar]

I'm noting that somewhere. I need to rework my sleepable mutexes first

They currently need a memory allocation to enqueue the sleeping tasks and that's really bad

[silk wren]

why do they allocate?

you can have a preallocated node either on the stack (if you don't swap kernel stacks) or in the thread structure.

[late oar]

[late oar]

it has a Vec of PIDs of tasks waiting on it. An insertion in a vector might require a memory allocation

that's a bad design. I didn't know what I was doing at the time

[silk wren]

then it should be as simple as replacing that Vec<PID> (why PID? why not Thread * or something which makes more sense in-kernel?) with an intrusive linked list which doesn't allocate.

[late oar]

PID because I didn't want to have unsafe code

but I'll do a stack or thread thing. That's been on my todolist for a few weeks

well I could even do it now, once I've committed my sigaltstack impl

[late oar]

If I decide to do it on the stack, how do I make sure the task is removed from the queue if it turns into a zombie?

[silk wren]

it's easy: disallow turning tasks into zombies except when running in that task's context, and only turn yourself into a zombie when you are not holding any shared resource.

if T1 wants to kill T2, T1 has to enqueue a function to run in the context of T2. this can be done e.g. via signals.

[late oar]

thanks!

[silent lava]

How do you do your (unit) tests, if any? I'd already asked Marvin since he's also writing in Rust but he doesn't have any tests.

[late oar]

Yes I do, and I use the thing mentioned here: #1287456798407790684 message

[silent lava]

I've read that but I'm tempted to make my own framework as I'd like to make tests run at specific levels of running (e.g. page table test would run between PMM and VMM init)

[late oar]

I just run them once virtual memory is setup

[balmy plume]

If i wanted to do kernel self tests, I'd just run them instead of init

Just finish the kernel boot, then run tests and terminate again

[silent lava]

Doesn't get there because page tables are broken. You know, something you could more easily fix if you had tests.

[balmy plume]

If you don't even get to the point where you can run tests, you don't need tests to find failures

The point of tests is to prevent regressions, not to debug broken code

[silent lava]

I would argue that testing small subcomponents of the VMM (e.g. converting generic PTE format to what the CPU actually wants and back) would make it easier to narrow down what could be broken. In addition, this is literally a regression because VMM worked before the refactor (it just didn't have all the features I'll need).

[balmy plume]

or rather: while some test failures can hint at what's wrong, that's not their primary purpose

tbh i don't think the overlap of "failure before the kernel even finishes booting" && "tests help with debugging it" && "can be tested inside the kernel in a meaningful way" && "can't be tested outside the kernel" is big enough to justify the cost

[silent lava]

Page tables can be tested outside the kernel, you just can't actually use them like that.

[balmy plume]

If the kernel doesn't finish booting because PTs are broken, something must be seriously wrong and that failure is easy to find and fix

[late oar]

I have this issue where my kernel does not work when built on one specific computer and I have no idea why it does that
#osdev-misc-1 message

[silent lava]

Turns out this wasn't page tables being broken - something testcases would've told me earlier.

[balmy plume]

I'm not claiming that tests won't help, so I'm not sure what you're trying to say

[silent lava]

It sounded like it

[balmy plume]

I claimed that the cost of having self tests running before boot is done outweighs the benefit

I'm not saying that tests are useless, I'm saying they should run on top of an already working kernel (even if they do run in kernel space)

[late oar]

Maestro running Minecraft when

Dead project

[inner pilot]

be the change you want to see

[late oar]

I have to finish implementing sleeping mutexes and condvar

[worn cosmos]

maybe writing a blog article will motivate you? 🥺

[late oar]

I don't have anything to say

I think I don't lack motivation. I lack time

well a bit of motivation too during the past few days but it's fine now (I was kinda depressed)

[shell tusk]

I remember you said you were running into some trouble, what were you running into?

when making sleeping mutexes

[late oar]

I don't have sleeping reasons and I didn't feel like adding those

I noticed I had timers that could wakeup a thread if setup with SIGEV_NONE, and I realized today I don't even need to wake it up I think? I can just do nothing. So I guess I don't need sleeping reasons anymore

[shell tusk]

the only wake up reason I think I even have in astral is interrupted which is for signal stuff

other than the normal wake up reason

[late oar]

I felt my mutex implementation could not work because if a thread got woke up by something else than the mutex, it could acquire the mutex even though it's not its turn

I'll try to make it work this evening probably

[shell tusk]

you can always just have specific sleeps not be interruptible. are there any cases in maestro where you would even want to be woken up by waiting on a mutex that you are not explicitly saying that its an interruptible sleep?

my mutexes dont support that

there is no reason to, if they want to wait on a resource interruptibly they use another primitive

[late oar]

so that's the reason why Linux has two kinds of sleep states? so that it may get woken up by a signal when its an interruptible sleep?

[shell tusk]

I am not really familiar with linux design but it could be

I pretty much do the same in astral, theres a distinction between normal sleep and interruptible sleep

[late oar]

they have interruptible sleeps and non-interruptible ones

[silk wren]

Linux has quite a few states. There's TASK_INTERRUPTIBLE (the best kind of sleep), TASK_KILLABLE (like TASK_INTERRUPTIBLE but can only be interrupted by SIGTERM or SIGKILL iirc), TASK_UNINTERRUPTIBLE.

actually IIRC TASK_KILLABLE might actually be named __TASK_KILLABLE in typical Linux manner and be a flag ORed with TASK_UNINTERRUPTIBLE. I'm unsure.

anyhow, the idea with TASK_KILLABLE is that e.g. in page fault handler path, if you are killed by someone, you no longer need to satisfy the page fault, because the faulting access is effectively aborted.

a side note, wake-up reasons need not be propagated to the task which slept (AFAIK Linux doesn't, for example). any task which wakes up can check for itself if the resource it waited for is available, otherwise look at pending signals/timeout.

TASK_UNINTERRUPTIBLE sleeps and big in-kernel loops that don't check pending signals anywhere are what translate into processes which have a long delay between you pressing C-c and the process exiting.

[late oar]

thanks! @silk wren

[late oar]

okay I remember why I did that. I use my timer implementation in nanosleep, with SIGEV_NONE, and so I added a wakeup so that the sleeping thread can get woken up when the timer fires, without having to use a signal

it seems Linux uses restart_blocks for that. Now I have to figure out how they work

https://elixir.bootlin.com/linux/v6.13.4/source/kernel/time/hrtimer.c#L2219
https://elixir.bootlin.com/linux/v6.13.4/source/include/linux/restart_block.h#L39

my timer code is so shitty, I'll have to rewrite it

the timers queue is a binary tree and to move an element to the back I have to free it and re-allocate it to re-insert it

[late oar]

okay that's for restart_syscall that I did not implement yet

[late oar]

so, I am starting to fix my timers so that it does not interfere with sleeping mutexes

okay I cannot finish this refactor now because I'm going to need intrusive binary trees to do it correctly. And honestly I don't feel like implementing those

[young ermine]

funny thing, that's exactly what i'm working on right now :^) (timers)

and, thankfully, i use this funny crate called intrusive_collections

[late oar]

I am psychologicaly unable to add a dependency to this project

my hands just won't do it

[nimble grove]

You depend on a compiler

And emulator

[late oar]

well yeah of course

[nimble grove]

And architecture

And other stuff

So why draw the line at a crate

[silk wren]

when do you need to remove and re-insert an element in the tree? the only case that I can think of is if you change the deadline of a timer, but when would you ever do this?

[late oar]

when the timer expires, I execute the associated action, and then I move it in the the queue for the next time. My queue is ordered by the timestamp of the next time the timer expires

[silk wren]

oh I just have timers that are one-shot. If you want a repeat behavior, then you have to enqueue it yourself in the handler.

[late oar]

yeah, makes sense. But the issue I have is that enqeuing requies a memory allocation because I don't have intrusive binary trees (yet)

[late oar]

Yesterday I've been trying to fix signals. Since Ctrl + Z doesn't work anymore for some reason

I have changed the way processes state work a bit. Now a process cannot transition from Sleeping or Stopped state to Zombie. It has to go back to Running first

Idk if it's a good thing wrt SIGKILL though?

[solid rose]

Maybe an off-by-one issue?

[late oar]

What's that?

[solid rose]

https://en.wikipedia.org/wiki/Off-by-one_error

but now that i think of it it wouldn't really make sense for it to be that

[late oar]

nah I don't think that's the issue

[solid rose]

at least you have proper signals unlike me

[late oar]

I've made some fixes yesterday and now the issue I have is that when I do Ctrl + Z, the process stops, but then resume immediately for some reason

[late oar]

Turns out it's non-trivial to implement

also I've realized my wait* syscalls aren't implement correctly I think. wstatus shoud probably be a field in the process's structure, and wait* should retrieve it from the child process's structure?

[late oar]

alright guys. After trying to refactor my signals and process state handling, turns out need the TTY not to do deadlocks to make it easier to debug (how queer????) so I am rewriting the TTY right now

I am getting fed up by this crap

[late oar]

okay I don't have a deadlock in the TTY anymore, now back to signals and process states

[shell tusk]

Signals are sucky

[late oar]

Yes

Fuck em

[late oar]

okay guys. I have fixed signals. I can now use Ctrl + Z to pause a process and go back to bash, and I can use the fg command to resume it

It used to work a few months/years ago but I broke it and I had never fixed it until now

[inner pilot]

how to signal

[worn cosmos]

you should remove signals from maestro as an act of protest to the unix overlords

[errant venture]

That would be funny in a Linux compatible system

[late oar]

lmao

Are there cases where, for a given mutex, I would sometimes want to lock them so that they can be interrupted by a signal, and some times not?

It has to take a boolean to know if it can be woken up by a signal, but I want to know if this boolean should be a function argument or a constant generic

Next blog article's title: UNIX signals are a pile of garbage

(kidding)

[solid rose]

lol

[late oar]

I am looking at intel virtualization instructions. It looks funny to implement

I kinda want to do KVM now

idk how similar AMD virt instructions are to Intel though

[late oar]

dead OS

ain't got time to work on it

maybe a bit this evening but unsure

[errant pasture]

this is a very cool project, if it's any consolation

I might "steal" some of your code for my own microkernel project if that's acceptable

I would consider filing PRs but like I said I have my own kernel I'm working on

[late oar]

Yes, go ahead

[late oar]

I think sleeping mutexes work, but I have reference counting issues. The process structure is never freed

[late oar]

still stuck on the reference counting issue

something something reference not dropped when removing an element from an intrusive linked list, implemented here:
https://github.com/maestro-os/maestro/blob/stabilise/utils/src/collections/list.rs

I use it in the process's structure so that I can have a queue of processes waiting on a mutex

the reference counter of the process is incremented when I insert it in the list, and is supposed to be decremented when I remove it

[late oar]

okay guys, I think I've got something
I have added a unit test in userspace on the linked list, on the function that removes an element and I get a segfault

omw to use valgrind for the first time in years

[late oar]

okay guys, it was fucking stoopid

To decrement the reference counter on the Arc, I was doing drop(Arc::from_raw(val)), where val is an &Arc<...>
So I was basically building a Arc<Arc<...>> from a raw pointer, then decrementing a random location on the stack when it was dropped

I've lost a week on this 👍

[late oar]

omw to support gzip

currently the kernel freezes when I run it. It seems to do a page fault

alright I've been mixing up RIP and CR2 for the last 20 minutes or so

[late oar]

btw I am pretty sure it's because of the ELF loader trying to zero mapped pages after the file's end. Resulting in a SIGBUS, which is not caught, so it fails in a loop

[late oar]

alive OS

[latent anchor]

take a screenshot and label it AI

[late oar]

I have a function named oom::wrap which retries a given closure until it passes without an allocation error, calling the OOM killer (which I didn't implement yet, so this function is useless) at each failure, in order to avoid having to handle memory allocation failures in some places. I absolutely hate this function
Turns out the few places left where I use it are all places where I should be using intrusive linked lists instead. And now that I have those, I should probably get rid of that shitty function

[late oar]

gzip almost works, there's just an error message when I use gzip -d ... saying the file at ... does not exist, but the decompression worked anyways 🤷

time to print syscalls to see which one failed

[late oar]

fixed. It was an issue when creating a directory entry in my ext2 implementation. When splitting a free entry to create the new used entry, I wanted to fill the remaining space with a new free entry, but instead I was filling the whole block, making the next entries disappear

tar seems to work

alright guys, I think it may be time to attempt building binutils ‼️

[shell tusk]

as in, make binutils work for maestro or build binutils in maestro

[late oar]

binutils already works

I want to build it on maestro

[shell tusk]

oooooooooo

hype

[late oar]

(nevermind, unpacking does not work. A syscall seems to be missing)

[shell tusk]

inb4 the syscall to set the create date

[late oar]

very likely

[late oar]

it's mkdirat

okay, what about I implement all the missing *at system calls so that I don't have this issue anymore? cause I already have the original (non-at) version for almost all of them

imma do that tomorrow though I think. Time to sleep

[shell tusk]

I implement like if not most then all the non *at as calls to *at syscalls :P

[late oar]

yeah, linux does that too

I didn't do it at the beginning because *at syscalls scared me at the time

now when I implement the *at version, I remove the previous syscall's impl and replace it with a call to the *at

[shell tusk]

yeah good work

sleep well

[late oar]

thx

[late oar]

I have implemented mkdirat, mknodat, renameat and readlinkat during my lunch break. I'll implement execveat and probably openat2 when I finish my workday

I guess I'll also do futimesat and its non-at counterparts

[shell tusk]

The grind never stops

[late oar]

just realized I've implemented sleeping mutexes but I use it almost nowhere. I haven't replaced spinlocks yet

[late oar]

indeed

[shell tusk]

Damn you should

Spinlocks everywhere is not good

[late oar]

yeah

shall make this OS BLAZING FAST 🚀🚀🚀

[shell tusk]

🦀 BLAZING 🔥 FAST 🚀

[late oar]

I use spinlocks on directory entries, but I guess I'll have to replace them by RCU eventually

[inner pilot]

does rcu even work with safe rust

LRU too

[late oar]

I think so

[late oar]

depends on whether you use intrusive linked lists or not

[silent lava]

It's mostly the preemption inhibition to worry about

But that should be easy with a guard type

[late oar]

yeah

[inner pilot]

i need to speed my shit up as well

also maybe RR is not the best scheduling algo

i do have sleeping mutexes, but uhh

I don't use them anywhere

[late oar]

it's pretty common for me to just implement/refactor stuff and the OS becomes faster without even knowing why

[silent lava]

I use sleeping mutexes by default

[inner pilot]

well i cant do that everywhere

e.g. in the scheduler

i think

[silent lava]

Of course

There you use spinlocks

I also use spinlocks if it needs to be acquired from an ISR or with interrupts disabled

Unfortunately, acquiring a spinlock is unsafe because disabling interrupts is unsafe.

[late oar]

why is disabling interrupts unsafe?

[silent lava]

Because Rust has assumptions broadly incompatible with it

For example, memory allocations are not ISR-safe

I may look into making a custom linter to guard against this so I can mark the interrupt guard as safe and have the linter check for invalid handling of interrupts.

[late oar]

I don't see the issue with memory allocations

[inner pilot]

huh

disabling interrupts isn't unsafe (except for the asm

enabling them is

[silent lava]

My allocator is not reentrant

[late oar]

mine is because it's wrapped in a global spinlock

[silent lava]

No that's literally what I mean, that is not reentrant

[late oar]

a global spinlock which disables interrupts

[silent lava]

It would be reentrant if you could get interrupted in the middle and have the same CPU run the allocator again in the ISR

However I use a spinlock on the page allocator

[late oar]

right

[silent lava]

And so it is not reentrant

My idea for the linter would see something like #[irqlint::isrsafe] and #[irqlint::isr]

So it could properly infer what is isr-safe code from the attributes

I would make a crate of it, perhaps the two of you could also find such a thing useful

[late oar]

maybe

[late oar]

okay openat2 is too complicated for me to do it today. It takes a structure with plenty of options in it

[balmy plume]

https://docs.rs/crossbeam-epoch/latest/crossbeam_epoch/ has an RCU impl

but it has an unsafe API

https://docs.rs/sdd/latest/sdd/ has a safe API

[late oar]

I've just noticed the fix makes the CI fail. sigh

tar xvf binutils-2.45.tar.gz seems to be working but very slowly

it seems to be going back and forth between the gzip process and tar process. I'll check if adding sleeping mutexes everywhere does any good

[inner pilot]

sadly not no_std

[late oar]

I think it has significantly improved performance

although it's still pretty slow

I think tar is parallelizing stuff. Let's try having 8 CPU cores (no it does not)

I've built the kernel in release mode and it's MUCH faster

unsurprisingly, I have a lot of time stamp ... is ... s in the future

and tar fails with: tar: binutils-2.45/bfd/po/fi.po: Cannot writeBus error

[shell tusk]

bus error, spooky

[late oar]

first hypothesis: shitty ELF loading

make also does a Bus error when I run it in the incompletely extracted binutils directory

nevermind, it does it regardless of the directory

grep too smh

[shell tusk]

I think your bus has errors

[solid rose]

have you tried checking for any broken solder joints on your bus?

[late oar]

actually I have rebooted and now there's no Bus error anymore

[late oar]

QEMU's famous solder joints

[late oar]

okay, having a pipe capacity of more than 512 bytes makes it faster

and now there's a deadlock

[solid rose]

512 bytes is tiny

[shell tusk]

wtf

512

thats like

smaller than pipe_buf

on linux

[late oar]

indeed. I mixed up pipe_buf with the pipe capacity when I implemented this

I don't have the guaranteed minimum atomic transfer size on pipes, idk how to implement it btw

[shell tusk]

I have it implemented in astral, its a bit messy but it works

its kind of just, if you want to write <= pipe_buf you can sleep until theres space, otherwise you can write it in parts

its not even a linux only thing, its a posix thing

https://pubs.opengroup.org/onlinepubs/009604499/functions/write.html

Write requests to a pipe or FIFO shall be handled in the same way as a regular file with the following exceptions:

    There is no file offset associated with a pipe, hence each write request shall append to the end of the pipe.

    Write requests of {PIPE_BUF} bytes or less shall not be interleaved with data from other processes doing writes on the same pipe. Writes of greater than {PIPE_BUF} bytes may have data interleaved, on arbitrary boundaries, with writes by other processes, whether or not the O_NONBLOCK flag of the file status flags is set.

    If the O_NONBLOCK flag is clear, a write request may cause the thread to block, but on normal completion it shall return nbyte.

    If the O_NONBLOCK flag is set, write() requests shall be handled differently, in the following ways:

        The write() function shall not block the thread.

        A write request for {PIPE_BUF} or fewer bytes shall have the following effect: if there is sufficient space available in the pipe, write() shall transfer all the data and return the number of bytes requested. Otherwise, write() shall transfer no data and return -1 with errno set to [EAGAIN].

        A write request for more than {PIPE_BUF} bytes shall cause one of the following:

            When at least one byte can be written, transfer what it can and return the number of bytes written. When all data previously written to the pipe is read, it shall transfer at least {PIPE_BUF} bytes.

            When no data can be written, transfer no data, and return -1 with errno set to [EAGAIN].

[late oar]

thx

[late oar]

I won't find the cause of my deadlock tonight. Altough I've made some good progress today. 8 system calls implemented

gn everyone

[late oar]

didn't do shit today because I felt like attempting to implement the Raft algorithm

https://en.wikipedia.org/wiki/Raft_(algorithm)

[late oar]

looks like I've fixed my deadlock? I just replaced a spinlock that blocks interrupts by a sleeping mutex

[late oar]

okay nvm, it's not fixed

[late oar]

I am moving the spinlock inside of the structure handling the page directory (VMem) so that I don't have to lock it when I just want to bind it. That should prevent some deadlocks

mood:

[strace 5] openat(-100, 0x7fffff7fd800 = "binutils-2.45/bfd/elfnn-loongarch.c", 559553, 384)

[late oar]

alright guys, I've fixed another deadlock, I am trying again to unpack that binutils tar

this is so slow

[late oar]

I think I'll go fixup https://github.com/maestro-os/kern-profile and then I'll try to figure out why it is so slow

QEMU seems to be using only around 30% of its CPU cores

[late oar]

https://github.com/maestro-os/kern-profile/pull/4

[late oar]

my TTY's history is contained in a linear buffer. When I want to write a new line, I do memcpy to shift all of the content up one line (erasing the oldest line)
Turns out, when I use the strace feature of my kernel, it spends ~40% of its time in that precise memcpy

omw to make a ring buffer instead

[late oar]

another thing my kernel spends massive amounts of time on is mapping and unmapping ranges on memory in page directories.
Notably, when I unmap stuff, I use a loop to check if a table has no more entry left, to know if I can free it
Instead, I should not attempt to free it, and reclaim it only when the system is low in memory (or when the page directory is destroyed)

also my function to map a range is just a loop which calls the function to map a single page (horrible)

[deft sonnet]

Is there a better way of doing it?

I’d assume you could make some function to map more pages than just one at once

[late oar]

indeed. when you are going down the tables, you can do all of them at once instead of re-doing the recursion for each page

[deft sonnet]

That makes sense

Yeah my map range is

Bullshit

int vmm_unmap_range(vmm_region_t *region, uintptr_t va, size_t pages) {
    for (size_t i = 0; i < pages; i++) {
        if (vmm_unmap(region, va + i * PAGE_SIZE) < 0)
            return -1;
    }
    return 0;
}

[late oar]

well yeah, I have the same

[deft sonnet]

Same concept for map_range()

[late oar]

I may re-do this better once I've finished refactoring my TTY

[deft sonnet]

[late oar]

lmao

[deft sonnet]

Would you do this for a direct mapping too?

Or just anonymous

[late oar]

my kernel spends 89% of its time in between grub and launching the first process in map_range

[silent lava]

That is quite a lot

[late oar]

I think so

indeed

[deft sonnet]

Damn 😭😭

[silent lava]

What are you doin ova der

How did you make it this slow

[late oar]

recursions that could be avoided

[deft sonnet]

sudo rm -fr —no-preserve-root /

Remove French language pack

[late oar]

and still the kernel boots pretty fast. roughly 0.3 seconds on my computer, on qemu, when built in release mode

[deft sonnet]

So how many times is map_range() called

[late oar]

I think it's not called that much, but it's being called on large ranges

[deft sonnet]

Makes sense

[deft sonnet]

Looking at mine now, about 60% of my kernel boot time is mapping ranges too 😭

Unfortunately I suck shit at programming so it takes about a whole second to boot my kernel

[late oar]

now you know how to make it faster

[deft sonnet]

Yessir

[late oar]

it used to take like 15 seconds to boot mine 2 years ago

[deft sonnet]

Dang. Yeah it takes me about 1 second until I jump into userspace

What would even make it take that long

[deft sonnet]

Mine took this long when I would map and write a to the entire page directory on boot

[late oar]

oh yeah I currently do that. I map all of the physical memory that can be allocated

maybe I shouldn't?

[deft sonnet]

It’s not necessary as far as I’m concerned

You can do it lazily

[late oar]

like, if I don't map it, then I have to do it when I attempt to access it. But if I map it when accessing it, I need to allocate memory for the page tables to map it, when needs to already be mapped in order to be used

[deft sonnet]

I think a solution to this is to pool memory within your vmm for those allocations

[tough onyx]

The real way to make it faster is to use larger pages

[late oar]

Oh yeah, that too

[tough onyx]

I don't think going through the pagetables is that expensive

It's like 4 memory accesses

[late oar]

I don't use large pages because I haven't figured a way to tell in my API which level I want to be a large page

[tough onyx]

I have a flag for that

I have what I call large pages and huge pages

Huge are the largest supported by the architecture

and large is the second largest

So like on a machine with no support for 1 gib pages then large and huge will refer to the same thing

But on a machine with 1 gib pages then large = 2mib and huge is 1 gib

Maybe I should rename them tho I'm not sure to what

Maybe large and largest, or level 2 and 3

[inner pilot]

I used to pass a "level" argument to the mmap family

enum VmLevel {
    Small,
    Medium,
    Large
}

the actual values are defined per arch

[silent lava]

My impl does hugepages implicitly

Just so long as the mapping is sufficiently aligned

[late oar]

how does that work when using the flags MAP_HUGE_2MB or MAP_HUGE_1GB on mmap?

[silent lava]

Well I don't have mmap yet but I'd simply do those by trying to alloc a 2M or 1G block from my buddy alloc. If successful, those will be sufficiently aligned and implicitly create hugepage mappings in the page table code.

[late oar]

alright guys, fuck my TTY's code, I am rewriting big chunks of it

[late oar]

alright. I've refactored my TTY and now the kernels spends 98.4% of its time in map_range during boot

[silent lava]

XD

[late oar]

(turns out the TTY now panics sometimes, investigating)

also the code is dirty. I have to clean it before I commit that

[late oar]

when I run tar xvf ..., QEMU is using only 0.5% of my computer's CPU. I think there's a problem

[deft sonnet]

Hmmmm, add a break point at that spot and check regs, check if the stack pointer and instruction pointer are different. I’ve noticed sometimes when the cpu somehow executes the stack it does that

[late oar]

I think the kernel's stack is mapped with no-exec

[deft sonnet]

Oh then hmmm

I would recommend placing a log at every context switch and see what happens

[late oar]

fortunately, I have a feature for that

[deft sonnet]

Ok lmk what happens

I’ve been experiencing a similar issue actually

That’s why I wanna know what happens for you

[late oar]

first I review my TTY refactor's modifications, then I am investigating that

[shell tusk]

Considering tar is i/o heavy perhaps it is just unoptimized disk drivers?

[late oar]

that's likely

[deft sonnet]

Yeah I was thinking that maybe ur nvme driver is inefficient somewhere

[late oar]

I don't have nvme, only pata

[deft sonnet]

Yeah that’s a whole can of worms

I’m working on nvme rn as a side project

One of the (I shit you not) 103 things in my OS todo list

[late oar]

I've heard nvme is simple. maybe I should do it. That would also make the OS a lot faster

[shell tusk]

It is simple

[late oar]

my pata driver doesn't even support interrupts. It only does polling

[deft sonnet]

Yeah the spec is super light

A decent OSdev Probably could implement it in one night

[shell tusk]

Especially since there is a lot less legacy garbage involved

[late oar]

that means it will take me 2 months

[shell tusk]

Then I dont think the low cpu use is a slow disk driver?

Since the cpu would be spinning waiting on it

[deft sonnet]

You’re above a decent OSdev bruh.

It would take MY ASS 2 months

[ancient galleon]

could be allocations too

at least for mine improving my pmm sped it up by like 20x

[shell tusk]

rfc3514 said that the issue was qemu not using the cpu so the cpu wouldve been halted most of the time

Could also be locking issues if they are using sleeping mutexes

Cant really say without them figuring it out

[late oar]

I suspect it might be the pipe implementation

since there is a pipe in between tar and gzip to uncompress before untarring

wait, maybe I could first unzip, and then try to untar, so that I am not using the pipe. That would allow me to test that hypthesis

okay it makes it faster, but not by much

maybe that means I should attempt to implement nvme and see if that's better?

[late oar]

I still have no idea why it's slow, but I've tried launching ./configure --help on binutils and...

[late oar]

I have rebooted and it does not seem to panic anymore. Likely a concurrency issue I guess? Or memory corruption

I have quite a few bugs to fix

okay so the kernel panics every time I run configure. And also when running neofetch

neofetch is deadlocking the kernel on this try

okay I have a panic but different this time. index out of bounds. Also I think I have a deadlock while printing the panic message

my kernel is completely destroyed

alright, time to sleep. gn

[late oar]

all those crashes may be due to a physical memory allocator issue. investigating

[late oar]

turns out configure in binutils has a space in its shebang #! /bin/sh. That fucks up its parsing in my kernel
also it seems I didn't handle the optional argument you can have in the shebang. So when neofetch has #!/usr/bin/env bash, the kernel looks for a file at the path /usr/bin/env bash

[inner pilot]

plenty of scripts do this

[late oar]

I'll fix that after work

[late oar]

I have a memory leak that has been standing here for a long time and I think I might have found the cause of it (or at least, a part of it)

When I open a file, I create a handle to it. But when I am not using it anymore, I need to explicitly call a function to close it. I did not implement that on Drop because it might return an EIO upon closure (in case the file was removed and this was the last handle to it, in which case the inode has to be removed from disk, which might cause a disk access, which might fail if for example, the USB stick has been unplugged)

Turns out there are plenty of places where I never call that function to close the handle

[late oar]

I have fixed shebangs handling, now let's try to figure out why bash segfaults when running neofetch

[late oar]

I still haven't done shit since

I guess I have to rebuild bash in -g3 to figure this out

[late oar]

alright, so far the only clue I have is that the segfault happens when freeing malloced memory

looking at info mem, there seem to be a single page mapped in userspace, at 0x8000 (I have lazy allocations)

bash's entry point is not in this page. Did memory mappings get fucked up at some point?

[late oar]

git bisect time

apparently this commit broke bash scripts:
https://github.com/maestro-os/maestro/commit/5d98d94bbe8b70e2e5a0e26c688b959beae9bce6

[late oar]

okay that's a TLB issue. Although there seem to be another issue making neofetch do an infinite loop. However adding invlpg makes binutils's ./configure run for a bit before crashing because of missing system calls

[late oar]

fix: https://github.com/maestro-os/maestro/commit/1f938977f046b31a318954b13926771784832009

[late oar]

I think this is also making ./configure very slow

gotta fix that at some point

alright. ./configure seems to miss sched_getaffinity and set_thread_area.
Actually set_thread_area is implemented but I just had it commented in the syscall table for some reason???

it seems to be missing getgroups32 too (my bash is built for 32 bit, the rest is in 64 bit). I guess I'll implement setgroups32 at the same time (and their 64 bit counterparts too)

[solid rose]

Why is your bash 32-bit?

[late oar]

Because I've never recompiled it when I've switched to 64 bit

[solid rose]

Lol

[balmy plume]

do you support compat mode?

[late oar]

Yes

[late oar]

I am currently attempting to optimise map_range to use the PAGE_SIZE flag.
When this is done, I'll attempt to optimise unmap_range so that it does not free tables (only unsets the PRESENT flag) so that they can be reused if remapped later. I'll also implement that thing where unused tables are freed if the system runs out of memory

[late oar]

dead project

I have implemented the PAGE_SIZE flag on map_range btw. But not the unmap_range thing.

Last time I worked on this I was implementing getgroups and setgroups

[inner pilot]

dead project

what's the PAGE_SIZE flag

[young ermine]

i believe it makes it use huge pages when appropriate

as in, if len >= 2M && virt & 2M == 0 && phys & 2M == 0 then map with 2M pages and probably the same for 1G

[late oar]

yes

[late oar]

this is done. also I've finally merged or closed the PRs other people made on Maestro that I've been procrastinating upon for 2 years

tomorrow I'm implementing sched_getaffinity and sched_setaffinity

[late oar]

current state of things

I am pretty sure the Not a directory error is due to doing stat on the terminal's file descriptor or something

[late oar]

also smp is buggy now apparently

update:

[fast halo]

Booting system with Linux kernel, release 0.1.0
wdym

[late oar]

had to replace the field in uname from maestro to Linux to make configure work

[fast halo]

what generated this output

i would've thought that's your kernel

[late oar]

my equivalent of systemd

which just uses uname

[late oar]

I would like to avoid this btw, do I have to patch every package I am building?

[silk wren]

probably. or you could have some "personality" like thing where uname behavior can be controlled by prctl or something

i.e. set prctl(PRETEND_TO_BE_LINUX, 1) before running configure.

[late oar]

Like, implement a command for it, to which I pass another command I want to do, and the command does the prctl, then executes the other command which inherits the prctl?

[silk wren]

yeah something like that

// pretend_to_be_linux.c
int main(int argc, char **argv)
{
        prctl(PRETEND_TO_BE_LINUX, 1);
        return execve(argv[1], argv+1, environ);
}

and then it's just pretend_to_be_linux ./configure ...

[zinc thistle]

https://github.com/maestro-os/visto does this a "x server from scratch" kinda thing or just the xserver to work on you os

[inner pilot]

what does this mean

[zinc thistle]

does he create another x sevrer like his own thing from scratch or just make the xserver work on his system

im a bit new to the programming world that why im suck a little

[silent lava]

Looks like a new X server based on the description

[late oar]

Yeah

But I think I'll rewrite it to use Wayland instead at some point

[silent lava]

Wayland is based

[inner pilot]

based

[shell tusk]

write a portable wayland compositor in c and I will port it to astral

[late oar]

will be in Rust, sorry :3

[balmy plume]

why don't you just port rust to astral?

[shell tusk]

I mean I could

[late oar]

or you can just cross-compile rust programs

and it should work just fine

[inner pilot]

[tough onyx]

Does it work?

[late oar]

it just colors your whole screen in red so far

[tough onyx]

Ah well that's a good start

[late oar]

been implementing a mocklinux command yesterday to make programs believe they are running on Linux. Not done yet

[late oar]

I've been implementing it in Rust, but maybe it would be a lot easier to do it in C

(for this particular command)

[late oar]

There's no way in Rust to just get a slice over argv. You can only get an iterator of OsString. So if you want to give to exec* you have to collect it, which does a memory allocation
Also I know this allocator is iterating on a Vec, which is built at startup using the program's arguments. That's yet another useless memory allocation

[silent lava]

lmfao really

[late oar]

yeah, that's crap

[silent lava]

Why don't they just give and take a slice of OsString

Just make OsString have the same repr as char *

[late oar]

OsString is the owned version (so that's yet other memory allocations). I think they should be using OsStr instead?

OsStr isn't null terminated and I think it's a slice underneath

[silent lava]

I get that this conversion nonsense fits slightly better into Rust's memory model or whatever but this is kinda dumb

[late oar]

yes

at the very least I would expect the vector to be built lazily (maybe it is, unsure) and I would like to have a unix-specific function where I can just retrieve the raw array

[silent lava]

IDK about other compiled to machine code languages but I can hardly imagine any of them do conversions this weird

Since rustc lets you change the signature of main, maybe it's time to make some PRs to them

[late oar]

I might attempt that

first have to check if someone already proposed the change (likely)

anyways, for now I'll write this in C and come back to it later

[late oar]

shamelessly stealing this code :3 (modifying it a bit to add extra checks)

[silent lava]

:3 is the best emoticon (it's impossible to change my mind)

[fast halo]

and yet i rarely use it why

[silent lava]

You haven't learned how great it is yet

[late oar]

make is working (if we don't count the fact that there is no way for the clock to know the current time, so the kernel currently thinks it's in 1970, which causes make to print a warning because of modification in the future )

[late oar]

how do I make it so that my kernel has the current time? I guess I need network support to use NTP if I don't want to use some legacy stuff?

[tough onyx]

rtc

[late oar]

that's legacy, isn't it?

[tough onyx]

no?

[young ermine]

or ask limine

it has the boot time request

[inner pilot]

you'd have to use limine for that

[young ermine]

i call that a skill gap

[balmy plume]

There's also a uefi runtime service (but on x86 it just reads rtc anyway)

[late oar]

fixing the Not a directory error (it's due to a wrong implementation of *at syscalls). And then I'll implement the RTC I guess

when I'll have the RTC I will also be able to add fsck in the CI to check whether integration tests are fucking up the filesystem (likely)

I commented this check because fsck gives an error if the timestamp of a file is too low

also fork is still slow as fuck because unmapping a range of memory is still slow. I'll have to improve that

[solid rose]

Do you have CoW?

[late oar]

yes

when I fork I unmap all writable mapping so that it triggers a page fault at the next access

[late oar]

I broke pipes by trying to fix them

I've refactored to add a pipefs and now pipes won't wakeup a process reading from it when the writing end is closed. This is a reference counting issue

[late oar]

I might have fixed it. ./configure seems to be doing stuff (I think)

I think my pipes may be dropping some data for some reason?

configure:4532: checking whether the C compiler works
configure:4554: gcc    conftest.c  >&5
conftest.c:2:4: error: expected '=', ',', ';', 'asm' or '__attribute__' before string constant
    2 | RT ""
      |    ^~
configure:4558: $? = 1
configure:4596: result: no
configure: failed program was:
| #define PACKAGE_URL ""
| RT ""
| /* end confdefs.h.  */
| 
| int
| main ()
| {
| 
|   ;
|   return 0;
| }
configure:4601: error: in `/home/luc/binutils-2.45':
configure:4603: error: C compiler cannot create executables
See `config.log' for more details

my guess is that those sources are going through a pipe and the pipe drops a part of it for some reason

[late oar]

the kernel is now reading time from the RTC at boot. I don't have warnings from make anymore

[errant venture]

i hope you use ACPI ™ to detect if RTC exists and other related properties

[late oar]

I've only checked if the century register is there

[late oar]

the answer is yes: there are filesystem errors

[late oar]

btw I also figured my timekeeping is too slow, there's like 1 second passing every 3 seconds

[fast halo]

do you know why

[late oar]

not yet

[solid rose]

are you using the PIT to calibrate

[late oar]

HPET iirc

but timekeeping is running with the RTC, which I don't calibrate iirc

also I represent the current time with a 64 bit unsigned representing the unix timestamp in nanoseconds, which I am not convinced is a good idea

[solid rose]

probably not

[solid rose]

ah, i only use the RTC to get the boot time and from there everything uses the HPET

RTC is slooow in vms so when i get the unix time i only read rtc at startup and then add hpet seconds to it (probably not a good idea but its faster)

[late oar]

maybe I should have the HPET do timekeeping, yeah

that would be an issue if there is no HPET though, but idk if I should bother about that

[solid rose]

you could use a one shot PIT to calibrate the TSC

[shell tusk]

The issue with the hpet is that it is piss slow, if possible support multiple timekeepint sources

My timekeeping priority is kvmclock -> invariant tsc -> hpet

[solid rose]

oh i forgot about kvmclock

[fast halo]

my issue with the HPET is that it can be disabled

its presence isnt guaranteed

[late oar]

how do I know if the tsc is invariant?

[solid rose]

its a cpuid bit iirc

[late oar]

and if it is variant I should switch to the hpet instead?

[solid rose]

if you want it to be invariant in qemu there's a flag for that, i think something like -cpu host,+invtsc should work

[solid rose]

variant is junk most of the time because the clock speed can change, so yeah

[late oar]

so I guess I'll do kvmclock -> invariant tsc -> hpet -> rtc

[solid rose]

you could also do a one shot PIT but iirc PIT is quite inaccurate in qemu

[late oar]

I guess I'll rework all of that once I get binutils/gcc to build as this should not be a blocking issue

[shell tusk]

acpi timer being forgotten once again: 💔

[solid rose]

oh yeah that exists too 😭

[late oar]

yeah I guess it could be used too. Before the HPET?

[shell tusk]

Idk I said it as a joke lmao

Idk how good of a timer it os

[late oar]

right

[inner pilot]

check menix code

[late oar]

*steal menix code

[inner pilot]

arch/x86_64/system/tsc i think

it's okay I'll nuke it sooner or later anyways

[balmy plume]

hm?

[late oar]

how do you guys do it?

[frail junco]

isn't that how linux does it too?

[fast halo]

It's a fine idea

You're not going to run out anytime soon

You have about 9,223,372,036.855 seconds since January 1, 1970

which is april 11, 2262

your kids' kids' kids aren't likely to live by that time, it's more than 200 years in the future

if you want insurance though just store it at the microsecond level

your clock is likely off by several milliseconds at least, anyway 😛

[tough onyx]

nah

I do invariant TSC -> HPET -> ACPI PM timer

and for calibrating the TSC I use the others

kvmclock i dont really get the appeal

I store everything in nanoseconds

[late oar]

Alright, guess I'll keep doing nanoseconds then

[tough onyx]

with nanoseconds you can do like 600 years

so its fine

[tawny drum]

it goes to show just how big a 64-bit value is

[solid rose]

But what about on x86?

[late oar]

?

[solid rose]

oh right you can just use uint64_t

[late oar]

Lately I've been trying to optimize memory mapping/unmapping by freeing pages only when the task dies or when the system is under memory pressure

[late oar]

alright guys I am fed up so that's gonna be for later

let's debug that pipe issue instead

[late oar]

turns out I am not on my usual computer and I don't have binutils, gcc and all that stuff built on the test disk I have here, so I instead I was working on my package manager so that I can have packages pre-built and stored on a remote server, so that I don't have to rebuild them for each computer I work on

[deft sonnet]

Smart

[late oar]

actually I was talking about only the pages used by paging (PDPT, page directories, pages tables, etc...)

my message was missing information

I would like to keep the entries and the underlying tables to make mapping/unmapping faster by reusing those tables

[deft sonnet]

Ahh I see

I’m currently making a queue of pages to free and having a worker thread do it for me

[late oar]

I am trying to upload packages to a s3 so that I don't have to rebuild them for each computer I work on, and:

upload failed: repo/dist/x86_64/binutils_2.44.meta to s3://pkg.maestro-os.org/dist/x86_64/binutils_2.44.meta argument of type 'NoneType' is not iterable

I fucking hate python

[worldly spruce]

@late oar how did u create the logo?

[late oar]

with a text editor. I wrote the SVG myself

I found it to be easier than using a drawing software

[worldly spruce]

Wow. That cool!

Wow

fr

But how did u make it look so good?

xD

[late oar]

it took 6 months of thinking and a lot of shitty attempts before coming up with something good

[worldly spruce]

Wow okay

[late oar]

btw I forgot to mention, but OKLCH might be useful for you if you want pretty colors
https://evilmartians.com/chronicles/oklch-in-css-why-quit-rgb-hsl

[worldly spruce]

Ok. I have heard about it. Thx!

[late oar]

still working on the package manager but since I am getting my usual computer tomorrow I guess I'll stop and continue debugging binutils's configure instead

fun fact: I can't build gcc anymore (on Linux) and I can't figure out why

aside from that, everything that I've managed to build is now on https://pkg.maestro-os.org
https://pkg.maestro-os.org/index has the list of packages in the repo, and each package can be downloaded with https://pkg.maestro-os.org/dist/x86_64/<package_name>_<package_version>.tar.gz (this is the package manager's job, not to be done manually)

[late oar]

gcc seems to be building so far (I didn't fix anything)

[late oar]

I think I am remembering I might actually never have packaged gcc in a tar with my build system, instead I probably installed it directly on the final disk

last time i went on holiday I implemented a cache for my package manager so that it doesn't re-download sources each time I attempt to build a package, as I noticed it was sucking all of my mobile data

[late oar]

dead project

[late oar]

this evening I'm continuing debug of configure

[inner pilot]

dead project

[late oar]

found my bug: O_APPEND not implemented

[late oar]

update:

time to re-run this with system calls printing to figure what's going on

[shell tusk]

damn you are super close

[late oar]

yeah

./configure is corrupting the filesystem, I'll have to check that

[late oar]

alright, rename and renameat2 are buggy, and the No error information messages are actually supposed to be Structure needs cleaning (the filesystem is fucked up)

[late oar]

been fixing up rename* during my lunch break, continuing after work

also the CI doesn't work anymore for some reason (can't find qemu)

[late oar]

btw I think I won't manage to fulfill my goal for this year which was to be able to code from Maestro itself (have gcc + vim + git with network)

networking won't fit before the end of the year (hoping I can even start working on vim)

[late oar]

managed to fix the CI, but the github actions runner only works on my local computer for some reason

[late oar]

fixed, that was a docker cache issue (the github runner runs in a docker)

let's get back to fixing rename and renameat*

[late oar]

My rename is a piece of shit, see: #filesystems message and following messages

[late oar]

Starting to think maybe implementing NVMe would make me save time when testing binutils/GCC compilation

[late oar]

@silk wren how do I do cycle-checking for rename? Do I iterate starting from the destination parent directory, going up until the root of the FS, checking if one of the node is the source file to be moved?

[fast halo]

as far as I know, it comes down to "the destination directory mustn't be located inside the target to be moved"

[silk wren]

https://cp-algorithms.com/graph/lca_binary_lifting.html

no but this is not information that you can reasonably maintain, so you're going to have to traverse parent pointers to test is_ancestor_of.

[late oar]

so that's what I've described?

[balmy plume]

dynamic cycle detection is hard for arbitrary graph changes

so yeah, in this case just walking up the path is the best solution

[balmy plume]

this is equivalent to dynamically maintaining the strongly connected components in a directed graph

and also equivalent to garbage collection

and the theoretically best known algorithms to that are not feasible in practice

[late oar]

yeah I'll walk up the path and have a per-filesystem rename mutex

[fast halo]

why do you need a rename mutex

[late oar]

to check for cycles, to make sure there's no change in between the time I do the check and the time I do the actual rename

[fast halo]

dont you need to lock just the vnode to move and the target destination

why not

[pastel summit]

i thought about this once and i'll be damned if i can remember the details of the renaming problem at the moment

in the meantime check out this fun code from dragonflybsd's tmpfs https://github.com/DragonFlyBSD/DragonFlyBSD/blob/d34567cafe70ead50df792facb580617c031e924/sys/vfs/tmpfs/tmpfs_subr.c#L1453

[late oar]

Didn't we have this conversation in #filesystems a few days ago?

[fast halo]

we did but i have my doubts over whether or not we ACTUALLY need the whole rename lock

[late oar]

funny

[late oar]

currently taking a break to avoid becoming crazy

[late oar]

I think I managed to rewrite rename correctly for ext2, although I haven't tested it yet because my kernel does not build. I have yet to fix the implementation on the tmpfs first before I can test all of that

[shell tusk]

@late oar do you use any tar parsing crate in maestro?

I'm thinking of using tar_no_std for the mlibc-demo-os filesystem but I am not sure how good that is

[late oar]

No. Do you want that for the initramfs?

I do it by parsing cpio myself

[shell tusk]

oh you use cpio

and yeah, to have shared loading support you can't just !include_bytes an elf :P

[late oar]

Yeah, easier to parse

[late oar]

Kernel still not building. I keep advancing on it (slowly) though

[late oar]

this evening I think I'll make a Maestro banner for my linkedin

Maybe a discord emoji with maestro's logo too. And I'll have to look at making stickers and tee shirts at some point

[inner pilot]

[late oar]

womenix

[late oar]

I don't have nitro so this is the best you guys will get on this server

[inner pilot]

L

[late oar]

no u

[late oar]

alright guys I am fed up. I am now refactoring the whole VFS

[late oar]

I have that thing Linux has where you have vtables represented by file_operations and inode_operations but instead of having a different one for regular files, directories, etc..., I had a single one per filesystem where I had a condition checking the file type (which kinda defeats the purpose).

So I am currently refactoring this crap to have an inode_operation per filesystem type and file type

that will simplify my code, making it easier to make rename work

also I'll probably be able to remove a mutex that wraps the target of a symlink in my tmpfs implementation (should be useless since this is set at the moment the symlink is created and can never be changed afterwards, but my code is not well architectured)

btw I've realized I have another issue with my VFS: When a filesystem is mounted at two places, I can have two dentrys pointing to the same actual entry on the filesystem (that's expected), however I've never implemented the thing to make sure when you remove a file on one mountpoint, the entry on the other mountpoint must be removed too

I think Linux does it lazily by calling d_revalidate while doing path resolution?

[inland mirage]

i dont get what you are trying to say. Also i think d_revalidate is for network filesystems

[late oar]

if the filesystem is mounted twice in different places of the filesystem, when you access a file in one, then access the same file on the other, you get two dentry in your vfs tree for the same file, right?

[inland mirage]

no its just one dentry

you create a dentry tree for that new mount and then the mount points point to the root dentry of that mount point/tree

so you don't duplicate the dentries for each mount if thats what youre asking

[late oar]

yeah

[inland mirage]

and on linux iirc if a dentry is a mountpoint you set a flag that it is a mountpoint and on lookup go to a hashtable indexed by that dentry's address -> to get the mount which has the pointer to the root dentry
https://elixir.bootlin.com/linux/v6.18/source/fs/namespace.c#L212

[late oar]

I think I have that lying somewhere in my code

[inner pilot]

bruh

[silk wren]

a path is a tuple of dentry (which is where in that particular filesystem a location points) and mount (which is where in the global hierarchy of mounts that specific path points).

so the same dentry tree can live at multiple locations in the filesystem hierarchy, just with different mounts.

[late oar]

yeah I saw that in linux but I didn't understand why it was necessary at the time

also I don't remember why I needed to have different dentry for two different mountpoint of the same filesystem

[silk wren]

Having two different dentries for the same location within a filesystem is essentially a guaranteed headache because you will have to come up with some invalidation scheme or otherwise workaround the issue of the trees going out-of-sync with eachother. And the two hardest things in programming are:

• cache invalidation
• naming things
• off-by-one errors
• scope creep
  so this problem is best solved by not having two dentries for the same location within a filesystem.

[late oar]

yeah, thx 👍

It's a good thing I talk about this project here. You guys can give me advises when I am about to write crap and that saves me a lot of time

[silk wren]

12304 brains are better than one, who would've guessed?

[late oar]

lmao

[inner pilot]

david is the goat

menix uses that idea for the vfs

[late oar]

this is done. I am now trying to fix filesystem corruption

also the CI is now doing fsck to check for filesystem errors after running integration tests

I didn't change the VFS to have only one dentry per file per filesystem yet

also at some point I'll have to implement that thing to check whether there's a mountpoint inside of another before unmounting (I guess that's just each mountpoint having a pointer to its parent mountpoint, and a linked list to its children mountpoints)

[late oar]

I am deleting a lot of code in my tmpfs implementation. Basically I had structures inside of it to represent files, and then at each operation I was modifying both these structures, and the VFS entries. I am removing those inner structure to only use VFS entries to make everything easier

[late oar]

I need to have a unique inode ID for each inode in the tmpfs, right? How do I do that? bitmap of allocated inode ID?

[young ermine]

just bump allocate them

[inner pilot]

why is this a problem?

you could bump allocate an inode every nanosecond and still not overflow in your lifetime

[balmy plume]

ig on 32 bit ino_t you have to keep track of allocations though

[inner pilot]

typedef uint64_t ino_t;

[young ermine]

lol real

a bitmap would suck though, if you want to store 1<<32 bits you need 512MiB of storage

unless you can somehow store a sparse bitmap

[balmy plume]

yeah that's true

[young ermine]

i'm not that smart it just sounds like a pain in the ass

i think thor did have a resource allocator of some sort that could be used for that?

not sure how that works

[balmy plume]

something like a set of intervals should work well

so that the size of that only grows linearly in the number of allocated inodes

[young ermine]

but then you'd have to scan for a free inode number, no?

[balmy plume]

no, you just store free intervals

[young ermine]

oh, free intervals

[balmy plume]

but you have to do coalescing when you free

[young ermine]

when you said it grows with number of allocated inodes i thought it would store used intervals lol

[balmy plume]

so you need some kind of ordered map (= a tree etc)

[young ermine]

yeah that's true

but it sounds relatively simple

and much better than a bump allocator, especially if your ino_t is 32-bit

[late oar]

that sounds like I should just use 64 bit inode IDs

[late oar]

I am pretty proud of the fact that this project's codebase is actually pretty sane now (that wasn't the case two years ago). I don't have to rewrite everything all the time

[late oar]

so I am using 64 bit inode IDs (was already doing that actually) and I have a bump allocator for the tmpfs

rn I am trying to make all integration tests pass. I still have directory listing and rename that are broken

[nimble grove]

numa support:

[late oar]

Oh yeah that's gonna be a big refactor. But it won't be coming soon

[late oar]

I have that issue with the tmpfs where, if you do getdents on a directory, and remove files in it, then getdents won't show the remaining files and that's due to the way I implemented it.
getdents takes an iterator on a hashmap containing the dir entries, then skips n elements (where n is the offset of the open file description). So once I did getdents once, this offset is updated, and after I remove entries, the offset is out of bounds, so the iterator does not return anything even though some entries remain

[young ermine]

i think that's fine, pretty sure linux also doesn't really solve that

this happens pretty much never

[late oar]

The rust stdlib seems to rely on Linux handling that

[late oar]

I think I have a null pointer deref (almost, there's an assert tripping before reaching it)

very likely an issue in the hashmap implementation

[inner pilot]

🚀 blazing null derefs

[late oar]

indeed

but that's unsafe code so that's my fault, not rust's fault

[late oar]

still trying to figure this crap out

[late oar]

@inner pilot would you fix the bug for me if I gave you 100€ for it?

[inner pilot]

which bug

[late oar]

I have this crash when unlinking a file after renaming it:

-- KERNEL PANIC! --
CPU: 0 Reason: unsafe precondition(s) violated: NonNull::new_unchecked requires that the pointer is non-null

This indicates a bug in the program. This Undefined Behavior check is optional, and cannot be relied on for safety. Location: /home/luc/.rustup/toolchains/nightly-2025-05-10-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:226:5
Callstack:
 <ffff80000021818a>: ???
 <ffff80000040a0ef>: ???
 <ffff800000403c44>: ???
 <ffff800000403cc7>: ???
 <ffff8000003ed74c>: ???
 <ffff8000003ebf08>: _ZN4core3ptr8non_null16NonNull$LT$T$GT$13new_unchecked17hdb4cd419be62f0d0E
 <ffff8000003ec114>: _ZN4core3ptr8non_null26NonNull$LT$$u5b$T$u5d$$GT$20slice_from_raw_parts17h92dfa8257571eabcE
 <ffff8000003ecc4a>: _ZN5utils11collections3vec12Vec$LT$T$GT$8as_slice17h1a42c19653d8d5e9E
-- end trace --
Halting...

I think it's the hashmap/hashset implementation being invalid and probably caused by this insert https://github.com/maestro-os/maestro/blob/4949eacb04e92ab46e72d20b19ab8c1573b863bf/kernel/src/file/vfs/mod.rs#L911 or the remove after it
That happens while running the integration tests

hashmap is implemented in here: https://github.com/maestro-os/maestro/tree/4949eacb04e92ab46e72d20b19ab8c1573b863bf/utils/src/collections/hashmap

(it's a swiss table implementation using simd. yes that was a bad idea in kernelspace and that will have to be changed at some point)

[young ermine]

simd in the fucking kernel?

[late oar]

yeah don't ask

[inner pilot]

bro

[young ermine]

bro nuke it immediately

[late oar]

turns out simd is not even enabled so the compiler emulates it

[inner pilot]

[young ermine]

lol

[inner pilot]

-100 performance

[late oar]

yeah

[shell tusk]

writes simd implentation for the challenge of it

[late oar]

worked pretty well until today (surprisingly). and i didn't touch it in 7 months

maybe I should just nuke it yeah

now I have to figure out how this is usually implemented without simd

[late oar]

@inner pilot do you happen to have a hashmap implementation for your kernel?

If you use the one in the stdlib I have bad news for you

[inner pilot]

what is it

[late oar]

Stdlib hashmap is a SIMD Swiss table too

[inner pilot]

but i don't use hashmap

only btreemap

[late oar]

Why no hashmap?

[inner pilot]

not available in alloc

[tough onyx]

swiss table is the abseil thing right?

it's neat

[late oar]

Yeah but simd

[tough onyx]

yes the abseil one is simd

[inner pilot]

ripperoni

[tough onyx]

the point of the design is to work with SIMD

since you can check like 16 elements at once

[inner pilot]

iirc alloc has no hashmap because of getrandom

[tough onyx]

why getrandom

cant you provide your own hash

[inner pilot]

apparently not

[tough onyx]

rust L

my shitty C code can do that!

[late oar]

Something something protection against denial of service

[late oar]

You can

[inner pilot]

but not the libc

[tough onyx]

https://abseil.io/about/design/swisstables

it's really cool

common google engineering W

[late oar]

Yeah I used that to implement mine

• a talk some guy from google did

[tough onyx]

I should look into implementing one

i already have my hamt thing which is good but who wouldnt want another datastructure?

[late oar]

The more I think about it, the more I think "Google" is a cringe name for a company

So 90s

[tough onyx]

Apple 💀

[inner pilot]

💀

[late oar]

do you guys think I could just reuse swiss tables but without simd (linear probing) or should I use another method?

[tough onyx]

yes of course

simd just allows you to do like 16 compares at once which is why its good with this design

[late oar]

alright imma start this refactor after dinner

[young ermine]

it was simd without using actual simd instructions anyway

[late oar]

yeah but in this case maybe there's a better algorithm?

[young ermine]

what is that for anyway

the swiss cheese thing

[late oar]

hashmap

[young ermine]

oh

[tough onyx]

just a simple chaining hashmap works fine too

the point of swiss table is to improve cache locality without having to deal with the disadvantages of open addressing

[late oar]

yeah but I liked that I could fit everything in a single memory allocation when doing a collect

[tough onyx]

do you really need collect?

id just do a chaining hashmap with avl trees or something

swiss table is really good with SIMD but without it it just ends up like regular open addressing

[tough onyx]

and you can have per-bucket locks too which is good

[late oar]

if i do chaining hashmaps, then I think I should make them intrusive. Otherwise I will end up in a lot of cases with having to make an allocation (for the linked list node) to store an Arc, which itself allocates memory to store the final object

however, intrusive stuff is annoying in rust

[tough onyx]

what do you use hashmaps for

what i would do is implement one ad-hoc everywhere I need and make the links intrusive

[inner pilot]

does that work with safe rust

[late oar]

in a lot of places

[late oar]

no

[inner pilot]

[late oar]

I think you can make intrusive stuff safe, except unlinking

[inner pilot]

only with atomicptrs

[ancient galleon]

just do regular chaining hash tables

they also allow you to do fine grained locking on buckets

[tough onyx]

[fringe pike]

btw there are tricks for making something like swisstable's probing work without the simd instructions

its obviously not as good

but its still not totally terrible

[late oar]

need motivation to do that

[late oar]

since the beginning of this project I have used my own collections to make sure I can handle memory allocation failures, but right now I am wondering if this was a good idea
instead, maybe I should make all allocations non-fallible and when we run out of memory, we always attempt to shrink caches or kill processes in order to get back the memory I need to make the allocation?

[nimble grove]

NO

youre doing the right thing

press on soldier

[late oar]

[late oar]

why is it the right thing though?

[nimble grove]

because i said so

jk

ill explain later im working rn and slacking off

[late oar]

alright guys I am fed up of the hashmap so I am nuking it and I am making the chaining one, with per-bucket locking, intrusive lists and that kind of stuff

[shell tusk]

Bro has a j*b 🤮

[late oar]

disgusting

[inner pilot]

couldn't you just kill the allocating process?

[late oar]

what if the allocating process is more important than the other processes that allocated memory before?

[late oar]

Maybe I should create a new branch and work on something else in order to get motivation again

What about NVMe?

[shell tusk]

xorg?

[late oar]

That's also a good idea

[inner pilot]

nvme was relatively easy even without interrupts

[late oar]

I want to start with nvme

then I'll go towards xorg I guess

[late oar]

got a friend who's really good at rust debugging my hashmap implementation and she didn't find any issue so far, so I am starting to think the issue might not actually be in the hashmap but maybe it is a buffer overflow overwriting the hashmap

[late oar]

https://github.com/maestro-os/maestro/pull/65

introducing the PR for the NVMe driver (wip)

[young ermine]

* Copyright 2024 Luc Lenôtre

crazy

time to update your file templates :^)

[late oar]

Yeah

[late oar]

very long working session 😔

[desert hearth]

lol

[tough onyx]

Your goal is to not ever reach OOM and having to kill random processes

Doing it like Linux sucks

[errant venture]

Fwiw Linux has an option to disable overcommit

[nimble grove]

does it do automatic swapfile expansion

[errant venture]

Dont think so

[errant venture]

Also Linux has lots of way to make it so its not random processes

You have oom_score_adj, you can write a psi monitor to implement userspace oom killer which systemd does, you can use cgroups

Etc etc

[tough onyx]

oom killer still sucks

[soft depot]

Also the way it starts slowing down dramatically (takes several seconds to switch VTs, etc) in low free memory conditions w/o swap isn’t much better

[late oar]

maybe I could also implement virtio-block aside from NVMe, and write a blog article about storage sometime

a NVMe will also show up as an IDE controller for backwards compat, doesn't it?
how do I avoid considering them as two different devices?

[young ermine]

since when does nvme show up as ide???

what black magic is that

[late oar]

idk I'd expect it to be that case since SATA has this kind of black magic, doesn't it?

for backwards compat

[inner pilot]

what

[solid rose]

I think he means like the PCI class

[late oar]

yes

[desert hearth]

Nvme has its own class/subclass

Thankfully its free from a lot of random bs

[desert hearth]

There will only be one pci device per nvme controller, so you won't have to worry about the second question here.

[late oar]

I may have misconfigured qemu but I remember seeing an IDE controller showing up too

I now have to figure out how the submission and completion queue work together

And find out how to get an interrupt when an operation is done (if that's doable)

[late oar]

should I have an I/O subimission queue per CPU core and disk?

[young ermine]

per-cpu IOSQ is something that's typically done, yes

but that of course requires your interrupt system to be able to allocate and route per-cpu interrupts properly

so many hobby OSes don't really end up doing it, they just do 1 IOSQ and call it a day

you can get very good performance that way too, so i wouldn't bother with per-cpu IO queues for now

[late oar]

Thanks!

[solid rose]

that may be the dvd drive

[late oar]

oh right

I boot the kernel with a .iso as a cdrom. Good catch

[solid rose]

ye

[late oar]

I have officially failed my goal for 2025 👍 (be able to code from Maestro itself). gotta make a blog article to say that

[late oar]

implementing nvme is a bit harder than I expected but it's fun to do 👌

[desert hearth]

how close did you get?

[late oar]

pretty far. I could not make binutils's ./configure run (I was trying to build binutils and gcc on Maestro itself to test stability). I didn't start porting Vim nor git and I didn't start the network stack

the only thing I have pretty much is binutils, gcc and make working

[desert hearth]

I see, impressive progress nonetheless.

[late oar]

thanks!

I could not run configure because rename was corrupting the filesystem. So I rewrote a chunk of the VFS, but I've got stopped by a hashmap bug (use after free)

[desert hearth]

ugh

[late oar]

I've got a friend who think they found the issue on the hashmap, but they didn't make a PR yet

(something something shitty remove function)

I was about to rewrite the hashmap to make it intrusive and use chaining (so that I can use less memory allocations and have per-bucket locking) but in the end I don't feeling like doing that right now

[young ermine]

which parts do you find hard to implement?

[errant venture]

Maybe if you've never dealt with submission/completion ring model before

Which is the same on basically all modern hw