Zinnia

[somber solar]

coroutines my beloved

[brisk totem]

on the other hand

C++ is way more scuffed

[near tartan]

ive never seriously written C++

[somber solar]

there's libasync

[near tartan]

most i've done is C++ that looks like C

xd

[brisk totem]

doesnt it hard require stl

[somber solar]

freestnd-cxx-hdrs ?

[brisk totem]

also afaik it doesnt do oom safety

[fickle elbow]

Why do people love async IO syscalls that much? Is it not way simpler to wrap a blocking IO op in a thread and thus make it nonblocking?

[brisk totem]

simpler yes, faster no

threads are expensive

very

[somber solar]

I looked at this recently, why is it even in a .cpp file
https://github.com/iillis/Hirie/blob/main/src%2Fhirie.cpp

[fickle elbow]

How so? Just curious

[near tartan]

i don't want any external libraries besides uacpi (and maybe smoldtb)

[brisk totem]

they need a stack, making them requires syscalls

scheduling more threads is pretty expensive

and rescheduling into a different thread is very expensive too

because its a syscall

[near tartan]

hm

maybe i should just scratch async

too much effort

not like i'm trying to do linux roleplay

[brisk totem]

async cool

also linux doesnt do (real) async

[near tartan]

yea ofc

async is cool

but i have a job 😭

[brisk totem]

lol fair

[near tartan]

that's the main motivation behind keeping things simple

but actually, C++ isn't a bad shout

i probably want a better type "safety"

[somber solar]

https://tenor.com/view/patrick-star-minor-s-spongebob-spongebob-meme-spongebob-squarepants-gif-10940848928744248792

[fickle elbow]

I feel like with a microkernel you won't have any use for an async pipeline inside the kernel since anything but primitive I/O is moved into user space

[brisk totem]

you still want to be able to do nonblocking IPC

[fickle elbow]

shared memory

[brisk totem]

you still need a doorbell mechanism

also shared memory is kinda expensive

[fickle elbow]

how so?

[brisk totem]

you need to allocate shared memory

most likely quite a lot of it

for every pair of processes that communicate

[fickle elbow]

I mean that's up to the process

the kernel should just deliver utilities for shared memory

and locking

[idle flower]

use core::{
    ops::{Deref, DerefMut},
    sync::atomic::{AtomicBool, AtomicUsize},
};

use alloc::sync::Arc;

use crate::{global::GlobalData, heap::ProcessNumaHeapAllocator, random::CSPRNG};

use super::{arch::cpu, memory::VirtLv1PageAddress};

pub const TLS: CoreLocalPointer = CoreLocalPointer();

pub struct CoreLocalPointer();

impl Deref for CoreLocalPointer {
    type Target = CoreLocalData;

    fn deref(&self) -> &CoreLocalData {
        unsafe { &*cpu::get_cl_pointer() }
    }
}

impl DerefMut for CoreLocalPointer {
    fn deref_mut(&mut self) -> &mut CoreLocalData {
        unsafe { &mut *cpu::get_cl_pointer() }
    }
}

pub struct CoreLocalData {
    pub(super) arch_data: [u8; 64], //TODO move it into a concrete type definded in arch::*
    pub core_id: u64,
    pub core_csprng: CSPRNG,
    pub system_plm_offset: VirtLv1PageAddress,
    pub process_plm_offset: VirtLv1PageAddress, //Move to a dedicated struct later
    pub global_data: Arc<GlobalData>,
    pub thread_data: TlsData,
    //pub process_data
}

impl Drop for CoreLocalData {
    fn drop(&mut self) {
        panic!("Undroppable Value Dropped")
    }
}

pub struct TlsData {
    pub process_id: u32,
    pub local_heap: ProcessNumaHeapAllocator,
    pub held_locks: AtomicUsize,
    pub yield_to_scheduler: AtomicBool,
    pub current_plm_offset: VirtLv1PageAddress,
}

[near tartan]

ah

[near tartan]

lol

pragma region my beloved

praying that this isn't a clang only thing

[near tartan]

okay i am probably going to take this

yea fuck it i'm going through with the rust rewrite

i like my interface design so much better than the C version

[near tartan]

someone please tell me wtf gnulib is doing

i've been here for a solid 5 minutes

i fucking hate gnu

[spark surge]

this is why we use tarballs

[near tartan]

how is that fixing anything

it's the download itself lol

[spark surge]

gnulib is shipped inside tarballs

also tarballs are provided using better mirrors

[near tartan]

ah

i kinda wish i could go with the llvm toolchain (like chimera) but my patches don't work yet

soon

™️

[pine rock]

who says you need to patch llvm

just tell it you're targeting freestanding and pass in all the defines and sysroot from the command line in configure

im definetly not putting off adding my own toolchain to llvm

[fiery marlin]

nooo (but seriously take a break and come back with fresh ideas, I find that always helps me if I get stuck)

[near tartan]

thx

[near tartan]

[near tartan]

@spark surge does limine not give a shit where the executable wants to load even if it's at !0 when ET_DYN?

[spark surge]

disable KASLR?

[near tartan]

oh true

[near tartan]

i was gonna yap about how this unclear from the protocol doc but who cares

reading helps

[near tartan]

am i stupid or is the tsc not a thing on my cpu

cpuid 0x15.{ecx,edx} is 0

[eternal wharf]

do you pass -cpu host to qemu?

[near tartan]

yes

ah im missing this check

[eternal wharf]

could also just measure the frequency via some other timer

iirc amd cpus don't have the frequency in msrs

[near tartan]

i don't really want to use the tsc to begin with

[eternal wharf]

why not?

[near tartan]

why would i

if i need another timer then atp i can just use that

[eternal wharf]

i mean what other timer are you gonna use

[near tartan]

hpet or lapic

in the C version i exclusively used hpet on x86

[eternal wharf]

fair, although when available the tsc is the nicest imo

[near tartan]

that's the point

it's not sometimes

[eternal wharf]

i mean it's pretty old at this point

invariant tsc that is

[near tartan]

well, at the point of enabling it i would need another timer to calibrate it

i guess i can give it higher prio than the HPET at least

or actually, i should move cpu init after firmware init

yea then i can also use the tsc

@idle flower need uacpi-rs

[hoary cave]

most cases you can get away without calibration for invariant tsc, and in vms which dont pass that information through you can use kvm clock

and maybe fallback to hpet

that covers about 99.5% of modern machines

imo

[near tartan]

wdym without calibration

how will i know what the frequency is without :P

[hoary cave]

u can figure out the frequency using cpuid

[near tartan]

where

[hoary cave]

.

[near tartan]

this doesn't work

[hoary cave]

yeah some hosts dont pass that info through

[hoary cave]

so just use kvm clock

[near tartan]

but it's there on real hw?

[hoary cave]

pretty sure if you have invariant tsc you have that

but idr

would be a good idea to check lol

i wonder if you can find info about that online

or maybe the sdm

im on phone so kinda hard to browse the sdm

[eternal wharf]

the main advantage is that it's just 1 insn to read vs having to do mmio

[near tartan]

true

[eternal wharf]

and with x2apic scheduling a deadline interrupt is a single msr write

in tsc deadline mode

[hoary cave]

invariant tsc + tsc deadline is the best you can get on x86

realistically

rolls over in god knows how long, dead simple to use, only downside is the calibration sometimes

[eternal wharf]

actually nvm, it's always an msr write

[hoary cave]

setting it up in xapic mode could be mmio though

but really you do it only once

and it doesnt go through pch

[eternal wharf]

yeah the lapic config might require mmio

[hoary cave]

so who cares

[eternal wharf]

but the deadline is an msr

[hoary cave]

lapic accesses are caught by the cpu

[eternal wharf]

always

[hoary cave]

yeah thats nice

[near tartan]

I'm doing this now

i still need to convert the other cases to use Hz

huh

what unit is the tsc frequency in

or rather

how do i properly convert to nanosecs

[eternal wharf]

stamp * 1e9 / freq

[near tartan]

uh

[eternal wharf]

makes sense, the initial value of the tsc is whatever

on real hw it'd at least include time spent in bios and the bootloader etc

[near tartan]

but this is supposed to be seconds already

lol

[eternal wharf]

oh i didn't pay attention to the large jumps between values

[near tartan]

this is all i have rn

atm it takes the second branch

and the tsc is cpu local, right?

hm, how would i best represent that in code?

i have one central module that does timekeeping

but i would potentially need to keep a record of the different frequencies, no?

[eternal wharf]

linux assumes the frequencies are the same on all cores

[near tartan]

ah

okay that makes it easy then

very cool

maybe that messes with big.little stuff?

who knows

[idle flower]

which apis do you need first?

[near tartan]

hpet and mcfg

need those tables

the rest idc about yet

and madt actually

[near tartan]

okay super weird

in order to make this actually calculate the correct value i have to multiply by 1000?

what

this looks wrong

[distant cypress]

in the calibration loop, why do you not read both the TSC and the reference clock before and after the wait? you might've been preempted in between.

[near tartan]

this runs before the scheduler

[distant cypress]

i.e.

t1 = read_tsc ();
ref1 = read_ref ();
sleep;
t2 = read_tsc ();
ref2 = read_hpet ();

[distant cypress]

regardless, it is a good idea

[near tartan]

what would that get me

[distant cypress]

better measurements probably

[near tartan]

in what way

[distant cypress]

it is quite likely that wait_ns waits slightly longer than 1s

that is mitigated by reading both the tsc and the reference clock before and after

iirc I did like a 100ms calibration loop, and that gives me very accurate measurements

[near tartan]

yea 1s is probably overkill

[distant cypress]

however you can calibrate over a longer time period. for example by scheduling a timer event for 1s into the future, and recalibrating the TSC then. syncing the recalibrated TSC to other CPUs might be a little tricky, though.

[distant cypress]

as in,

• measure tsc and ref
• sleep >=1s (+ scheduling jitter)
• measure tsc and ref

[near tartan]

okay i think i know what went wrong

i didn't check for the 0x15 leaf

i only check for the tsc bit

[somber solar]

I hope you mean the invariant tsc bit too?

[near tartan]

suuure

[somber solar]

lol

[near tartan]

midnight coding

ill fix in the morning

pragmatic means it works on my machine

call that optimization

[near tartan]

hm

[near tartan]

okay checking the leaf didn't work either

[fiery marlin]

Leaves 0x15 and 0x16 are quite new I believe, and are only available on intel cpus.

You'll want to support calibrating the tsc with another timer (hpet, acpi, pit) or if you're running under KVM you can use pvclock which gives you all the info you need.

[somber solar]

cpuid 0x15, kvm, acpi pmclock, hpet, pit

in that order

[near tartan]

i would use the hpet if uacpi-rs was ready

[somber solar]

didn't you say you were giving up on rust?

[near tartan]

.

i'm bipolar probably

idk

i change my opinion on things every day

[somber solar]

https://tenor.com/view/metronome-gif-17433149157757821754

marvinome

[near tartan]

real

[pine rock]

just write some interop code so you can write your kernel in C, C++, and Rust depending on your mood

also the hpet really isnt that hard to do yourself

[near tartan]

i had that ironically

[near tartan]

it's not about that

[pine rock]

its just an acpi table and some mmio

[somber solar]

cursed

[near tartan]

i already had hpet in C

but i need uacpi bindings

because i dont want to duplicate table defs

[pine rock]

the table always has the same in memory layout, just memcpy it

[near tartan]

rust

[pine rock]

or std::bit_cast it if you're feeling fancy

oh yeah shit language i forgot

[near tartan]

i have a transmute thing

[pine rock]

fuckin uh unsafe { std::mem::transmute } it

[near tartan]

nono

it's safe

[pine rock]

"""safe"""

[near tartan]

https://github.com/menix-os/menix/blob/rust/kernel/src/generic/sched/process.rs#L31

yes safe

[pine rock]

incredible that rust needs a library to read(fd, buffer, sizeof(buffer))

what will they think of next

[grave peak]

yep, thats truly insane

[pine rock]

if the C++ comittee was like 30% less brain damaged rust would be screwed

[somber solar]

more democracy into C++

[pine rock]

sadly this is not a perfect world

[somber solar]

developers vote for features

[grave peak]

https://stackoverflow.com/questions/25410028/how-to-read-a-struct-from-a-file-in-rust

take a look at this

[pine rock]

lol, lmao even

[grave peak]

rofl, if you feel so inclined

[near tartan]

i mean

i get their point

reading random data into structures is generally not a good idea

[grave peak]

just dont deserialize anything

[near tartan]

if you can guarantee that it's not bogus (like POD types) it's fine

[grave peak]

keep everything in memory

[near tartan]

real

🤓 unsafe rust is still safer than C

[grave peak]

uhhh

[near tartan]

this is going to make reading from disk really fun

tbh i might just make that part unsafe to get some speed

[somber solar]

safe language users when they need to use unsafe everywhere:

[near tartan]

not an argument btw

[pine rock]

rust users when they need to store indicies into global arrays but its definetly not shared mutable state because borrowck is ok with it

[magic charm]

a common style of pattern i was forced into when i was experimenting with rust (don't blame me for it, a lot of people experiment with rust in their youth)

take when i was wondering how to express a graph structure in rust

the pattern i was told to use was have nodes instead of pointing to other nodes directly, instead have indexes into some node vector or similar

it can be argued this is in fact less safe

[near tartan]

just Arc it

[magic charm]

because if you run your C program with valgrind, you can in fact get nice errors if you try to dereference a pointer to a node that was deallocated. but with indices or some other kind of identifier, you'd need to add your own logic for never reusing identifiers in a debug mode

[storm bobcat]

Yea this is the kind of stuff that's annoying

I tend to have to do a lot of hacks like that

[grave peak]

bruh

[uncut jackal]

I am fucking tired from explaining people "having to use unsafe" is not an argument

outside of my kernel and some langdev project where I was doing pointer tagging (which is obviously hilariously unsafe) I've used the unsafe keyword maybe 10 times

and like 6 of them was for FFI

[somber solar]

it was a joke

[uncut jackal]

borrowck is still borrowcking the global arrays unless you're doing something very wrong

[somber solar]

hence the trollface

[uncut jackal]

and Weak<> exists

[hoary cave]

unsafe impl Sync

[uncut jackal]

yea not necessarily targeted at you I should have made that clear

just something I hear very often (and often unironically)

[uncut jackal]

this is usually very bad lol it's worse than most transmutes

[hoary cave]

ik

(hence trollface)

[uncut jackal]

there's a fat bird here

I think this bird is fat

idk looks fatter irl

[somber solar]

chonky boi

[uncut jackal]

right above it is a tiny bird

so funny

I think this bird ate its siblings

[somber solar]

[near tartan]

this should be correct now

still doesn't work properly in tcg but idc anymore

[somber solar]

tcg doesn't have invariant tsc

you can't enable it

at least I don't think you can

smh "at list"

I just wrote a linked list

[near tartan]

at list

i hate it

is there another timer i can use without acpi

[somber solar]

pit

[near tartan]

[somber solar]

kvm pvclock

but as you would imagine, only on kvm

[near tartan]

duh

[grave peak]

I mean uacpi rs has table support no?

[near tartan]

i dunno what the state is

[pine rock]

man it is a single struct

please

[near tartan]

vro

actually it's 2

[pine rock]

[near tartan]

hm, looking at my old interrupt code rn

how would i "rustify" this signature

is there a way to not have to resort to a void*?

some generic bs maybe?

idk

[storm bobcat]

actually I think it works if your host cpu supports it but qemu doesnt report it

which is retarded ofc

[cosmic birch]

Just box a closure and you'll have all the context that you want

[near tartan]

what

[storm bobcat]

I dont think a box here would work

that would require memory allocation

[cosmic birch]

You don't need to allocate them in the interrupt handler, you can just make an array of boxed closures that act as irq handlers

[storm bobcat]

yea but isnt Box on the heap

[near tartan]

can yall give some example

[storm bobcat]

he's looking for a way to store a pointer to callbacks

which boxed closure would be the solution in userspace

[cosmic birch]

You also have alloc in no_std rust

[storm bobcat]

yes

but this is presumably in early kernel init

[near tartan]

i want to avoid having to transmute the context pointer

that's what this is

[storm bobcat]

I think just an array of fns would work no?

[near tartan]

this is not my problem

[storm bobcat]

ah the context

yea just use *u8 or something

theres not really a way to make it generic and safe I think?

unless you make each irq a different type

[near tartan]

yea i couldnt think of anythin either

[storm bobcat]

you could probs do something funky

with generic types

like

[cosmic birch]

Or you'd need to use the heap and do dynamic dispatch, but if you don't have inizialized an heap you really can't

[near tartan]

rn i do this

[storm bobcat]

idr rust but something like

type Irq32Handler = IrqHandler<Irq32Context>

[near tartan]

i have heap at that point

[storm bobcat]

yeah

I just assumed no heap

[cosmic birch]

Then you can make irq_handler an array of boxed closures like this: [Box<dyn Fn()>; 256]

[near tartan]

what's the point of doing that

[storm bobcat]

generic functions

but then you dont get the return type

[near tartan]

that doesn't solve the issue in any form

just makes it more awkward lol

[storm bobcat]

does it not?

[near tartan]

at some point i still have to transmute

i don't think there's any way around it

[storm bobcat]

yeah

do you really need a context parameter tho

what is it for

[near tartan]

so the interrupt has device specific info

[storm bobcat]

theres probably a better way to design this

[near tartan]

linux does it this way :P

ofc linux is not perfect

but it has its uses

also global state = yuck

[storm bobcat]

this is a microkernel, right?

[near tartan]

no

modular

but mono

[storm bobcat]

oh i thought it was

damn

[storm bobcat]

the NT way is using DPCs and stuff

which is probably what you want anyway

[near tartan]

i have no idea about any of that honestly

[near tartan]

is it normal for qemu tcg to say my TSS is TSS64-avl, but kvm to say it's TSS64-busy

i mean it doesn't matter since it goes through, but i seem to be doing everything like the SDM wants it

[near tartan]

hm

now i need to think of a way to build crates as either rust lib or dylib

[idle flower]

For dynamic linking?

[near tartan]

kinda

my C version could either link the code dynamically as a module or have it be built into the kernel

maybe i should drop builtin support altogether

[idle flower]

you want to dynamically link rust-rs to your kernel?

[near tartan]

rust-rs?

[idle flower]

uacpi-rs

[near tartan]

no

drivers

[idle flower]

then they arent crates

unless you want to directly fetch drivers from crates.io

[near tartan]

wut

what else are they

if not crates

[idle flower]

a project afaik

[near tartan]

no

projects != crates

https://doc.rust-lang.org/book/ch07-01-packages-and-crates.html

[idle flower]

til

[near tartan]

anyways

i'll probably have it create a dylib

[idle flower]

yes afaik rlibs are not real binaries but rusts intermediate language which has the same version stability as the rust abi currently aka none

like std and core are shipped as a rlib

[near tartan]

technically i could make it emit a staticlib

ah wait no that has C linkage

[near tartan]

how cursed is loading .dynstr and .dynsym in rodata?

as long as it works

[near tartan]

@spark surge what's the actual difference between the exec file feature cmdline vs the cmdline feature?

does the latter simply not allocate the kernel file?

[spark surge]

they are the same

the latter is a short hand for the former

[near tartan]

so how does it work? does it always allocate the kernel file?

id like to avoid using memory that i don't need

[hoary cave]

well it has to load the kernel file into memory to load it i guess

and instead of freeing it, it's just passed along to the kernel in a response

[spark surge]

don't worry about it kitten

[near tartan]

🥺

[sick flax]

@spark flax

[hoary cave]

osdev kittens

[modern hamlet]

you can reclaim it if you need the memory

[near tartan]

hello

[near tartan]

bruh this is new

im stupid

rela stuff has to be in the text segment

okay i should print what symbol is unresolved

ah i see the problem

because i link the .dynsym and .dynstr as PT_LOAD, limine can't access them anymore as they are now in virtual space

@spark surge is this something PR worthy?#

[spark surge]

are they supposed to be PT_LOAD?

[near tartan]

i want them to be PT_LOAD so i can access them from the kernel

but nevermind

[grave peak]

just make limine provide u with the kernel binary?

[near tartan]

i had a typo

but still

unresolved symbols should maybe be ignored?

not sure if it's panic worthy

[spark surge]

i am not sure what you need tbh

if it's something valuable then yeah it's PR worthy

[near tartan]

it was a fault on my side

limine does handle symbol table offsets properly

i had a typo in my linker script, but since i compile the kernel as a dynamic library, it simply does weak linkage

which limine doesn't like

maybe it could print which symbol is unresolved

@spark surge

[spark surge]

that's fine

i'll merge this

[near tartan]

thx

[grave peak]

why not resolve the index to a string

[spark surge]

yeah that's a fair point

[near tartan]

i can do that

shouldnt be too hard

[spark surge]

btw

you shouldn't do common: ... in commit msgs

do lib/elf: or something

[near tartan]

ah sry

[spark surge]

i'll force push this commit away

then you can send one all together with the fixed commit msg

[near tartan]

@spark surge like this?

[spark surge]

yeah i guess

[near tartan]

🥺

[spark surge]

looks aight to me

[near tartan]

dope

[somber solar]

@near tartan Marvin alt

[near tartan]

melvin

[pine rock]

marvins evil twin

[grave peak]

he probably works at redhat

[steady echo]

are you sure he is the evil one

[near tartan]

i need some rust gods to help me

for some reason rust includes sse2 shit in my build

but my target turns it off

[somber solar]

https://tenor.com/view/crab-rave-dancing-dancing-crab-gif-16543314

this is the rust god

[near tartan]

that's uhhh

funny

okay it's triggered by BTreeMap::insert

but wtf is it doingggg

never mind

me when i double lock 🗿

what the fuck is this code

this shit sucks so bad

lmfao

[grave peak]

Lock one more time just to be sure

[near tartan]

well my slab allocator is completely broken

now it's doing null derefs

cool

[pine rock]

debug build moment lol

isnt pause an llvm intrinsic

[near tartan]

idk

it's interesting though

my allocator is completely fucked

because the translation from C is incorrect

the page wise allocation works

yay

[pine rock]

does the rust stdlib not just come with a pile of premade allocators for you

[near tartan]

no

[pine rock]

oh

[near tartan]

why would it

[pine rock]

idk it seems like the sort of language that would

[near tartan]

no language does afaik

[pine rock]

zig does

[near tartan]

well, theres probably allocator crates

[near tartan]

how does it work in no_std

[pine rock]

the allocators just take void *ptr, size_t size in their constructor

then allocate from that

[near tartan]

i dont really see how that can work in a no_std environment

[pine rock]

i dont see why it doesnt

[near tartan]

what about reserved regions

or can you pass multiple regions to the allocators

[pine rock]

for some of them you can iirc

i mean i've got like half a dozen allocators i've stolen off github that were made for hosted targets that work just fine freestanding

its not a particularly hard thing to do even by accident

if you've got a block of memory to allocate from you can store all your control structures in it

[near tartan]

im tempted to just use a crate for allocations

[hoary cave]

zig has an Allocator api, basically an opaque pointer + vtable pointer that has a few methods like allocate, reallocate, free
it can ship allocators in the std because they are built on top of each other, a GeneralPurposeAllocator needs a backing allocator which can provide memory for it and by default it's std.mem.page_allocator which uses mmap in userland but you can override it to do what you want iirc, or just tell GPA to use your own allocator which can call into pmm + map the pages

[pine rock]

no shame in using a crate for an allocator

[brisk totem]

if its a bad crate, lots of shame

imo

[brisk totem]

this design has one flaw which is that it makes cpu speculation unhappy

because vtables slow

[somber solar]

they aren't that slow

almost the same as regular pointers to functions

and can be the same in some cases

[near tartan]

any recommendations?

https://github.com/SFBdragon/talc

this one looks nice

also has oom handling

[brisk totem]

i mean given that macos malloc uses a vtable dispatch on the critical path

yeah lol

[brisk totem]

i NIHed mine

if you want a general purpose one, i.e. not for a kernel, dlmalloc probably

[near tartan]

for a kernel

my slab impl isn't rusty at all

it's just a huge unsafe block

[brisk totem]

thats fine tbh

[near tartan]

and doesn't work

[brisk totem]

oh thats less fine :^)

[near tartan]

yea

only page wise allocations work

some sort of corruption happening

but i cant be asked ngl

[brisk totem]

lol

[near tartan]

🗿 the redox slab impl is fucking shit

[brisk totem]

lol

this is the allocator from shkwve: https://paste.sr.ht/~pitust/43699e812923d956c892d94cbfd9b204624a6866

[near tartan]

i think i'll just use talc

[near tartan]

transition to talc complete

now we can go to userspace finally

[grave peak]

hows rust osdev so far

[near tartan]

it's not too bad

you have a few global state anchors here and there

but the rest feels pretty nice

page mapping is the worst offender, that's almost purely unsafe

but it'll be way more fun when i'm building abstractions around this

now that the allocator is actually fast + efficient, i don't really have a lot of things left to do on the low level / mostly unsafe side

[grave peak]

thats nice

[near tartan]

next step is remaking the vfs and fs abstractions

i want to focus more on fuse support

[near tartan]

i think i will buy this thing https://de.aliexpress.com/item/1005006965588873.html

[uncut jackal]

there are

e.g. there's linked_list_allocator

[near tartan]

already done :)

[fair lintel]

port managarm

[near tartan]

bet

i already have a wip mlibc port

[near tartan]

this is the most useless fucking table ever

[near tartan]

the loongarch manual is ass

okay it's just the html version

mental illness

yes, please give me XVFRSQRTE.D apples

[eternal wharf]

the names makes sense though? eXtended Vector Floating-point Reciprocal SQRT Estimate (dot) Double-precision

i did cheat slightly, i had to look up what the E stood for

but the other parts are pretty obvious

[near tartan]

didn't say that it doesn't make sense

x86 simd instructions aren't better in this regard

https://tenor.com/view/pearto-falling-down-stairs-fruit-pear-kasane-teto-gif-7247886229247103273

[near tartan]

okay i need to finish the mlibc loongarch port sometime soon

i need it to build limine

[somber solar]

@near tartan is the rust menix public?

[near tartan]

yes

different branch

ill switch to it as the main branch once i have everything back to the state of the C version

still missing are the scheduler, vfs and fbcon

[near tartan]

please save me

i'm on my 9th llvm rebuild

[pine rock]

ah the joys of writing an llvm toolchain

i must've rebuilt llvm 50 times over 3 days when i did mine

[near tartan]

#

[near tartan]

question

[pine rock]

👁️

[near tartan]

how do you deal with compiler built ins

i can't use libgcc

but for some reason i can't build libclang_rt.builtins.a either

[pine rock]

oh that was a fun one

[grave peak]

just implement yourself lool

[analog crane]

happens for using llvm

skill gap

[near tartan]

[spark surge]

you know what is a skill gap

[pine rock]

so you build llvm for the build machine including compiler-rt, then install the headers for your libc, configure llvm again to only build libc++ and compiler-rt using your own sysroot, and then you can use that

[spark surge]

being a mlibc maintainer but being scared to merge things but at the same time literally no one is telling me yes or no

[near tartan]

trolled

[analog crane]

schrodingers PR

its neither fine nor to be closed

[near tartan]

what about 1323 btw

[pine rock]

remember to specify -DCMAKE_C_COMPILER_WORKS=1 -DCMAKE_CXX_COMPILER_WORKS=1 when configuring libc++ and libunwind for your target

[spark surge]

i am talking about 1323

mlibc/mlibc when

[near tartan]

yea when merge

real

[spark surge]

whenever korona tells me "ok"

[near tartan]

so next week

[spark surge]

or Arsen

[pine rock]

this is my full option set until i have a working rtld implementation

[spark surge]

or ideally both

[spark surge]

it's really fucking annoying

i am quite burnt out about PRs

[pine rock]

then dont make them

[spark surge]

i've been sending many PRs, so many that i lost count

not just to mlibc

to other things too

it's so annoying, you just wait on other people and often other people are dead

and you're just sitting there waiting

i am losing my marbles

[near tartan]

do you have a full set of build instructions?

[spark surge]

open source development is a failure

[pine rock]

oh wrong paste my bad

https://github.com/apache-hb/BezOS/tree/master/repo/packages/tools you'll probably care about llvm-libcxx.xml and llvm-compiler-rt.xml

[near tartan]

interesting build system

[pine rock]

and this is the patch for my toolchain https://github.com/apache-hb/BezOS/blob/master/repo/patches/llvm/add-bezos.patch

[analog crane]

https://www.youtube.com/watch?v=RIfS5HcjiVc

[pine rock]

wrong patch my bad

updated the link

[pine rock]

dont worry about it

i hate yaml more than i hate xml

[near tartan]

fair

[grave peak]

just press the merge button

you're an mlibc maintainer anyway

[pine rock]

just say you're following googles live at head policy and that breaking other peoples builds simply improves product quality

[near tartan]

@pine rock i kinda don't understand

why enable compiler-rt and libcxx in llvm

[pine rock]

thats to build them for the build machine

[near tartan]

ah

@spark surge well you seem to have green light by arsen

[spark flax]

you're not gaslighting mint are you

[near tartan]

i wouldnt

maybe

[spark flax]

I'm just joking

[spark surge]

miracle

[near tartan]

lfg

[spark surge]

no that's actually not what he said

i think he wants me to add a mlibc-crossers for loongarch

mlibc/mlibc when

[near tartan]

what's the point again

[spark surge]

none really

pedantry

he wants to have a "proper" "targetted" "toolchain" even though in this case it makes hardly any difference practically other than making the CI more annoying

[near tartan]

finally managed to have a "decent" way of declaring module metadata

[hoary cave]

am i blind

i see some strings at 2080h

but where's the metadata

also what if you just used an elf section to store your metadata

like .note.menix-module or smth

[near tartan]

guess what

[near tartan]

that is the metadata

it contains module name + description + author + dependencies

[hoary cave]

yeah that's what i gathered :^) but they look like unterminated strings, which means you have to store the length somewhere

[near tartan]

i store them in the phdrs

[hoary cave]

ahh that makes sense

[near tartan]

though for dependencies i'm tempted to just parse DT_NEEDED

[hoary cave]

what if a module wants to have more than 255 dependencies with names longer than 255 characters

i think u might be cooking with dt_needed

[near tartan]

yea

[near tartan]

i could've made it use usize but dt_needed is cooler

@hoary cave hell yes

seems to work

[hoary cave]

epic

how do modules work? how are dependencies defined? do modules expose a C API or are they just regular rust crates?

[near tartan]

regular rust dylibs

dependencies are gathered automatically through cargo

[hoary cave]

so you can basically use any public module from your kernel in your modules

same with other modules

that is kinda cool

[near tartan]

yep

this is already much cleaner than the C version

where you had to hardcode your module dependencies

this is how you use it

[hoary cave]

yeah i checked out the code

looks pretty nice

but the question is, how do i contribute :^)

[near tartan]

huh

did i break the contribution guide

[hoary cave]

oh no

is there one?

i am blind :^)

[near tartan]

i think

should be on the main readme

[hoary cave]

nope, but there is instructions on how to build it

ah in main branch

[near tartan]

check rust branch

at the bottom

[hoary cave]

wow i am blind

yeah looks about what i'd expect

but it was more of a question of uhh

what needs to be done

[near tartan]

ah

[hoary cave]

no issues = nothing to do

:^(

[near tartan]

i can open up some feature requests

there's a lot that needs porting from C

most importantly vfs

:^)

[hoary cave]

i see

[near tartan]

also the scheduler is not in yet, but that's kinda blocked by uacpi-rs

scheduler needs interrupt needs timer needs hpet needs uacpi

I haven't checked the state of uacpi-rs yet

[hoary cave]

lol

im assuming you need the barebones stuff?

[near tartan]

ye

should be enough

i have a pretty large diff rn with the module changes

and the module loader

question is

are the kernel and modules going to fit in the -2GiB address space

probably

then i can just load kernel modules after LD_KERNEL_END

[hoary cave]

you can, and you probably should, give them a chunk from the 0xffff'8000'0000'0000..0xffff'ffff'7fff'ffff address range

but what's left after the kernel end is probably more than enough lol

but keep in mind KASLR could make that smaller

[near tartan]

the c version just hard mapped it to 0xffffc..+

[near tartan]

good point

i don't see any way to have the dylibs not named libfoo.so

is there a way to do that with cargo?

like instead of naming it libfoo.so => foo.ko

but not a big deal if not

[hoary cave]

.ko implies it's an object, which it isn't

but i think you can change that in your target spec json

[storm bobcat]

It is an object tho

[hoary cave]

yeah but it's a shared object

[grave peak]

nope

its a normal object

[hoary cave]

no no i mean

menix modules

are shared objects

[grave peak]

ohh

[hoary cave]

linux is using normal objects, that is true

thats why i said that .ko is a bit misleading

but not entirely untrue, it is an object

[grave peak]

.kso

[near tartan]

lol

[hoary cave]

you can use dll_prefix to remove the "lib" i think

[near tartan]

meh

not that big of a deal

i now store version, author and description

the rest i can encode in the elf

[hoary cave]

{
  "dll_prefix": "",
  "dll_suffix": ".kso",
}```

[near tartan]

like entry point, dependencies, name

[hoary cave]

:^)

[near tartan]

where

[hoary cave]

target json

[near tartan]

ah

[hoary cave]

i think you can just slap that at the top level

i dont know if there's a way to do this globally instead of per-target

[hoary cave]

the dll_suffix might be wrong but i think that's what i did last time

might need to remove the dot idk

[near tartan]

doesn't seem to be right

[hoary cave]

weird

https://doc.rust-lang.org/beta/nightly-rustc/rustc_target/spec/struct.TargetOptions.html#structfield.dll_prefix

[near tartan]

ah i got it

with -

not _

[hoary cave]

ah i got trolled

it makes sense

but yeah empty prefix and .kso suffix will produce foo.kso

[near tartan]

lfg

[hoary cave]

hell yeah

[near tartan]

@hoary cave i'm going to add some cards in the github kanban board

[near tartan]

i made it private 🤦‍♂️

[hoary cave]

oops lmao

[near tartan]

well the issues should be open

ill make it readonly though

[hoary cave]

yup issues are there

[near tartan]

i probably want to redesign the vfs because the c version sucked major balls

[hoary cave]

also wasn't your plan to make a microkernel?

[near tartan]

#1287456798407790684 message

it's cool but ultimately too much work imo

my goal with menix is to have some sort of desktop and oss games running on it

[hoary cave]

so unix like kernel with mlibc

the buzz astral meme and all

[near tartan]

how original

[hoary cave]

i mean making it run on all 4 platforms will be cooler than it being able to do wayland/xorg

imo

[near tartan]

true

if i do ever get my hands on hardware, i would also consider ppc64 support

but for now uefi/acpi based systems seem like a good base

i have a few aarch64 devices with acpi

[hoary cave]

i don't, but i have an m1 mac that i can probably try to get menix running on which should be fun :^)

[near tartan]

sure

i also have one

[hoary cave]

once the aarch64 port is up of course

[near tartan]

yea

i try to keep arch stuff to an absolute minimum

oh right, the m1 uses dt boot

[hoary cave]

yep

[near tartan]

well i should add support for it sooner or later if i want to realistically run on !x86

[hoary cave]

wait im sorry what is dt boot

i thought about something else

[near tartan]

device tree boot

[hoary cave]

like acpi vs dt?

[near tartan]

ye

[hoary cave]

ah yeah

i thought there was already some code for dt

[near tartan]

only a stub

when it checks for xsdp/dtb

i will prefer acpi 100% if i get the xsdp

but i think uboot efi emu should be decent enough to boot from

[hoary cave]

yeah it's fine-ish

nothing crazy

[near tartan]

i dont really need the crazy protcols

or i could just rely on limine

[hoary cave]

based

[near tartan]

i abstracted the boot method away anyways

mb1

[hoary cave]

multiboot would be a nice addition

even though

the protocol is absolute dogshit

it's still nice to have diversity

[near tartan]

you can pr it in

[hoary cave]

to make sure your abstraction is good enough

sounds fun

[near tartan]

you only need to call 3 functions and add a feature for it

[hoary cave]

i wonder how putting it all in one binary will work

[near tartan]

maybe a prekernel?

like eir

[hoary cave]

that would work, yes

[near tartan]

then i could also just map my own hhdm instead of having to stick what limine gives me

actually yes, let's try doing that

[hoary cave]

yeah at that point kernel doesn't care what booted it

[near tartan]

that's the better abstraction probably

at the very beginning of the rewrite i was planning to do that but i failed to come up with a good way to share e.g. vmm code

[hoary cave]

share as in...

between kernel and "eir"?

[near tartan]

yes

[hoary cave]

why would you need to do that

i mean

i guess the code deduplication is nice

but it's like 100 lines of code you need for the page table manipulation

it's not that big of a deal imo

[near tartan]

maybe the uptick in effort is not worth it lol

fair

hm

i need to think this through

maybe the prekernel can also handle smp setup?

no that would be dumb

[hoary cave]

i mean

not that dumb of an idea

limine does it :^)

[near tartan]

i need to be able to control cpus at runtime

[hoary cave]

you could do it in the prekernel to save yourself some trampolines

[near tartan]

like

i would also need cpu setup code in both binaries

and the acpi stuff should also be part of the kernel

yea no i think that's easier

just let it be a boot proto abstraction

for now

[near tartan]

okay im making the board readonly now

https://github.com/orgs/menix-os/projects/2

[near tartan]

i wonder if there's a way to get rid of .rustc

it currently takes up 50% of the kernel size

[pine rock]

/DISCARD/ : { (*.rustc*) } and see what happens

[near tartan]

dies

[pine rock]

damn

[near tartan]

it "needs" it to link

but i don't actually need it to load

it's NOLOAD anyways

but it's annoying

[pine rock]

objcopy and discard the section i guess

objcopy -R .rustc kernel.elf i think

[idle flower]

What does it contain?

[near tartan]

linking info

metadata

[eternal wharf]

get rid of it by rewriting in c++

[near tartan]

🤫

[near tartan]

i'm experimenting

trying to find a nice and ergonomic api for accessing packed structures

[hoary cave]

SpecFieldLE + SpecFieldBE maybe?

struct DtbHeader;
impl DtbHeader {
  const MAGIC: SpecField<u32> = SpecField::new_ne(0);
  const TOTAL_SIZE: SpecField<u32> = SpecField::new_ne(4);
  const OFF_DT_STRUCT: SpecField<u32> = SpecField::new_ne(8);
  // ...
}
fn test() {
  // ...
  _ = space.read(DtbHeader::MAGIC);
}```

[near tartan]

misc.rs

/// Aligns a value to the next higher multiple of `alignment`.
pub const fn align_up(value: usize, alignment: usize) -> usize {
    return value.div_ceil(alignment) * alignment;
}

/// Aligns a value to the next lower multiple of `alignment`.
pub const fn align_down(value: usize, alignment: usize) -> usize {
    return (value / alignment) * alignment;
}

pub trait Primitive: Copy + Default {
    fn swap_bytes(self) -> Self;
    fn from_ne_bytes(buf: &[u8]) -> Self {
        #[cfg(target_endian = "little")]
        return Self::from_le_bytes(buf);
        #[cfg(target_endian = "big")]
        return Self::from_be_bytes(buf);
    }
    fn from_le_bytes(buf: &[u8]) -> Self;
    fn from_be_bytes(buf: &[u8]) -> Self;
    fn to_ne_bytes(self, buf: &mut [u8]) {
        #[cfg(target_endian = "little")]
        self.to_le_bytes(buf);
        #[cfg(target_endian = "big")]
        self.to_be_bytes(buf);
    }
    fn to_le_bytes(self, buf: &mut [u8]);
    fn to_be_bytes(self, buf: &mut [u8]);
}

macro_rules! impl_primitive {
    ($($ty:ty),*) => {
        $(impl Primitive for $ty {fn swap_bytes(self) -> Self {<$ty>::swap_bytes(self)}
        fn from_le_bytes(buf: &[u8]) -> Self {<$ty>::from_le_bytes(buf.try_into().unwrap())}
        fn from_be_bytes(buf: &[u8]) -> Self {<$ty>::from_be_bytes(buf.try_into().unwrap())}
        fn to_le_bytes(buf: &[u8]) -> Self {<$ty>::to_le_bytes(buf.try_into().unwrap())}
        fn to_be_bytes(buf: &[u8]) -> Self {<$ty>::to_be_bytes(buf.try_into().unwrap())}
})*
    };
}

impl_primitive!(
    i8, u8, i16, u16, i32, u32, i64, u64, i128, u128, isize, usize
);

firmware/fdt.rs

// OpenFirmware flattened device tree handling.

use crate::generic::spec::MemoryField;

struct DtbHeader;
impl DtbHeader {
    const MAGIC: MemoryField<u32> = MemoryField::new_be(0);
    const TOTALSIZE: MemoryField<u32> = MemoryField::new_be(4);
    const OFF_DT_STRUCT: MemoryField<u32> = MemoryField::new_be(8);
    const OFF_DT_STRINGS: MemoryField<u32> = MemoryField::new_be(12);
    const OFF_MEM_RSVMAP: MemoryField<u32> = MemoryField::new_be(16);
    const VERSION: MemoryField<u32> = MemoryField::new_be(20);
    const LAST_COMP_VERSION: MemoryField<u32> = MemoryField::new_be(24);
    const BOOT_CPUID_PHYS: MemoryField<u32> = MemoryField::new_be(28);
    const SIZE_DT_STRINGS: MemoryField<u32> = MemoryField::new_be(32);
    const SIZE_DT_STRUCT: MemoryField<u32> = MemoryField::new_be(36);
}

pub fn init(dtb: *const u8) {}

[near tartan]

okay col

@hoary cave i solved the .rustc issue

it's only used during compilation, so after the build i can just strip the section from the .ksos

the metadata is irrelevant to the kernel during the actual dynamic linking

and that still works

[hoary cave]

that's a lot of symbols

[hoary cave]

i thought you needed the section during runtime to figure out which modules have to be loaded and all?

and to provide metadata, but i wasn't sure what you wanted to do with it

ah right, you get the required modules from DT_NEEDED

[near tartan]

that's not the dynamic section

it's just some abi info in .rustc

[hoary cave]

yeah i know, you are talking about your own metadata section

oh

you mean the .rustc section

i totally missed the dot

[near tartan]

lol

[spark surge]

vc when

[near tartan]

it cuts down kernel size from 1.9M to 500k

out of which 300k is symbols

[near tartan]

soon

going to take a shower (rare) then eat

first ill do loongarch

[spark surge]

(rare)

[near tartan]

then mlibc menix

[hoary cave]

i already ate and had a shower

so i can vc soon too :^)

finishing up jinx stuff probably

or working on mmio thing

[near tartan]

no shower = no vc

[near tartan]

okay mlibc stuff is almost done

i need to update my gcc patches to take the crts into account

le device has been obtained

might convert into a server for my rack

[near tartan]

https://www.amazon.de/dp/B0D8TJB7BL

i might buy this

[uncut jackal]

tbh at this point, targeting a qcom laptop is better than loongarch imo

also since when is menix ported to 4 architectures lol

weren't you rewriting

[near tartan]

yea

[near tartan]

why not

[uncut jackal]

i was surprised as hell

how feature complete is each port?

[near tartan]

that's a different story

the rewrite isn't even that far yet

but i want to have a different approach

instead of doing x86_64 only first, i want to work on all 4 ports so i can make the most portable abstractions

i want to have very little arch-specific code in menix

[near tartan]

both have uefi, that's enough for me

[uncut jackal]

i tried that and doing page tables for 3 architectures blew my brain out

i tried that and doing page tables for 3 architectures blew my brain out

ok discord wtf

[near tartan]

lol

the paging code already works for x86 and riscv

[uncut jackal]

risc-v has the easiest page tables imo

[near tartan]

true

[uncut jackal]

and x86 was second

aarch64 is crazy

[near tartan]

what's different?

[uncut jackal]

aarch64 has a lot of different configurations and (n)G(n)R(n)E is comparatively harder to encode IMO

[near tartan]

huh

what sort of different configurations?

tbh aarch64 is the arch i know the least about

[uncut jackal]

e.g. 16k pages

[near tartan]

oh

yea no i'm not supporting that

just 4k

[uncut jackal]

and there's also 64k iirc??

idk if it's actually implemented anywhere

[near tartan]

linux

[uncut jackal]

i mean in hardware

[near tartan]

oh

[uncut jackal]

16k is somewhat common tho

[near tartan]

well macbooks are 16k, right?

or the iommu at least

[uncut jackal]

but every UEFI capable device has 4k

[near tartan]

menix exclusively boots in uefi environments

this is a hard requirement

[uncut jackal]

i believe something requires the host system to be 16k but in virtualised environments 4k is supported

so the 4k is there in silicon, most probably iboot or something drops you in 16k

or whatever

idk

[near tartan]

all that i care about is that snapdragons are 4k

[uncut jackal]

raspberry pis also have 16k as an option

[near tartan]

🤢

[uncut jackal]

and raspberry pi is pushing it a little slowly

there's performance gained apparently

[near tartan]

btw how did you notice this