Zinnia

[fiery marlin]

yeah that works

[hoary cave]

how?

you're bump allocating

you don't have aligned memory unless you skip over a bunch of unaligned memory

[fiery marlin]

yeah you can do that, just have to keep track of it

or start bumping from the largest region of memory, and track the offset you started at

its not pretty but it works

[hoary cave]

which is not pretty, and you can only keep track of so much without a proper allocator in place

or, well, you can throw away a bunch of .bss for that

[fiery marlin]

you can also modify the memory in-place if you want

[hoary cave]

and you can reclaim it later

[fiery marlin]

you dont need to know about all those reserved memory regions

[hoary cave]

you can also just separate the physical allocator from the page allocator and you can use the physical allocator during init to create page tables, only downside is that you don't get page struct pointers at this point yet but i think adapting from physical addresses to page structs shouldnt be too hard

[fiery marlin]

if I may though, I dont know if its worth supporting bigger translation sizes for something like struct page, because you'll end up including memory for small gaps in the memory map. With 4K mappings, you might have saved a page or two, but a single 2M mapping would cover all of it.

I was trying to make use of 2M (and 1G lol) mappings myself for this, but it didnt seem worth it imo, especially with the added complexity of finding 2M chunks of physical memory early on.

[hoary cave]

as i said, i might have overthought it

[fiery marlin]

eh maybe (and maybe not), I've also just had a thought: I mostly test with < 256M of ram allocated to qemu, so there is probably some benefit to 2M mappings for larger amounts of ram, for the inevitable big block of memory you end up with.

[near tartan]

i test with 2G ram

[hoary cave]

it's mostly a trade off between code complexity and memory usage

[near tartan]

i don't really mind losing a few MBs, but i don't want to lose too much to simple code

[hoary cave]

it's not exactly simple, but yeah

i think just mapping with 4K pages should be fine

[near tartan]

simpler*

[fiery marlin]

well you would potentially waste more using 2M mappings

[near tartan]

what about the hhdm? that too with 4k?

[hoary cave]

well, you don't allocate memory for the hhdm lol

but for the pages you should use 1GiB pages if possible

[fiery marlin]

I was thinking more about the increased number of page table walks, since there's more translations with 4K mappings

[near tartan]

yea i already use 1g pages for the hhdm

i just map from 0 - limit including the holes

@hoary cave you're in vc tonight, right?

[hoary cave]

if i feel liek it, sure

[near tartan]

would be cool if we can do some brainstorming, i'm literally blocked by this shit

i can't do anything else

i have a bunch of things that i want to implement but it's all brought to a stop by the allocator being a POS

[idle flower]

i will use the following method:
take the highest continous memory area (with size > ? pages), calculate the needed amount of memory for the buddy allocator, take pages out of the memory area to create the needed page table entries, build the buddy there and mark the used pages as allocated. Allocate and build all remaining buddy allocators.

[near tartan]

okay, i'll do a bootstrap bump allocator to get page tables and early allocs running

then switch over to the buddy allocator after the new pt is live

[near tartan]

just thought of something funny to implement

kexec

[somber solar]

when fun and funny aren't the same

[near tartan]

well at this point it's just not practical

but would be cool to have an interface for it

[near tartan]

what the fuck man

[idle flower]

Your code?

[near tartan]

not really

that's limine rs

[somber solar]

and people complain about auto &&...args lol

is & in rust a reference?

[near tartan]

yes

[near tartan]

that's not the same

[somber solar]

I know, but it's more than one symbol stuck together

[near tartan]

i understand though how it got to &&&

it's an iterator over a reference to a slice of &Entry

[hoary cave]

i don't think the ref to a slice makes a difference here

[near tartan]

you could theoretically avoid this by doing that

[hoary cave]

.iter() should make it an iterator over &Entry

which it is

i mean, what's the point?

[near tartan]

idk

[hoary cave]

auto deref is a thing

entry.length works

just fine

no?

[near tartan]

it's a non issue yea

[hoary cave]

just funny looking

[near tartan]

just funny inlay hints

btw i might actually have to make the memory map take owned memory

i think

or i could just not reclaim limine structures

i guess that saves me like 1 or 2 pages at most

eh not worth it

[modern hamlet]

idk how you can bear with this language

my name says I program in C++ but I saw some absolutely incomprehensible C++ code yesterday which ruined my day

[near tartan]

skill issue

it doesn't really affect my code

[somber solar]

hyprland be like

well not really incomprehensible

just bad

[hoary cave]

wow, yes, blame the language and not the programmer :^)

[modern hamlet]

tbh maybe if i had learned rust at 14 or 15 i might be proficient in it

[modern hamlet]

well the programmer was to blame for that bit of c++ i mentioned

[near tartan]

bruh?

[modern hamlet]

but i dont understand rust as a whole, like, the referencing system and borrow checker and things like that

it kinda goes over my head

[near tartan]

you have to work a bit in it

[modern hamlet]

i'm so used to writing code in one way that rust is a paradigm shift to me

[hoary cave]

you have references in c++ too

[modern hamlet]

but maybe i should give it a shot

[modern hamlet]

oh did i say references

[near tartan]

if i have to choose rust vs c++ ill choose rust any time, but rust vs C, rust just can't compete

[modern hamlet]

i meant lifetimes

[hoary cave]

ah, yeah

i mean, lifetimes are a nice way to reason about your code

[somber solar]

I never undestood C > C++
C++ is just C but with more stuff

[hoary cave]

more ways to blow your foot off? :^)

[near tartan]

c++ does what c does but fucks it up every time

[somber solar]

you don't like something? don't use that thing

there are million other things

[near tartan]

yes, i wont use c++

because I don't like it

[somber solar]

I meant a feature in C++

[somber solar]

#1287456798407790684 message

[near tartan]

how is that related

the language is c with more obstacles

[somber solar]

how is it not related?

[near tartan]

so i can just use c and not have to deal with it

[somber solar]

it's a matter of knowing a language

[near tartan]

no

[hoary cave]

generics in C:

[somber solar]

knowing a language doesn't mean memorising every stl function and the syntax

[near tartan]

mf called LIST(name, typ)

[somber solar]

it means knowing how to actually use it

[hoary cave]

criminal

but you know what's worse?

void hashmap_insert(struct hashmap *hm, void *key, void *item);

[somber solar]

map[key] = item;

[near tartan]

i know how to use it, and i still don't like it

[somber solar]

well

I say skill issue

[near tartan]

i know how to use it
skill issue

[somber solar]

that means that I doubt it

I "know" rust but not well enough to actually like and use it

[near tartan]

let me put it this way

c++ has no features that i'd use that c doesn't have

yes, generics are cool, but not a hard requirement

and most of the stl things are just wrappers around the same concepts

[somber solar]

that's a better reason

almost every language is just a wrapper around libc though

[near tartan]

duh

[hoary cave]

yes almost every compiler is a wrapper around execve(/usr/bin/as, ...) if you think about it

[near tartan]

real

[modern hamlet]

not even classes?

and the general 9 yards about OOP

[near tartan]

classes are structs

[modern hamlet]

structs with field protection types and methods

[near tartan]

constructors are glorified __attribute((section(".init_array")) void foo();

[modern hamlet]

(well structs in c++ can have methods too, but i am purely contrasting to c)

[near tartan]

i don't need any of that if i'm writing C

public_cast

[modern hamlet]

but you'd use these features if you wrote C++ right

[near tartan]

sure

[modern hamlet]

or would you just write C in C++

[near tartan]

do you know what orthodox c++ is

[modern hamlet]

something about the right way to write c++?

[near tartan]

https://gist.github.com/bkaradzic/2e39896bc7d8c34e042b

that's what i'd write

my c++ opinion has been influenced a lot by company code

and yes, this may be a skill issue

[modern hamlet]

most of my C++ code respects these rules actually 😉

[near tartan]

based

[hoary cave]

rtti is awesome

why would you not wanna use it

[near tartan]

rtti
kernel

[hoary cave]

?

[modern hamlet]

who said anything about a kernel

[hoary cave]

why would they be mutually exclusive

even in a kernel

[near tartan]

don't you have to turn that off?

idk

[brisk totem]

thats pretty much how i end up writing C++ ngl

[modern hamlet]

but anyway the exceptions are, well, exceptions (my protobuf impl uses those), and cstdio, cstdlib, cstring etc

[hoary cave]

???

it's literally

code running at the end of a scope

[brisk totem]

i looked into making rtti in kernel

its really not that bad lol

[hoary cave]

OH

RTTI

Sorry

[modern hamlet]

im pretty sure RTTI means runtime type information

[hoary cave]

im fucking retarded

💀

yeah

[near tartan]

💀

[brisk totem]

dynamic_cast can fail without exceptions

[modern hamlet]

you thought of RAII

[hoary cave]

i am thinking RAII

[near tartan]

oh yea i agree with that

[hoary cave]

RAII is awesome

[brisk totem]

RAII yes ofc

i mean personally i prefer sane move semantics

[hoary cave]

RTTI is kinda shite but it's whatever

[modern hamlet]

dont count not using the STL which uses memory allocation because i do not care about memory management

[near tartan]

exceptions are dookie

[brisk totem]

it comes in handy

[modern hamlet]

and it says "dont use STL functions which use memory allocation unless you dont care about memory management"

[brisk totem]

my personal policy is to avoid the STL at all costs

that includes implementing my own version of <concepts>

i should try writing my own common C++ library with sanity™️ baked in

[hoary cave]

that's true but generally you don't really need it

[brisk totem]

need no

nice to have yes

also it helps reverse engineering because now you have rtti metadata in vtables :^)

[hoary cave]

real

[brisk totem]

the only place where i disagree is that i want to misuse Boost.PFR

because its cool

[near tartan]

i don't use boost at all

[brisk totem]

boost.pfr is standalone

[near tartan]

a

[brisk totem]

its also the only cool part of boost

also i obviously nihed a shitty reimpl

lol

[near tartan]

if the answer to most SO answers is "use boost", you really gotta start thinking if your stdlib is good

i felt that so much during game engine dev

[brisk totem]

and you end up with "wonderful" generated code like this

who doesnt like that

and of course count is another bit of wonderful code

that i think i codegened wrong lol

[hoary cave]

what the fuck

[brisk totem]

oops forgot i

it should count up

[near tartan]

bro

[brisk totem]

not be 1

lol

[hoary cave]

lmao i thought i was insane

[brisk totem]

lol this is the right version

fun fact i cant write code

even if its python code

[near tartan]

if you can't then i'm giving up programming

which i will do regardless because i can't get the brainpower together to calculate the amount of pages to reserve for my allocator

https://tenor.com/view/it's-over-over-it's-gif-3551756699416095564

[brisk totem]

just allocate some

and hope its fine

[near tartan]

okay pitust i dont think you saw the discussion

[somber solar]

you just link some code from gcc and boom

[brisk totem]

no

[somber solar]

yes

[brisk totem]

without linking gcc code

obviously lol

linking gcc code is easy and therefore cheating

[somber solar]

by gcc I don't mean the compiler

[brisk totem]

i know

also gcc is the L compiler

[near tartan]

but i was thinking of starting with a bump allocator to bootstrap the kernel page table and buddy allocator

[somber solar]

clang is better yes

[brisk totem]

all my homies link llvm

[brisk totem]

yeah thats the actually good solution lol

[near tartan]

question is

should i bother with trying to calculate the amount of pages i need beforehand

or just pray that the next free memory region big enough???

oh wait i can just get the largest entry 🗿

[brisk totem]

MM_Allocator MM_init(MAP_MemoryMap* map) {
    MM_Allocator alloc;
    alloc.addr = 0;
    alloc.size = 0;
    alloc.size_orig = 0;
    for (usize i = 0; i < map->num_entries; i++) {
        if (map->entries[i].type != kUsable) continue;
        if (map->entries[i].size < alloc.size) continue;
        alloc.addr = map->entries[i].base;
        alloc.size = map->entries[i].size;
        alloc.size_orig = map->entries[i].size;
    }
    return alloc;
}
u64 MM_alloc(MM_Allocator* alloc, u64 size, u64 align) {
    u64 target_base = align_down(alloc->addr + alloc->size - size, align);
    assert(alloc->size >= size, "MM: out of memory");
    assert(target_base >= alloc->addr, "MM: out of memory");
    alloc->size = target_base - alloc->addr;
    return target_base;
}
void MM_finish(MM_Allocator* alloc, MAP_MemoryMap* map) {
    MAP_add_entry(map, kLoaderPrivatePageTables, alloc->addr + alloc->size, alloc->size_orig - alloc->size);
    MAP_cleanup(map);
}

answer 3: steal shkwve code :^)

[near tartan]

[brisk totem]

all you need to do is to implement the easy MAP_cleanup function :^)

[near tartan]

"easy"

why does that sound so ominous

[brisk totem]

its not actually that bad

its just easy to fuck up

and hard to do without O(n^2)

[near tartan]

idk what it entails

[brisk totem]

in my impl, sorting and merging entries

you can also just shrink the memory map entry in place tho

i do it this way because idfk lol

i think because i already implement a memory map cleanup function and this was easier

[near tartan]

sort by what?

[brisk totem]

base

actually its more complicated than that

[near tartan]

my memory map is already guaranteed to be sorted by base

[brisk totem]

my impl sorts all endpoints

then for each pair of endpoints figures out the right type

and builds a memory map out of that

[brisk totem]

yea you dont actually need to do that you can just store a pointer to the entry you are allocating from

and bump alloc out of it

i wanted to make it reclaimable later which means i need to do accounting for it

[near tartan]

ah

[brisk totem]

also i have fdt support so i needed this shit anyway

[near tartan]

yea i also need fdt support

since i want to support riscv and aarch

[brisk totem]

so youll have the joys of memory map fixup anyway lol

unless you just use limine

[near tartan]

i use limine

[brisk totem]

yeah then it doesnt matter

[near tartan]

rn it's hard coded but i may experiment with a prekernel

à la eir

[brisk totem]

shkwve is freestanding so i needed that crap

i should work some more on shkwve

[near tartan]

true

shkwve is nice

[brisk totem]

well no

it would be nice if it worked

which it doesnt

[near tartan]

and tbh for a bootloader rust is actually overkill

a kernel has much more complex things going on

[brisk totem]

i switched to C++ a while ago

but its still not working

[magic charm]

how goes shkwve for that matter?

[brisk totem]

not

i should work some more on it

[near tartan]

it stands

tbh I'd love to see shkwve running on the VF2

[brisk totem]

same

[near tartan]

extra points if it can load menix

[brisk totem]

but i cant focus on projects

unfortunately i just had an even worse idea than using C++ and so now ill probably be rewriting it again

[near tartan]

ada

[brisk totem]

no

swift

[near tartan]

you're unwell

[brisk totem]

lol

yes

but actually i mean gclang good

and swift is a gclang with reference counting

therefore, swift is a great fit for a kernel

there is a clear connection there

okay maybe i wont use swift thats just stupid

oh yeah i remember what i got stuck on last

[near tartan]

the best gclang is still go

[brisk totem]

C++ async crap

[brisk totem]

yeah 100%

[near tartan]

but i dont like gclangs to begin with

[magic charm]

there is potential in a kernel where (most of) it is written in a gc lang

[brisk totem]

but i cant use go in shkwve

[near tartan]

yet

[brisk totem]

im insane but im not that insane

[brisk totem]

theres tinygo but its not really usable for this iirc

[magic charm]

SPIN from the 90s was one (modula-3, has both unsafe untraced and safe traced pointers; loadable modules have to be safe)

while the MIT pdos group did a kernel in go

i dislike that both of these however had a fixed-size, unexpandable/unshrinkable gc'd heap (i understand why it was much easier this way, but it's still a big price to pay)

[near tartan]

i mean if you want to have any real progress in a short amount of time you should stick with C/C++

[brisk totem]

yea tinygo has that problem too

[brisk totem]

yep

im still not having a lot of progress because of other issues but yk

the main problem is that i am too skill issued to design a good device abstraction

[near tartan]

bruh

me too

[brisk totem]

the problem is that its a really fucking hard problem i think

[near tartan]

idk what rustisms i want to do for my abstraction

[brisk totem]

imo none

or minimal at least

[near tartan]

well I'd have a device trait

so i can get display name and so on

have some sysfs stuff

[brisk totem]

idk if you saw how the rust version of shkwve worked

(using a lot of insane stuff)

[near tartan]

didnt really look into it

tldr

[brisk totem]

i had a special version of Rc<T> which also can get you at the header

which has a pointer to the metadata which is used for all the dynamic dispatch stuff

then i have a tree of devices

each device node has some set of parents

which are just struct fields + a function that turns any device handle into the parent rc

through the power of codegen magic

#[device(traits = [InterruptController])]
pub struct GIC2 {
    parent: DeviceRc<fdt::Node>,
    #[prop(addr)]
    addr: usize,
    #[prop(addr)]
    addr_cpu: usize,
}
impl InterruptController for GIC2 {
    // logic
}
// setup
pub fn probe_gic_from_dt(node: &Device<fdt::Node>) -> Result<()> {
    // ...
    DeviceRc::new(GIC2 {
        parent: node.dup(),
        addr: gic_addr,
        addr_cpu: cpuif_addr,
    }).poke();
}

ish

i switched from rust mostly because of annoyances around async and ownership

then i also had this insanely cursed query macro which could perform regexes against the device tree ```rs
let gics = query!(> as driver::gic2::GIC2);

would not recommend but it was fun to implement

    let cpu = query!(>(type: "fdt-node" compatible: "riscv") as fdt::Node)
        .into_iter()
        .find(|n| {
            n.node
                .get("reg")
                .ok()
                .and_then(|h| h.u32())
                .map_or(false, |hartid| hartid == boothartid())
        })
        .ok_or(Status::PLATFORM_ERROR)?;
``` so you can do shit like this

                for idev in query!(self(irq: irq) as dyn Interrupt) {
                    if let Err(err) = idev.irq_deliver(irq) {
                        log::error!("plic: failed delivering irq {irq}: {err}");
                    } else {
                        return;
                    }
                }
``` or this

[marble salmon]

i think that's right, yes

however, the cost is that rust essentially takes the whole async function + all the async functions that it awaits as a generic parameter and monomorphizes it into a single object

[near tartan]

interesting

[marble salmon]

you can emulate that in C++ by putting everything in headers or by using LTO :^)

[near tartan]

i wonder how much dynamic linking affects all of this

[marble salmon]

for rust?

[near tartan]

yea

i have dynamic modules

[marble salmon]

probably not at all

Rust's futures are concrete types

whereas C++ coroutines are type erased

[hoary cave]

i'd assume that dynamic linking + generics = something similar to templates in headers or dynamic dispatch

[marble salmon]

C++ coroutines are the equivalent of Box<dyn Future>

with some extra magic to elide allocations

whereas in Rust, it's the executor's job to take an impl Future and store it somewhere

[near tartan]

i know that using generic functions across dylib boundaries causes code duplication (duh)

[marble salmon]

yeah that's expected i guess

[hoary cave]

so you can't dynamically link against them

yeah, expected

[near tartan]

swift can do it

[hoary cave]

just like templates in headers in c++

which makes sense

so can c#, kinda, i guess

or .net in general

[near tartan]

both have runtimes

i wonder if i should just not have dynamic modules for now

[hoary cave]

while it is cool, it is also kinda pointless

at this point in development

[marble salmon]

sometimes (e.g., in tokio) you see patterns such as fn poll_read(self: Pin<&mut Self>, ctx: &mut Context) -> Poll<usize>

[near tartan]

agree

it only gives me headaches in debugging

[marble salmon]

which allow you to "reconstruct" a Future implementation

while avoiding the monomorphization

:^)

[near tartan]

interesting

i really have no idea how rust async works i guess

[marble salmon]

https://docs.rs/tokio/latest/tokio/io/trait.AsyncRead.html#tymethod.poll_read

^ for example here

but that's ofc very ugly to write

[hoary cave]

is the Pin<&mut Self> what does the magic here?

honestly idk how that works or what the point is

[marble salmon]

well, this just allows you to create an object that implements future

[hoary cave]

i would need to take a good look at it

[marble salmon]

and then you can await it

[brisk totem]

swift does insane shit to make it work

[hoary cave]

and .net isn't even native, it's bytecode

though there's nativeaot

but idk how that works wrt generic across assemblies

[brisk totem]

nativeaot cant dynamically link

[hoary cave]

ah

[marble salmon]

you can use this convenience function

https://doc.rust-lang.org/beta/std/future/fn.poll_fn.html

to turn a poll function into a future

[brisk totem]

yeah

[hoary cave]

so all assemblies are just squashed into a single native executable?

[near tartan]

it also can't do runtime reflection

[brisk totem]

i think it can do some reflection

[near tartan]

i said runtime

[hoary cave]

it can

[brisk totem]

yes

unity's il2cpp can also do quite a lot of reflection

and its doing the same thing

[hoary cave]

il2cpp is shit tho

not really the same thing

[brisk totem]

its the same kind of thing

also whats the problem with it?

[near tartan]

idk if it changed, but in core 7 aot you could only do static reflection or the reflection methods would return null

[brisk totem]

i mean apart from closed

[hoary cave]

the codegen is absolute dogshit iirc

[marble salmon]

so this allows you to do async rust without the overhead of monomorphization but at the cost of making your code ugly af

:^)

[near tartan]

function > looks

[brisk totem]

also i hope you dont want async recursion

because rust has horrible perf there

[hoary cave]

and it's not really generating code instead of CIL, it's generating code from the CIL

[brisk totem]

yes duh

thats the good way to do it

[hoary cave]

lol no

[brisk totem]

yes

[marble salmon]

i bet async recursion needs some very ugly hacking in the compiler

[hoary cave]

okay

[near tartan]

idk if i even want async rust in the kernel to begin with

[brisk totem]

nope

[near tartan]

async io sounds funny

[brisk totem]

you just need Box<dyn Future>

or well not dyn

[marble salmon]

wait, it is just not allowed?

[brisk totem]

it is allowed

you just need to box the future lol

and yes that has about as much performance impact as youd expect

oh and because futures are polled at the root future, and not at the leaf future like in C++

performance is horrible

[marble salmon]

so i can't write

async fn foo() {
    bar.await;
}
asnyc fn bar() {
    foo.await;
}

?

[brisk totem]

you cant

[marble salmon]

lovely

[brisk totem]

but you can write rs async fn foo() { bar().await; } async fn bar() { Box::new(foo()).await; }

[marble salmon]

ngl that's ugly

[brisk totem]

yes

[marble salmon]

i mean, it's expected

but only if you know the implementation details lol

[brisk totem]

yeah its rust i dont know why you want good performance

or pretty code

[marble salmon]

so it's probably unexpected to most people who just try to write some simple async app

[brisk totem]

its not like you use rust specifically when you want to maximize performance

[lofty copper]

C++ also has all of this?

[lofty copper]

What's wrong with that, my kernel used rtti and exceptions

[modern hamlet]

borrow checker? lifetimes?

[lofty copper]

Move semantics

[modern hamlet]

they're not like rust

[brisk totem]

yes the C++ move semantics are garbage

[modern hamlet]

they essentially empty out an object leaving only a shell behind, to put the data in another object

[lofty copper]

it gets optimized away by the compiler anyway

[near tartan]

@hoary cave bruh rust macros are fucking goated

wdym i can just do this shit

that's cool af

[near tartan]

do i need atomics to read pointers? or is that guaranteed to be an atomic operation on x86_64?

for the task pointer in the percpu block

[hybrid island]

Br0 why wasn't I following teh Marvin thread

[near tartan]

:)

[hybrid island]

Well now I am

[near tartan]

gib ⭐

[hybrid island]

lol sure

[near tartan]

😃

[hybrid island]

Now I ask you to return the favour for Lily-CC :P

[near tartan]

i'm currently converting a bunch of "by convention" apis that i made into static traits

like the architecture specific stuff

[pine rock]

i belive you wont get torn reads for 8 byte reads as long as they're aligned to an 8 byte boundary, but you might not get the current actual value without using atomics

[idle flower]

how do you get thread local stuff working without std?

[near tartan]

it's the same macro definition

but with my per cpu logic behind it

[idle flower]

so just copy pasta mostly?

[near tartan]

mostly

[idle flower]

you copy .percpu section per core?

[near tartan]

yes

[idle flower]

sounds complicated

[near tartan]

nah

it's easy

you just accumulate all percpu data in a percpu section

[idle flower]

two symbols for start end of .percpu and then memcpy?

[cyan nexus]

why do you create a new interrupt handler table per cpu?

[near tartan]

then when calling it you do PERCPU_START + id * sizeof(.percpu)

i don't

[cyan nexus]

here i saw you do this

[near tartan]

do you mean irq or idt

oh that

yea you want that

[cyan nexus]

why?

[near tartan]

so you have more interrupts?

[cyan nexus]

you need more than 256?

[near tartan]

you never know

also you want to balance interrupts between cpus

[idle flower]

replace that with gs relative load and its free of static vars and supports non continous apic ids easily

[idle flower]

have you heard of nvme?

[cyan nexus]

how many does it use?

[near tartan]

that's what it does

just to setup

[idle flower]

with nvme each drive can have up to 64k of io queues and each of them should have its own msi-x vector by spec

[near tartan]

eh, no controller supports that really

like the CAP structure tells you how many queues you can actually make

but yes, in theory you could have 64k queues

[cyan nexus]

so 64k interrupts?

or what are queues?

[idle flower]

we all know hw devs, somewhere at some time a hw dev will yolo and allow 64k queues

[cyan nexus]

how do i balance interrupts btw?

[idle flower]

they are used for native command queueing, same as with sata just extended for flash memory

[cyan nexus]

like, rn i have a static handler vector, so in theory the cpu that receives the interrupt will execute that handler

but i have 256 max

[idle flower]

what is line 68 for exactly?

[near tartan]

self pointer

so i can return the structure

otherwise i'd have to read the msr

[idle flower]

so where does this come from?

couldnt you just give each hw thead its own gs value?

[near tartan]

what do you mean?

[idle flower]

gs:0 would return the base address of the gs segment, and when initializing the tls, set each threads gs to a different address to the base of its tls area that you allocated somewhere (heap, page,...)

[hoary cave]

bro what

[gs:0] literally does *(void*)(gs_base + 0)

and there is a pointer to itself

so instead of rdmsr gsbase you can do that

[spark surge]

it's faster

than reading the MSR

[idle flower]

also when defining multiple tls statics maybe you could get their offset in the tls area and use that gs:[offset]

at least of archs that have such relative address semantics for their tls register

[idle flower]

is your tls code already pushed?

[near tartan]

yes

check the latest branch

remove-talc

[idle flower]

it seems i can copy rusts default tls with only a single change

[near tartan]

yea

[idle flower]

like rusts full version

[near tartan]

how

[idle flower]

if im not wrong, LocalPointer stores/represents the the LocalKey (ptr) and its marked with a attribute which is used by the compiler to place it into the elf tls template section and stuff
https://github.com/rust-lang/rust/blob/master/library/std/src/sys/thread_local/native/mod.rs

like if my assumtion is right, all you have to do is replace LocalPointer with something that does gs relative accesses and the rest should just work out of the box

[near tartan]

i guess

it kinda does assume elf tls, right?

[idle flower]

well afaik #[thread_local] compiles down to elf tls

nothing else

[near tartan]

oh yea

but can you use that in a kernel context?

[idle flower]

no but you replace the entire LocalPointer implementation

[near tartan]

eh

i like my solution

[idle flower]

is it possible to have variables that only exist at compile time? like a atomic counter with fetch_add that only exist on const evaluation?

but its side effects persist?

[near tartan]

no

atomics are runtime

[idle flower]

is the const evaluation race free as in i can throw around a counter and it behaves like a atomic one without actually beeing atomic?

like the counter for index calculations of the tls fields?

[near tartan]

well tls only assumes a single cpu at all times

that's kind of the whole point

at least my tls does

[idle flower]

i mean at compile time

[lofty copper]

Isn't TLS supposed to be per kernel thread?

[near tartan]

per hart

[lofty copper]

Then it's CPU local storage, and you can still have race conditions?

[idle flower]

my idea was that LocalPointer contains a const eval only counter that increments each constructor call and passes the counter value with it so that it can be used as a index into the tls area

[near tartan]

who said that

[lofty copper]

I don't get it

[near tartan]

im confused

[lofty copper]

You load different gsbase per kernel thread, right?

[near tartan]

yes

[idle flower]

yes

[lofty copper]

Oh, then it's fine

[near tartan]

there shouldn't be any race conditions

[lofty copper]

I don't

[near tartan]

why?

is there any benefit

[lofty copper]

My kernel is nonpreemptive

[near tartan]

guh

[lofty copper]

So there is no typical context switching

[near tartan]

don't you want that in a microkernel

[lofty copper]

Idk

[near tartan]

like if a server hangs doesn't that just lock up the entire system lol

[lofty copper]

The kernel can't preempt itself

The userspace can be preempted of course

[lofty copper]

I mean the one that's used by kernel, not by userspace

Why is x86 so confusing?

[near tartan]

risc-v my beloved

no features = no complexity

[idle flower]

you mean as long as you dont support any extension?

[near tartan]

exactly

RV

but jokes aside, for risc-v i'll target RV64GC

[idle flower]

doesnt RV mean no atomics?

[near tartan]

RV means nothing

the most ocmmon impl is rv64gc

which is imafdc_zicsr_zihintpause_zifencei

[idle flower]

excuse me?

[near tartan]

i know that from the top of my head

integer, multiplication, atomics, float, double, compressed, CSR access instructions, PAUSE instruction and FENCE.I instruction

that's what GC is

G meaning general purpose

i will also support Vector though

[idle flower]

given that const eval is race free/single threaded i can replace all instances of #[thread_local] with a wrapper type that derefs to its inner value with gs:offset where offset is the current offset into the thread local storage area

[near tartan]

yes that's what i do pretty much

[idle flower]

and a counter is passed around and each time the wrapper type is constructed the counter is used to determine its offset

[near tartan]

i mean what i do is just take the offset of the static var relative to the start of the percpu section

[idle flower]

but that only works with a single struct?

[near tartan]

no

basically, PerCpuData has a dummy variable for storage

that lets the compiler know a specific address

i calculate the offset here

[lofty copper]

FP was kinda annoying because you've got to catch invalid instruction exception (or whatever is its name) and parse it instead of a separate thing on x86 to do lazy fp context switching

[lofty copper]

But PerCpu ≠ thread_local?

[near tartan]

i changed the name to per_cpu now

i could also call it cpu_local

[near tartan]

i think I'll do an actual prekernel

that can bootstrap the structures and pass control to the kernel

there are apparently some u-boot systems that only do linux protocol boot

and i want to be multiprotocol anyways

but that's for after the alloc PR

[eternal wharf]

use eir

[near tartan]

lol

i could make a rust eir

[idle flower]

eir?

[near tartan]

the managarm prekernel

[near tartan]

slowly getting back there

[hoary cave]

based flanterm user

[near tartan]

today is scheduler time

then vfs and we're back in business

[idle flower]

the allocator ub waiting to create race conditions

[near tartan]

yea

what allocator ub

[cyan nexus]

i just filled my VMM with spinlocks

lol

(and i managed to cause a deadlock bc i was returning before unlocking it, lol)

[idle flower]

adding spinlocks to prevent race conditions, causes race conditions by adding spinlocks

[cyan nexus]

XD

[near tartan]

@somber solar you mentioned the BFS algorithm before

do you know what kind of data structure that is?

intrusive linked list?

i can see that it's a weighted round robin basically

[somber solar]

description said a doubly linked list

[near tartan]

ah

yea i think i might implement bfs as scheduling algorithm

[somber solar]

I looked at it again and saw this

apparently O(n) for lookup and O(1) for insertion becomes O(1) and O(logn)

[near tartan]

hm

[somber solar]

Not recommended for 4096 cpus.

[near tartan]

lol

[near tartan]

hm

so either bfs or cfs

cfs doesn't seem that complicated

as long as you don't look at linux sources

[near tartan]

i have plans

[near tartan]

i think next week i'll have pcie and drivers running again

now that most of the allocation bullshit is done

uacpi score is up to 3.6mil

[somber solar]

@near tartan how's bfs going?

[near tartan]

i've been moving from spin to my own sync impls

[somber solar]

I want to implement something else other than round-robin I currently have

[somber solar]

hm?

[near tartan]

i'll probably not do bfs

but cfs

[somber solar]

bfs is supposed to be simpler

[near tartan]

eh

both are simple

with cfs there's less context switch overhead

[grave peak]

rust crate ig

[near tartan]

yes

just primitives like mutex, rwlock, etc

i'm rolling my own now

thank you clippy

[somber solar]

asm!("nop");

[near tartan]

not portable smh

i can have a hint probably

[somber solar]

all the sane architectures have it

[near tartan]

this is cool

[grave peak]

dont u need tons of unsafe for a kernel

[near tartan]

that's why i'm trying to keep it to a minimum

it's not a hard requirement, but a linter check

whenever you use unsafe you better give a good reason why

it's mostly for the low level bits

[ionic lava]

@near tartan you said, you have rk3588. Orange pi5 (plus)? Haven't you installed edk2 to it?

[near tartan]

not orange pi

i have a radxa 5b+

with edk2

@grave peak i need 182 unsafe's atm

and many of those are for uacpi and so on

[somber solar]

minimise the number of unsafes to make your kernel safer
put everything in one big unsafe

[ionic lava]

Ah, good boards. I had such - Rock Pi 4B. The very Tom Cubie gave it to me. Couldn't you please test my code on the board? It's a quick test. rk3588 is the only one SoC left untested and the most desirable to get this jump into kernel test.

https://ant-upptech.sourceforge.io/doc/Try.htm#downloads

It would work without display attached, but the kernel can't yet print into uart, so the success wouldn't be visible. So if you will test it, attaching a monitor is necessary. Thank you in advance. I'm not "pushing" 😁 just asking for a favor.

[near tartan]

if you can give me an image i can flash it

[grave peak]

Damn

[near tartan]

it's not too bad

most of the time it's willingly done

where i want to keep the semantic of something being "unsafe" at its core

[pine rock]

unsafe { cpuid() } cmon man

running cpuid and destroying the entire computer

[somber solar]

welp I made my scheduler fair (with just a b-tree and a vruntime thingy)

I don't know if it's completely fair, but it seems to be at least a bit fair

[ionic lava]

The archive contains 2 dirs: efi and Lybid, unpack it to any FAT partition on a storage device, you can connect to the board (USB stick, SD card, eMMC) so these 2 dirs are in the root dir of the partition, no need to format anything. This code doesn't write to any storage, just reads from these 2 dirs. Launch it via either UEFI shell fsN:\efi\Upptech\antload.efi or via UEFI Boot Manager's "Boot from file". If you can take the pic, please do it. Especially of interest is EL in which edk2 runs, since so far I tested edk2 only with OVMF, all real hardware tests were with uboot. 🙂

[lofty copper]

For some reason I couldn't get mainline Ubuntu to boot on it in DTB mode

[near tartan]

mainline opensuse tumbleweed works on it

ubuntu L

[near tartan]

i'll get to it on the weekend

[lofty copper]

I can try it as well...

[near tartan]

having a really hard time figuring out this sparse struct page shit

this is what i have so far but it seems wrong

[near tartan]

hell yes it works now

i think

[pine rock]

what are you storing in your page struct?

[near tartan]

buddy stuff

[pine rock]

ah the github may be a bit out of date

[near tartan]

ye

[pine rock]

i was going to say if its only flags and a refcount you can share page structs between multiple pages

[near tartan]

did you look at the main branch?

[pine rock]

ah github search only works on the main branch

actually even if you're storing buddy order only its probably worth using the same page struct for adjacent pages

[near tartan]

for now it's too much effort

i'll also be storing refcount

[pine rock]

alot of the time many adjacent pages will have the same refcount

if a subset is referenced or dereferenced you can split the page struct into segments

this is totally not incredibly painful to get right

[near tartan]

that sounds like mega double aids

[pine rock]

definetly not dont worry about it

[near tartan]

cursed

rust: make that 4000

[pine rock]

oh thats just the tests

the implementation is another 1500 sloc

[near tartan]

wtf

[pine rock]

on the bright side my pmm uses less than a page to track the entirety of the systems memory

which i think is pretty good with 384GB of ram

[near tartan]

sure bro

[pine rock]

i imagine the worst case memory complexity is higher if every process decided to map single 4k pages in a tight loop but for 95% of cases its much better

[pine rock]

only a spoonful

i really recommend having some nice formatting for common units

[near tartan]

more like it

[pine rock]

everytime i have to read some gnu tool output that contains 100237543kb my eyes glaze over

[near tartan]

ouch

that's not good

[pine rock]

you really want to write a massively complex page metadata tracker

you should definetly do it

[near tartan]

later maybe

but rn i need to get to userspace first 😭

[near tartan]

i need a link to that pmm

[pine rock]

my pmm?

[near tartan]

i wanna see

[pine rock]

https://github.com/apache-hb/BezOS/blob/master/sources/kernel/include/system/pmm.hpp
https://github.com/apache-hb/BezOS/blob/master/sources/kernel/src/system/pmm.cpp
https://github.com/apache-hb/BezOS/blob/master/sources/kernel/test/system/pmm.cpp

the vmm is implemented in a similar way but without refcounting

[near tartan]

thinking back, what exactly was wrong with using a bitmap as a pmm?

because my buddy impl is causing me brain damage

it was also faster than the buddy allocator

[vast lotus]

bitmap is more complex (if only slightly) and slower than the actual good option (freelist)

[cyan nexus]

why a buddy for pmm?

[near tartan]

hm

[cyan nexus]

freelist is basically the only good option for pmm

[near tartan]

i'll do a freelist then

:3

[cyan nexus]

i have a good implementation if you want

[near tartan]

sure, if it's okay for me to use

@cyan nexus do you have a link

[cyan nexus]

yeah

was waiting for it to load XD

[near tartan]

bruh

[cyan nexus]

https://git.voxelgames.eu/diamantino/horizonos/-/blob/main/src/kernel/kernel-common/src/memory/PhysicalMemory.hpp?ref_type=heads
https://git.voxelgames.eu/diamantino/horizonos/-/blob/main/src/kernel/kernel-common/src/memory/PhysicalMemory.cpp?ref_type=heads

my webserver is being slow

[near tartan]

wow, that's...

a lot simpler

[cyan nexus]

yeah

vmm has the mapping impl too

and the allocator uses a linked list

it can allocate any size

[cyan nexus]

i plan to add a small "defragmentation" to merge the free list entries too, but that's super simple

even if they don't use up any memory, but if i need multi page allocation it will be needed

[lofty copper]

Buddy can give you contiguous allocations

[cyan nexus]

yeah, but why on pmm

[near tartan]

dma

[lofty copper]

can't that be faster than freelist?

[cyan nexus]

where you alloc in 4kb chunks

[near tartan]

what if you need a framebuffer for the gpu?

[lofty copper]

Like doing one allocation for memory chunks for slabs or whatever

[cyan nexus]

freelist can too

[lofty copper]

My kernel uses some segregated fit weirdness, which is O(1) for allocations and frees, and gives contiguous memory

[vast lotus]

no matter what allocator you use allocating more than one page at a time is going to be slow during normal operation because to make it reliable you need to have a mechanism to defragment the physical address space

[lofty copper]

but you need to scan all of it, so it's very slow and impractical

[vast lotus]

relying on physical memory to not be too fragmented is a recipe for disaster

[lofty copper]

but you can fall back to multiple allocations

[near tartan]

bitmap is super fast until you reach the end of the memory region then it's slow

[lofty copper]

Buddy (or similar) naturally allocate smaller sizes first, so the memory kinda defragments itself unless it's almost full

[cyan nexus]

same as freelist

it is very fast when you don't have a lot of fragmented stuff

[lofty copper]

You don't rely on it, but use as an optimization

The problem with buddy is memory splitting, right?

[cyan nexus]

buddy can be slow sometimes

[lofty copper]

Like if you allocate 1 page out of 1MB chunk, you have to split it 8 times

[cyan nexus]

for me, i find that freelist is the best for pmm

no mem usage, very fast, and easy to implement / manage

[near tartan]

hm

@cyan nexus do you happen to know a little bit of rust

i'm pretty sure i've translated the algorithm correctly, but i keep getting really odd behavior

lol

oh wait don't tell me this is a fucking zeroing issue again

you don't happen to zero out physical memory, right?

this is what i do

[hoary cave]

bro wtf is that

[near tartan]

💀 bro dw

it's the translation of this thing

idk if i'm doing something wrong

i have triple checked it but i can't spot what's wrong

it tries to read garbage at one point

wtf 😭

[near tartan]

odd

the second it hits 0 pages remaining it corrupts

[near tartan]

it seems to be handing out the last page, even though it stores its metadata there

@cyan nexus did you double triple check that what you're doing is 100% correct?

or rather, with how much memory do you test your kernel?

because mine crashes with this qemu command line:

qemu-system-x86_64 \
    -m 2G \
    -serial stdio \
    -smp 4 \
    -no-reboot \
    -no-shutdown \
    -netdev user,id=net0 \
    -device virtio-net,disable-modern=on,netdev=net0 \
    -drive if=pflash,unit=0,format=raw,file=ovmf/ovmf-code-x86_64.fd,readonly=on \
    -device virtio-gpu \
    -cpu host,migratable=off \
    -accel kvm -machine q35,smm=off \
    -drive format=raw,file=menix.img,if=none,id=disk \
    -device nvme,serial=FAKE_SERIAL_ID,drive=disk

works with 4G

[cyan nexus]

I do know a bit

[cyan nexus]

It works on mine, and can alloc correctly

[near tartan]

how much are you allocating in qemu?

[cyan nexus]

8G

[near tartan]

because on 2G memory it panics

[cyan nexus]

What does it say?

[near tartan]

well it overwrites memory

and as far as i understand it, retAddress is not updated here

so it still points to the last entry you just said is free

which then gets overwritten, thus losing the metadata

[cyan nexus]

Hmm

[near tartan]

what qemu do you have?

maybe the same happens for you on 2G mem

because that causes a small region at the end of memory to spawn

or rather, can you please test if you ever hit cur_entry->count == 1 when allocating?

[cyan nexus]

I do

It happens when uACPI frees that mem

So i have 1 count entries

[near tartan]

hmm

[cyan nexus]

Hmm, wait

Maybe no

I think you are right

[near tartan]

can you unhook the free function?

[cyan nexus]

i think that is a bug

[near tartan]

lol

[cyan nexus]

thx for showing me that XD

[near tartan]

trying to think about what is supposed to happen

ret_address should be updated to something different

[cyan nexus]

not retAddress

that should stay like that

[near tartan]

oh yea true

[cyan nexus]

it's this->listPtr that should change

prob setting it to the prev entry if it exists

else to the next

something like this should work

[hoary cave]

what the hell is this even supposed to be

[near tartan]

pmm

i couldn't get buddy to work

i want to get it working first, then i'll rustify it

[hoary cave]

that is literally so broken but okay

also this code seems so overcomplicated

idk why ur copying someone elses code

[cyan nexus]

wdym so broken?

[hoary cave]

instead of writing ur own

it's a freelist man 💀

[hoary cave]

copying c/c++ into rust without writing it as idiomatic rust

[cyan nexus]

does this fix that bug btw?

[hoary cave]

because he will have to rewrite it entirely anyway

[cyan nexus]

ah yes

[near tartan]

are you going to kill me if i tell you something

[hoary cave]

probably

[near tartan]

i've never looked at how a freelist works

[cyan nexus]

XD

[near tartan]

https://tenor.com/view/guy-arguing-guy-talking-to-wall-talking-brick-wall-gif-18667615

[somber solar]

it's just a linked list

[hoary cave]

linked list of free entries

free-list

the linked list is a detail

[near tartan]

yea but my brain always shuts down when i try to read any impl of it

[hoary cave]

why read an implementation

do it urself

[near tartan]

idk what it's supposed to do, like what's the tldr of its operation

[hoary cave]

to keep a list of free entries bro

[near tartan]

bro i get that

[hoary cave]

what else do you need 💀

u want me to draw it in paint for u?

[near tartan]

yea :3

[cyan nexus]

well, it should point to free mem entries, and say how many pages they have

when you alloc it, the entry gets overwritten

so no memory is actually used

[near tartan]

so how do you free shit? hook an entry between it?

[cyan nexus]

just set that page to all 0

and add it back to the list

[hoary cave]

you just append it to the list

or prepend if you're usinbg a singly linekd list

[cyan nexus]

(set the count to how many pages you freed tho)

[grave peak]

marvin most complicated alloc pages speedrun

[near tartan]

:(

[grave peak]

i swear linuxes impl is simpler

[near tartan]

what does linux do

[cyan nexus]

mine is pretty simple too

just had a bug that somehow never happened on my OS

idk how

[near tartan]

okay im doing it from scratch

should've done that in the first place

[cyan nexus]

XD

[near tartan]

i feel genuinely retarded

brain is numb

[cyan nexus]

it will help you understand it better

[somber solar]

struct page { page *next = nullptr; }
page head;
// init
page *cur;
for (pgaddr in memmap_pages)
{
    pgaddr += hhdm;
    if head.next == nullptr
    {
        head.next = pgaddr;
        cur = head.next;
    }
    else
    {
        cur.next = pgaddr;
        cur = cur.next;
    }
}
// alloc
ret = head.next - hhdm;
head.next = head.next.next;
return ret

I haven't written a freelist allocator before but probably it's something like this

[cyan nexus]

prob store the amount of free pages in the entry too

[somber solar]

one entry == one page

[cyan nexus]

so you can have less of them

[cyan nexus]

if you set the count into an entry, it will go faster

and easier when allocating multiple pages

[somber solar]

you generally don't need to allocate multiple pages

unless you're doing something wrong or dma

[near tartan]

i do

[somber solar]

for what?

[near tartan]

dma

[cyan nexus]

i just made mine for that, just in case

[somber solar]

then why are you doing a freelist?

do buddy

or even bitmap

freelist with multiple page allocation support is unnecessarily overcomplicated

[cyan nexus]

no?

[somber solar]

bro saying no doesn't invalidate someone's statement

[cyan nexus]

it's not that diff from the single entry one tho

[somber solar]

do bitmap if you want simple
do buddy if you want good

simple as

'nuff said

[near tartan]

because different people are telling me different things 💀

[somber solar]

who is telling you what?

apart from the diamond guy

[near tartan]

monkuous

[somber solar]

what did he say?

[near tartan]

to use a freelist as pmm

[somber solar]

I'd rank freelist right after bump

[near tartan]

.

[somber solar]

bitmap isn't more complex than a multiple page freelist

[cyan nexus]

but it uses memory

[near tartan]

i dont really mind that#

[somber solar]

it uses (mem_size / (0x1000 * 8)) bytes more

I think

[hoary cave]

free list is good

fym

[somber solar]

I dislike it

[hoary cave]

it's fast, and you don't need more lol

[grave peak]

a bitmap is way worse tho

[hoary cave]

bitmap is slow by default

[grave peak]

it gets O(N) complexity

a freelist is instant

[hoary cave]

u can make it faster but it's always gonna be slower than a freelist

[cyan nexus]

buddy is also slower

[hoary cave]

freelist is literally unlinking a page from a list

but buddy has nice properties

[somber solar]

I don't doubt that

[hoary cave]

that tend to outweigh the performance

[cyan nexus]

so yeah, you decide if you want speed or buddy

[somber solar]

-# (do buddy)

[cyan nexus]

i am still surprised that the bug i had did not nuke my memory smh

[somber solar]

-# (I'm going to switch to buddy as well)

wikipedia page for buddy allocator sucks

[cyan nexus]

i found a good one

[near tartan]

then i need to redo buddy allocation because the impl i have right now is not coping with a single struct page region

[cyan nexus]

(maybe can help if you want)

[somber solar]

send it

[hoary cave]

pseudo rust free list https://gist.github.com/48cf/8928ada29766f6853e6f4e1ce0434e3e

[cyan nexus]

https://opendsa-server.cs.vt.edu/ODSA/Books/CS3/html/Buddy.html

[near tartan]

tbh i just want to have working allocations again 😭 this shit is taking up so much of my time

[cyan nexus]

found this nice visualization

[somber solar]

I already know how it works in theory 😭

[near tartan]

i dont

[somber solar]

I'm missing implementation details

[cyan nexus]

i searched too, but couldn't find a lot of good stuff on it lol

i just went for a linked list for my heap alloc

that can alloc the size i want

[somber solar]

you have multiple freelists for each 0x1000*2^n sizes
when you run out of one size, you take one page from n+1 and split it into two, allocate one, put the other in n freelist
when you free, you just put it back into those lists

[hoary cave]

if you want to take a look at my implementation i can link it

it's pretty simple

[somber solar]

but where would I store metadata