Keyronex

[hexed solstice]

edk2 goes nowhere under qemu -enable-kvm

[sinful jetty]

This is normal, sometimes the code gets corrupted

Get a new .fd file

[hexed solstice]

it's the same .fd as works fine here

[sinful jetty]

Ah

[hexed solstice]

and the one in the debian qemu-efi-aarch64 or w/e its called package also

[hexed solstice]

this is not normal btw

you shouldn't have to hope you get lucky with your copy of an essential to the stack

[sinful jetty]

As in its a common bug

[hexed solstice]

i do get what you mean, i think it's just frankly disgraceful and an appalling indictment on the state of the much-vaunted alternative arches

[sinful jetty]

True

[hexed solstice]

now that uacpi_map_memory or whatever it's called actually maps stuff uncached instead of giving HHDM pointers, in any case, keyronex on pi 4 gets straight to the furthest point it can, without delay

currently i'm trying to chase a bug that seems to pop up occasionally under qemu (no kvm) though

[hexed solstice]

looks like mis-managed interrupt priorities

[sinful jetty]

That's based

[hexed solstice]

amd64 really spoils you with its %cr8

the gicv2 does provide a (memory mapped) priority register, i don't know whether they have something like amd64's vTPR which lets VMs avoid a vmexit in the happy case, and in any case i don't get to enjoy the amd64 simplicity - i can't have splx and splraise just write to the gicc priority register, it's not even mapped early in boot

[neat steppe]

Why not

[hexed solstice]

it's not mapped (or rather it's not mapped with appropriate caching attributes) and my code is written for m68k and amd64s so spl calls are everywhere from the very moment things start running

what i can still do is lazy ipl management with the gicv2's priority register

since there should be no IRQ coming in until i've already mapped the GICC registers

[neat steppe]

This is one reason I find it very convenient to have a loader to map stuff in and you could probably do something similar with some subsystem that runs at boot and never again that's capable of allocating physical memory by chopping off pieces of the physical memory map

[hexed solstice]

undoubtedly it saves a lot of grief

[neat steppe]

I do tons of shit in my loader to ensure a sane environment from the first code that runs

On amd64 I'll probably chainload it with limine or something

It allocates the processor blocks, idle thread stacks, PFN database and other stuff so it's Just There when Ke starts up

I'm actually unsure how you'd do it otherwise lol

You kinda need something with a trivial physical memory allocator for that stuff

How do you do it

[hexed solstice]

the higher-half direct map breaks various cyclic dependencies

[neat steppe]

But you still need to carve out some physical memory

[final hatch]

I do this

subsystem that runs at boot and then never again that's capable of allocating physical memory by chopping off pieces of the physical memory map

[hexed solstice]

vmem is also equipped to support allocating virtual memory without a direct map, but as an optimisation, i just allocate single pages of kernel wired memory frmo the direct map

[final hatch]

I use it to initialize the PFN database only though

[hexed solstice]

the first pages of each contiguous region of free main memory are used to store the pfndb

[neat steppe]

I like virtually sparse pfn databases so you can just index it by pfn without wasting any memory in case of gaps

[final hatch]

same

[hexed solstice]

(and how does vmem do it? it includes a miniature slab-like allocator, and it's pre-initialised with a few dozen boundary tags. it refills these before most operations, but there are enough that it works to allocate virtual address space for a while before it has to start refilling them)

[hexed solstice]

i do too, it will arrive in keyronex some day

[hexed solstice]

still plagued by this bug

i haven't figured it out

it's reproducible non-SMP, thankfully. that makes things easier to figure out, if i'm lucky

[vale pelican]

strange, i've had it working when testing limine

[hexed solstice]

anything noteworthy about your environment?

i tried it straight on, what do you call it, the debian that's designed for the pi

[valid sequoia]

can you steal the limine makefile test

that has kvm on arm too no?

[hexed solstice]

and i tried out qemu 9 and what-you-may-call-it as well

a known working edk2

[hexed solstice]

i'll give it a try shortly

trying to figure out why somewhere i'm trying to write past the start of the stack

unfortunately adding the printfs i want inhibits the problem

[vale pelican]

i tried it on aarch64 void linux on the pi4

the only thing really noteworthy was that the rootfs was mounted over nfs

[hexed solstice]

i'll retry later with the limine skeleton thing

as i have learnt the tiniest differences in qemu invocations make all the difference for meme arches

right now i am losing my mind trying to figure out why sometimes a trap handlers tries to write past the end of the stack

(not very far past!)

at least it doesn't look very far past

oh, this time it happened while loading the values back off the stack

[vale pelican]

one thing i can think of is forgetting -cpu host or at least -cpu <some-aarch64-core>

because the default for -M virt is cortex-a9 or something, which is 32-bit

even for qemu-system-aarch64 for some reason

[hexed solstice]

i tried that

it rejected -cpu almost anything other than host

but i'll pick this up again after i've solved my current bug

i am undone, i can't believe for hours i have been defeated by what should be a trivial fix

[valid sequoia]

if it has symbols you could check up where it hangs lol

or if screen is broken somehow

[hexed solstice]

not that, i don't care much about it, i haven't touched it in hours

it's this bizarre thing going on where seemingly sp ends up with an absurd value

[hexed solstice]

well, try as i might, i haven't been able to replicate it in about 25 runs

you know what

ban me from osdev forever

i am a clown

i've spent the last 8 hours looking furiously at specfically the irq handler logic

the right place to look was the sync exception handler, where interrupts were re-enabled and not turned off again before exit

[hexed solstice]

as to the not working qemu kvm, no luck with the limine template's command line either

neither with nor without KVM

when i can be bothered, i will build my own qemu from trunk and try out some random edk2s

[hexed solstice]

i've got it going with qemu built from trunk

it's quite slow

the limine smp response reported 1 cpu when truly there are 2

seems to report 1 regardless of how many

oh, hold on, given 4, it reports 2

but with -cpu max instead of -cpu host it seems the limine response is as expected

[hollow hull]

@vale pelican ^

[vale pelican]

hmmm

let me try it on my pi

[hexed solstice]

it could be the edk2 to blame

i tried another and got expected behaviour

[vale pelican]

lol i forgot i set the limine timeout to 999999 and was wondering why my pi hasn't booted yet

[hexed solstice]

it really is a game of russian roulette with uefi on aarch64

[vale pelican]

god bless at least arch linux arm ships with a working ntp daemon set up in the base install

[hexed solstice]

do you know what, i'm not sure it is the -cpu option affecting the outcome of limine smp response

i swear it's just randomly varying on me now

but it's hard to tell as it takes a minute to get anywhere

[hexed solstice]

incredible, nothing happens until i go into the qemu compat monitor and start typing

that's why it alawys takes 5 minutes

[vale pelican]

i've been waiting for like 30 minutes for pacman -Syu to complete so i can install qemu

i have no clue what it's doing but archlinuxarm-keyring is taking forever to update, and gpg is not even using any cpu?

perhaps i should use a faster storage device than this shitty usb stick

[hexed solstice]

in the meantime i am trying to figure out one of those bloody synchronous exceptions of "Unknown reason"

[vale pelican]

iirc esr=0 is just invalid instruction?

[hexed solstice]

ESR=0x2000000

[vale pelican]

er i mean ec=0

which is the case

the only bit set says it's a 32-bit instruction (which literally means the instruction takes 4 bytes in memory, not that it's 32-bit arm)

[hexed solstice]

the instruction appears valid

[vale pelican]

what is it dying on?

[vale pelican]

nice website to have btw https://esr.arm64.dev/#0x2000000

[hexed solstice]

oh no, now it's doing this thing where it grinds to the worst slow execution imagination

1 character printing a second

[vale pelican]

wtf

[hexed solstice]

0x40008000: str xzr, [sp, #184] <- sp is fine and 16-byte aligned

[hexed solstice]

i have no idea

usually you don't have to go into the compatibility monitor and type info registers to make qemu work, but on this bloody thing, you do

everything is completely fucked up

[vale pelican]

is this with kvm or tcg?

[hexed solstice]

kvm!

[vale pelican]

hmm

[hexed solstice]

tcg totally fine

well, except for a random 9p failure

anyway i have no idea why it does this slow printing sometimes

it pops up at random times

never mind, the slow printing also happens under tcg - which is helpful to know

obviously something is very, very wrong with this qemu

bizarrely it happens with the stock debian qemu on the pi too

i have no idea why

[hexed solstice]

alright

no more slow printing with -display spice-app - fine, i can deal with that

as i suspected my ESR=0x2000000 appears to be to do with keeping the icache in order after demand page-ins

[hexed solstice]

$4 = {revision = 0, flags = 0, bsp_mpidr = 0, cpu_count = 3, cpus = 0xffff000041b5a000}
lmao, this time we got 3 described by limine

should be 4

[vale pelican]

wtf

[hollow hull]

kinda funny

it worked flawlessly on SDXE

[hexed solstice]

apart from i can't test smp yet due to this oddity, things seem to work in kvm now mostly

with the exception of what i suspect is some oversight in dealing with the icache

occasionally a stray data abort with esr 0x2000000

[hexed solstice]

wtf

i can flush and invalidate the entire dcache and invalidate the entire icache and still encounter this

perhaps it's not stale icache after all?

@vale pelican is managarm perfectly reliable on aarch64?

[earnest salmon]

the real question is:
is managarm perfectly reliable at all

[violet tinsel]

assertOS

[sinful jetty]

Yes until it assets

[hexed solstice]

i am stumped

esr 0x2000000 with far that bears no semblance to any address that the instruction at elr should be touching - it sounds like it should be stale icache

but i've went to such efforts as invalidating the entirety of it everywhere to see if i can get things to work - they don't

[vale pelican]

i haven't really had strange issues like you've been having

i did have nonsensical exceptions with ec=0 happening constantly but that was solved by adding icache invalidations

[hexed solstice]

i have thrown the bastards in everywhere

the only thing is i do it through the direct map. whch should be no problem as debian says on boot "[ 0.000000] Detected PIPT I-cache on CPU0"

i don't get it

i really just don't get it

el0 sync: frame 0xffff120000393e40, elr 0x416491ec, far 0x4024b000, spsr 0x80000000, esr 0x2000000; esr->ec = 0x0

(gdb) x/i frame->elr
0x416491ec <_ZN3frg9slab_poolI16VirtualAllocator13FutexLockImplILb0EEE8freelistC2Ev+12>: str xzr, [x0]
(gdb) x/x frame->x0
0x4024b000: 0x00000000

sometimes things look like this, with the FAR appearing to be an address the instruction will actually be modifying, other times FAR is the IP itself

there's only one thing for it, let's see what happens if i turn off the icache altogether

well, it's extremely slow, unsurprisingly

i note i occasionally get a spurious page fault for writing to a page when in reality it's A) mapped writeably and B) the TLB was invalidated on making it writeable, and appropriate barriers inserted after doing so

and unsurprisingly, i cannot replicate the esr = 0x2000000 exception with icache disabled

i just don't get it, i've even gone to the length of fucking the dcache and icache into the bin at every page fault

[hexed solstice]

@vale pelican for that matter, where does managarm maintain the icache?

not a single ic (nor dc for that matter) in the repo

[vale pelican]

its on a branch on my fork

let me find the commit

[hexed solstice]

i could do worse than to compare to something that should be working

though i've already replaced my own cache maintenance code with openbsd's to see if it made a difference

[vale pelican]

https://github.com/qookei/managarm/commit/c65fb0a76d0da0f202738a22bdb6b101984e9df3

this was enough to at least get it working

[hexed solstice]

cheers, no such luck for me

it's not even explainable by e.g. linux lying to me about PIPT icache

as i've gone to the extreme of throwing ic iallu in there instead

that's the frustrating thing

and i've only had limited time these past few nights, it sucks wasting time on this shit

this is a second night i've wasted what little time i have to myself on this

i've thrown in invalidation of both by vaddr too, unconditionally after every page fault

[hexed solstice]

i'm done, i've wasted enough time on this tonight

aarch64 has completely defeated me

i simply cannot figure this out

[hexed solstice]

back now. i am just going to tidy up everything and make a commit with what i've done

[hexed solstice]

https://github.com/Keyronex/Keyronex/commit/3e57118265b664c0947ff8696651a1abe9f0c01d

that's it then

still doesn't work reliably

i see esr=0x2000000 exceptions regularly still on userland execution, sometimes odd delays in execution in new threads

as i am too skill issued with aarch64 to figure out what's going on (i assume some icache problem, but i'm honestly at a total loss as to what, as i am still having this problem even if i insert ic iallu after cleaning and invalidating the dcache in the needful places. it shouldn't matter anyway. this code is not yet equipped to deal with VIVT or VIPT icache, but linux reports my pi 4 has a PIPT icache, so there should be no problem with ivau on a direct map mapping), i might start work on a riscv64 port soon

[neat steppe]

you have either:

1. missed a spot
2. it is not an icache problem after all

probably #2

[hexed solstice]

i am beginning to wonder that myself

it's not as though it's totally black magic to run on aarch64

plenty of other kernels do it

osv had same pattern of problem (see https://github.com/cloudius-systems/osv/issues/1100) and solved it there with icache maintenance they hadn't been doing before

and that's exactly what you would expect from esr=0x2000000, it is usually indicating some junk in the icache

there has to be a skill issue on my part involved somewhere, linux is running fine on my pi so it's not posessed by demons, and even many hobby kernels seem to run fine on aarch64

[hexed solstice]

after several tries i cannot replicate this if i make sure that the process never sees a page replaced

i do still see odd delays before 2nd and 3rd threads run

but that appears to be unrelated

i am not trimming the standby list so it should be the same physical pages getting used

yeah, problem swiftly returns when i am allowing replacement again

[neat steppe]

TLB?

prob not cuz its just the same physical page again if youre not reclaiming from standby

so if you had a TLB invalidation bug itd be asymptomatic

[hexed solstice]

the mystery remains

now at least i feel slightly better about it though

it may be that keyronex deliberately churning local page replacement could explain why other hobby kernels, even ones which seem to involve less barriers and cache invalidations than mine, don't die on userland code execution so often

[neat steppe]

yeah id be shocked given the standards of most hobby kernel development if they actually explicitly tested their cache invalidation code

you may have to look at linux

[hexed solstice]

i have had a look at openbsd and i seem to be doing similarly to them

i haven't looked overly closely yet

[hexed solstice]

i'll leave it for now and hopefully something clever comes to me

tomorrow if i feel keen i will start the riscv port

[hexed solstice]

it's tomorrow and the aarch64 problem is still a bee in my bonnet

but i still haven't the foggiest idea on what to do about it

[valid sequoia]

what else could you do to at least work around it? flush the whole tlb and reload ttbr at every pagetable modification?

[hexed solstice]

well, funny you say this, because i just tried commenting out ic ivau and i observe absolutely no change in the pattern of the problem

or at least i don't observe much change

[valid sequoia]

esr=0x200000 is odd too, as that's an unknown exception?

[hexed solstice]

apparently it is commonly observed when there's a junk instruction

i have not been able to find any reliable workaround yet

apart from turning off the instruction cache altogether, but maybe i just didn't test that enough times (takes too long)

[valid sequoia]

looking at ELR, the instruction is just random right?

[hexed solstice]

stripping out all cache maintenance does at least cause the problem to occur much more quickly

but turning commenting out the entire icache synchronise routine, that appears to have nil or negligible effect

[valid sequoia]

at this point i wonder if it's not something stupid, have you checked that the cache line size reported checks out?

[hexed solstice]

it tends to happen at particular PCs, the instructions look sound

sometimes FAR is even loaded with the address i would expect the instruction to be touching

[valid sequoia]

i guess they're just the first instruction on a freshly paged in page, as expected

[hexed solstice]

sometimes however it's loaded with a "reasonable-looking" address i would not expect to be modified

(gdb) x/i frame->elr
   0x416491ec <_ZN3frg9slab_poolI16VirtualAllocator13FutexLockImplILb0EEE8freelistC2Ev+12>:     str     xzr, [x0]
(gdb) x/x frame->x0
0x4154c000:     0x00000000
(gdb) x/x frame->far
0x4154c000:     0x00000000
``` here's one i often see

[valid sequoia]

@hexed solstice i think i have a lead

dcache_line() and icache_line()

[hexed solstice]

good spot, let's see if it works if i fix those

[valid sequoia]

words is 32bit btw

so total would be
dcache_bytes = 4 << ((ctrl_el0 >> 16) & 0xF)

and likewise for icache

[hexed solstice]

i've tried but unfortunately no change

[neat steppe]

is there any chance any of these things are implemented in assembly language

if you improperly raise the stack pointer more than the red zone distance above active stack frame data it could be getting corrupted randomly by exception handlers

iirc thats what the red zone is. i dont do anything with anything that has one

[valid sequoia]

sysv has no redzone on aarch64 iirc

[hexed solstice]

all c++, and no redzone in the aarch64 sysv abi

[neat steppe]

this still applies even if theres no redzone but if its c++ its not likely to be fucking up the stack

unless the abi youre using is assuming a redzone and you arent allowing for one

might be worth disassembling it and checking what its doing with the stack

just to rule that out

thats my contribution because the only time i had a bug kind of like this where there were bizarre inexplicable crashes happening at random it ended up being certain stack frames getting overwritten by exception handlers

i only found it because i was looking through some old usenet posts from like 1990 and i found someone complaining about an identical bug in an i860 C compiler and realized thats what it was for me as well

after like a month of looking

[hexed solstice]

i miscoded the exception handler the first time and that caused a bit of fun

but thank god it was easy to figure out

[neat steppe]

whenever i had cache related bugs they were pretty easily resolved within like an hour or two

they didnt fuck me over for this long

and my caches require more management than these ones do

[hexed solstice]

as whenever i had problems i noted it was always consistent with some register's value not being what it ought to be at a given point in the execution of some code

[hexed solstice]

yes, it's embarassing

[neat steppe]

im not saying youre worse and thats why its fucking you over

im saying the fact its fucking you over might be indicative of this not being what that is

a cache invalidation bug

if it was i think you would have resolved it within an hour or two of clocking it as one

[hexed solstice]

given that omitting icache invalidation entirely seems to have minimal if any impact i don't rule it out that it might be something else

[valid sequoia]

i'm thinking what else could it mean that the issue disappears with icache off

[neat steppe]

okay but like you arent doing any physical page replacement for this right

[hexed solstice]

that's the bizarre part. but i can always test that a few more times and make certain of it

[valid sequoia]

considering how crippled aarch64 becomes without caches

[neat steppe]

only invalidating and re-validating ptes

for the same physical pages

if so then this shouldnt be causing cache bugs

right

[hexed solstice]

exactly that

[neat steppe]

if you had a cache bug itd be asymptomatic if youve never executed from these pages before

and you only write that one instruction sequence to it

[hexed solstice]

and that's the thing, ic iallu (which invalidates the entirety of the icache) makes no difference

[neat steppe]

is it possible its really a dcache thing due to assuming some type of coherency that doesnt exist

with dma or something

[hexed solstice]

i spent a while wondering whether i was doing something inappropriate in that i would invalidate against a direct mapped page, which i expected should be fine, since linux reports the pi icache is PIPT

but just in case, i just threw in invalidation of the entire icache - to no avail

[valid sequoia]

what's the difference between vmalle1is and vmalle1?

[hexed solstice]

i wondered this so i pessimistically clean and invalidate the relevant ranges immediately prior to and after dma

[neat steppe]

people were doing coherency with dma since the early 90s so id be shocked if a pi cpu wasnt doing it

[valid sequoia]

it doesnt

[neat steppe]

but its a thing i know exists

[valid sequoia]

but qemu ought to be coherent

[neat steppe]

odd

[valid sequoia]

it's not a cpu issue, it's a chipset issue iirc

[neat steppe]

it can be a cpu issue

[valid sequoia]

so the arm internal l1 is designed to be coherent

but the broadcom shared l2 isnt

[hexed solstice]

vmalle1is applies to inner shareable pages

[neat steppe]

if the cpu simply doesnt have a way to interface with the dcache to perform that sort of coherency work then its a cpu thing

[valid sequoia]

it does

but broadcom didnt do it with pci

[neat steppe]

and this was the case for like 80s mips and some other things

but by the 90s everything workstation class i know of was doing coherency with dma

and 90s workstation class is like 10% of 2024 raspberry pi class lol

[hexed solstice]

pi is far from workstation class

[valid sequoia]

you think too high of a 10$ set top box chip from 2008

[hexed solstice]

riscs might have once been cool workstations and servers but now they are weird proprietary embedded socs

the days of sparcstations and alphastations are over

[neat steppe]

thats like saying "the days of shit in a shoebox and platinum"

sparcstations sucked.....

they were the lowest cost entry-level workstation for a reason

[hexed solstice]

i never had one but i had a sunblade 2000 and that was really nice

[neat steppe]

probably the 90s risc platform closest to the PC in performance

still better but not by much

[neat steppe]

well yeah but thats a server

[hexed solstice]

workstation

this bad boy

[neat steppe]

weird

i assumed blade referred to this concept https://en.wikipedia.org/wiki/Blade_server

apparently for that system its unrelated and they just called it blade because it was a cool word

more confusingly there were later "sun blade" systems which actually were blade servers

[hexed solstice]

very interesting thing

it's very particular PCs that the exception is almost always at

(gdb) x/i frame->elr
   0x416491ec <_ZN3frg9slab_poolI16VirtualAllocator13FutexLockImplILb0EEE8freelistC2Ev+12>:     str     xzr, [x0]
(gdb) x/x frame->x0
0x414db000:     0x00000000
(gdb) disassemble _ZN3frg9slab_poolI16VirtualAllocator13FutexLockImplILb0EEE8freelistC2Ev
Dump of assembler code for function _ZN3frg9slab_poolI16VirtualAllocator13FutexLockImplILb0EEE8freelistC2Ev:
   0x000000004000fbac <+0>:     sub     sp, sp, #0x10
   0x000000004000fbb0 <+4>:     str     x0, [sp, #8]
   0x000000004000fbb4 <+8>:     ldr     x0, [sp, #8]
   0x000000004000fbb8 <+12>:    str     xzr, [x0]
   0x000000004000fbbc <+16>:    nop
   0x000000004000fbc0 <+20>:    add     sp, sp, #0x10
   0x000000004000fbc4 <+24>:    ret

this one in particular

[valid sequoia]

hmm why the nop after?

-O0 compilation artifact?

[hexed solstice]

the page containing that was faulted in a little time ago prior to the exception too, so it looks like it was already executing in it

[valid sequoia]

hmmm, 0x1ec

i am thinking that somehow it's something unexpected too, like only invalidating the first cacheline

hmm what if you return from the exception? what happens

[hexed solstice]

what a fun idea

[valid sequoia]

interrrupt handling has plenty of barriers so it might have one of the working ones

but it might fail if the state gets corrupted before, and the elr is just of a funky instruction that causes the issue to manifest

[hexed solstice]

returning, things seem to proceed just fine

[valid sequoia]

that's even more confusing

[hexed solstice]

it doesn't seem to do anything weird

other than that sometimes it takes a while for a new thread to get to printing its hello

it's not repeatedly producing esr=0x2000000 exceptions as i print out whenever that happens

[valid sequoia]

printing would trap into qemu and then then cache gets completely replaced by sheer amount of code churn

[hexed solstice]

let's see if i observe any other slowdowns if i don't print and simply return

the delays before my new threads print anything, that has me suspicious, but if there's a bogus in the icache issue i would really be expecting to observe funny behaviour

mlibc is sensitive and something should break

aha

(gdb) print badesr_counter
$1 = 685

so i increment this guy every time i see an ESR = 0x2000000 and return

[valid sequoia]

great, we went from "it doesn't work, i don't know why" to "it works, i don't know why"

[hexed solstice]

it looks to be on a few, small numbers of addresses

[valid sequoia]

https://lore.kernel.org/linux-arm-kernel/[email protected]/ this looks related?

osv mentions that their error handling was insufficient and that's why the exception was appaearing as unknown?

[hexed solstice]

After very long and painful research, I think I am 99% confident what the root cause is. For more details please read this question and related replies I posted on the ARM forum.

In short, I ended up modifying the OSv page fault handler to clean data and invalidate instruction caches for the virtual memory area of the handled page right after adding a new page tables entry and filling the page with the content of the ELF file. More specifically I ended up calling the GCC __clear_cache built-in which executes dc cvau, <reg> and ic ivau, <reg> for each D- and I-cache line within the page. As I understand these instructions clean data and invalidate instruction caches by virtual address to the Point of Unification. This needs to happen before OSv dynamic linker can start executing application code loaded and mmap-ped into memory otherwise sometimes the instruction cache may have invalid instructions entries causing "Unknown Reason" (EC=0) synchronous exceptions.

this is the osv bit i read

so what i have right now then is that the bad ESR will happen several hundred + times before it's resolved, and these are all appearing to be on the same instruction or on a few instructions

so i can experiment with adding various invalidations and barriers in the esr = 0x2000000 case and see what makes the number of unknown ESRs fall from hundreds to a few

ic iallu; dsb sy; isb doesn't do anything

let's try tlbi vmalle1

[valid sequoia]

the only thing left of barriers

is full icache and/or dcache invalidation

[hexed solstice]

yes, i am actually curious about the dcache here

because i'm trying this with virtio

so qemu is emulating it

[valid sequoia]

qemu has coherent DMA

so the resultw ill comfortably be in dcache

but might not have reached ram, so you need to clean-and-invalidate instead of invalidate only?

[hexed solstice]

this is what i was thinking

let me have a quick look at what i do post-page-readin

oh never mind

    size_t dcline = dcache_line(), icline = icache_line();

    for (vaddr_t it = ROUNDDOWN(base, dcache_line()); it <= limit;
         it += dcline)
        asm volatile("dc cvau, %0\n\t" ::"r"(it) : "memory");

    asm volatile("dsb ish\n\t" ::: "memory");

    for (vaddr_t it = ROUNDDOWN(base, icline); it  <= limit; it += icline)
        asm volatile("ic ivau, %0\n\t" ::"r"(it) : "memory");

    asm volatile("dsb ish\n\t"
             "isb\n\t");

that's what i do already to synchronise the icache with the dcache

i have found something that reduces the number of unknown ESRs from hundreds+ to 1

tlbi vmalle1 when the unknown ESR is encountered

[valid sequoia]

now what if you put the tlbi vmalle1 at the end of the post pagein path

[valid sequoia]

TLB shenenigans fit the description of this

[hexed solstice]

if it's only there post-pagein where the page actually had to be read from the disk, no change

so let's try after transitional PTE faults and after exists-in-the-vm-object faults

odd, placing it there doesn't do anything either

wait a minute, i didn't add it after transitional PTE to valid PTE

bizarre

it still has the unknown esr even if i tlbi vmalle1 after both page-in, map page from vm object, and convert transitional to valid PTE cases

i'll take a break now and see what i can think of when i get back

at least now we seem to be getting on to the right lines of enquiry

[hexed solstice]

tlbi vmalle1 on every page fault's conclusion + after every PTE becoming invalid is not sufficient to extinguish the problem

yet even invalidating only the tlb entry covering the elr address is enough to extinguish the problem

let's see what happens if i have this workaround and do aggressive page replacement

it appears to work fine...

still awfully slow on the second and third threads firing up

[hexed solstice]

workaround is committed. i will look into why the second and third threads are slow to get anywhere at some point in the future

[hollow hull]

aarch64 is fun isn't it?

[valid sequoia]

i've just noticed that you have <= limit here

not the issue, i hope though

[hexed solstice]

unfortunately not

[hexed solstice]

very

[valid sequoia]

the only thing left is full i/d cache flushes

tlb's already flushed enough

[earnest salmon]

common risc arch L
caching issues

[hexed solstice]

i experimented with this around the pageins/outs but that didn't seem to do anything

[valid sequoia]

riscv doesnt have this issue

to this extent

[earnest salmon]

x86 doesn't have this issue

[hexed solstice]

i notice esr=0x2000000 is also what you get when udf is executed

[earnest salmon]

at all*

*still tlb stuff tho

[hexed solstice]

so i gave the randomise memory option to limine to see whether i'd see more interesting outcomes, but no change

[hexed solstice]

what is irking me is that i understand what cache maintenance is necessary here

or at least i thought i understood it

and indeed i don't get the kind of wild behaviour that i would expect if that were deficient, even when i recycle pages furiously and have limine randomise them at boot

[hexed solstice]

riscv is next and probably soon

[valid sequoia]

but also, there's no hardware for cheap that has good hypervisor support

[hexed solstice]

porting to aarch64 was fun apart from this malarkey and i think it'll be nice to add a 4th port

[valid sequoia]

there is a very weird IoT board though that cost pennies

[hexed solstice]

i may be interested

[valid sequoia]

pine ox64

[hexed solstice]

maybe they'll even have a low quality nic driver for freertos or something that i can use as reference for what will inevitably be an undocumented proprietary thing

[valid sequoia]

6$, it's basically a microcontroller chip with good RF, and a C906 core (give or take, pentium 2 performance)

it has ethernet, wifi, bluetooth, zigbee though

[valid sequoia]

random question, can keyronex run with 4mb of ram?

[hexed solstice]

it can run on m68k with about 5 or 6 already, and could manage with quite less than 4 on the same with even a little work

[valid sequoia]

hmm another platform that would be neat has 32

[hexed solstice]

the m68k bootloader is very wasteful with its allocations

[quick saddle]

• 2 32-bit cores

[valid sequoia]

nommu though

[quick saddle]

no clue why they would chose to add two 32-bit cores one at half the speed and another at quarter the speed instead of just adding a second 64-bit core

[neat steppe]

presumably it was cheaper

[hexed solstice]

one thing i want to do soon

implement an abtraction layer like netbsd's bus_space and bus_dma

so that a driver is given bus space handles made up by the platform layer, and then operates on those

[hexed solstice]

god almighty

i deprecate and detest the gcc build system

i'm not wasting any time on it

libgcc is apparently built to use hardware floats by default and i'm trying to figure out the incantation to make one that isn't

it's probably some multilib thing with -mabi=lp64

but passing that gcc for the link command yields /ws/Projects/Keyronex/build/riscv64/tools/host-gcc/lib/gcc/riscv64-keyronex/13.2.0/../../../../riscv64-keyronex/bin/ld: unrecognised emulation mode: elf64lriscv_lp64 Supported emulations: elf64lriscv elf32lriscv elf64briscv elf32briscv from ld

[vale pelican]

-msoft-float?

[hexed solstice]

no such option

[vale pelican]

hm

-march=rv64ima?

[hexed solstice]

in any case i don't even have a libgcc fit for this

it doesn't look like one was built

[hexed solstice]

i've already got to specify -march=rv64imac_zicsr_zifencei

[vale pelican]

oh

hmm

[hexed solstice]

by contrast the riscv64-elf-gcc i built who-knows how long ago has all these variants

i'm not wasting my time on this

i'm just going to copy the conveniently made /opt/gcc-13-riscv64-elf/lib/gcc/riscv64-elf/13.1.0/rv64imac/lp64/libgcc.a which is softfloat

because i can't even specify -mabi=lp64 without ld complaining too

[hollow hull]

use cc-runtime

[hexed solstice]

what's cc-runtime?

[hollow hull]

https://github.com/osdev0/cc-runtime

just copy paste this somewhere in your source tree

done, no need for libgcc anymore

it's what Limine also uses

it would be insanity to depend on an actual libgcc with all the ports and all the hosts Limine can build on

[hexed solstice]

what a convenience

thanks for this

[hollow hull]

np

[hexed solstice]

now i never have to waste my time figuring out gcc and binutils hopefully ever again

[hexed solstice]

riscv64 is another blue port

[sinful jetty]

thats how u know its better

just enable the big endian mode

[hexed solstice]

it's arms that have that

[sinful jetty]

isnt there a riscv-be?

[hexed solstice]

there is mention of an endianness bit, i think for user mode only

and only for data, not instructions

[sinful jetty]

thats uhhh rather interesting

[hexed solstice]

i have to admit to actually being a little startled that keyronex seems to be as portable as it is

[sinful jetty]

when did you start the riscv port?

[hexed solstice]

last night, i have spent about 2 hours on it so far

[sinful jetty]

damn that was quick

[hollow hull]

the Limine effect™️

next do loongarch64

[sinful jetty]

yeah no assembly startup stubs and stuff is a big time save for ports

[hexed solstice]

all i have to do to get this far is define things like the basic layout of the virtual address space, the format of PTEs, and implement a few functions like disable/enable interrupts, set the current page table, that sort of thing

[light pulsar]

What archs have you ported it to so far?

[hollow hull]

that may be a bit more involved because it needs a software refilled TLB

[hexed solstice]

there is no interrupt management yet so it doesn't go any further than this (initialisation continues in a thread after this point)

[sinful jetty]

can u even buy a normal consumer anything with that

[hollow hull]

maybe

the person that ported it has a board

[hexed solstice]

m68k, amd64, aarch64 (sort of, it should be broken but seems to work with a stupid workaround), starting riscv64 now

[light pulsar]

Wow

[sinful jetty]

is aarch64 fully done?

[hexed solstice]

i'm considering it, other than the software refilled tlb it doesn't look at a very quick glance like there'll be a huge amount of difference v.s. risc-v

[hexed solstice]

enough to run userland code under KVM, and get as far as it can get without a root fs on bare metal pi 4

[hollow hull]

was that SMP issue you mentioned fixed?

[hexed solstice]

it isn't, i have no idea what's going on

SMP does work perfectly fine on the pi 4

but under qemu depending on what i give to -cpu and sheer randomness there's a varying set of cores reported

[hollow hull]

works pretty reliably for me

[hexed solstice]

under kvm?

i think i have a shite edk2 build

[hollow hull]

oh no

not under KVM

with -M virt -cpu cortex-a72

idk under KVM because i do not have an aarch64 device handy to test that on

maybe @vale pelican can

[hexed solstice]

without KVM, totally fine, no problems

[hollow hull]

have you tested Limine itself

[sinful jetty]

maybe its because the linux kernel enables stuff at el2 u dont expect?

[hollow hull]

aka clone it, run ./bootstrap then ./configure --enable-all and then make test-clean clean uefi-aa64-test

[vale pelican]

kvm only lets you emulate starting at el1 iirc

[sinful jetty]

yeah

thats what im saying basically

if it works fine bare metal

[hollow hull]

seriously, please do this, then run qemu-system-aarch64 -m 512M -M virt -cpu cortex-a72 -bios ovmf-aa64/OVMF.fd -net none -smp 4 -device ramfb -device qemu-xhci -device usb-kbd -hda test.hdd -serial stdiobut with kvm

i wanna see if smp works in the test

[sinful jetty]

couldnt u just add -enable-kvm to the command line u sent lol

[hollow hull]

idk if that works as simple as that with aarch64

[sinful jetty]

i think so?

ive tried on my m1 asahi linux

[hexed solstice]

it will complain about -cpu cortex-a72 and i'll have to put in -cpu host or -cpu max even though the pi 4 does have an a72

[vale pelican]

hmmmmmmm i wonder if it just times out ```c
for (int i = 0; i < 1000000; i++) {
// We do not need cache invalidation here as by the time the AP gets to
// set this flag, it has enabled it's caches

    if (locked_read(&passed_info->smp_tpl_booted_flag) == 1) {
        return true;
    }
    //delay(10000000);
}

return false;

[hollow hull]

yeah do -cpu host

[hexed solstice]

aarch64-linux-gnu-ld: limine.o: in function `ap_entry': /ws/limine/test/limine.c:240: undefined reference to `__aarch64_ldadd4_acq_rel'

never seen that one before

[hollow hull]

did you change anything?

[vale pelican]

try -mno-outline-atomics iirc

oh wait

[hexed solstice]

nothing

i should ntoe that's only on the test target

mno-outline-atomics defeated it quickly

[vale pelican]

strange that i've never ran into this

[hollow hull]

what gcc version?

the CI tests gcc too and hasn't ran into this

i recommend building with llvm for this test, please

just to have a baseline and not throw other variables into the mix

[hexed solstice]

that's without KVM

[hollow hull]

yeah that's fine

[hexed solstice]

now testing under kvm

and that's it

[hollow hull]

amazing

is this with gcc?

[hexed solstice]

it is

[hollow hull]

try llvm

[hexed solstice]

another run it got a bit further lmao

doing a clang build now

[hollow hull]

run ./configure again, if you have clang/llvm/lld all installed, it will pick that

[hexed solstice]

some runs with clang build

[hollow hull]

aight

@vale pelican you may wanna take a look, idk if this is normal

[vale pelican]

no it isn't, it used to work without problems

[hollow hull]

it could be that it's working, and it's just the printing

time to git bisect? :^)

[vale pelican]

hmm

i'll try to reproduce this locally

i think i'll try to set up netboot again because this usb stick is just painfully slow

[hollow hull]

bruh has still not bought a good usb stick

i'll donate you the money i swear

[vale pelican]

i mean i do have a faster one lol

watch gparted completely fuck up the ntfs partition on that one

the classic plasmashell crash when removable media is plugged in

oh for fucks sake the rpi firmware always tries the first partition, not the one marked as the esp

[valid sequoia]

yes

more powerful than anything riscv you can buy either

like, zen2 class hardware

[hexed solstice]

no timer interrupt yet, but it reaches second-stage init regardless as anything takes priority over the idle thread

and @sinful jetty perhaps the first time uacpi did anything on risc-v?

[sinful jetty]

indeed

cool that it works

[sinful jetty]

i like how it says class Aarch64

[hexed solstice]

oh, so it does

[sinful jetty]

understandable

linux riscv support is also mostly copy pasted arm

[hexed solstice]

realistically i could have a single LiminePlatform for these 3

[sinful jetty]

i think u should

[hexed solstice]

i could even collapse it into the ACPIPlatform as that's where all the actual logic is

[sinful jetty]

i think nt does a similar thing

[hexed solstice]

these 3 just probe it and a few of the pseudo-drivers

[sinful jetty]

in the device manager the root is acpi-pc or whatever

[hexed solstice]

the only platform driver that actually does anything specific is the m68k-virt one

since there's no ACPI

[sinful jetty]

makes sense

[vale pelican]

okay i just copied stuff off of it and nuked the ntfs partition altogether

[vale pelican]

hm that is definitely broken https://uwu.foundation/9Cp2k8Q.png

[hexed solstice]

glad it's not just me then

[vale pelican]

at a glance it seems multiple cores are trying to print at once? which unsurprisingly leads to graphical corruption

and looking in the qemu monitor two of the cores crashed

[vale pelican]

esp since it'd scroll for every line there

but this is weird, because the cores are supposed to only let the bsp progress after they're done printing

ofc -d int doesn't work so that's no use

[vale pelican]

and they are both somewhere in the body of flanterm_fb_double_buffer_flush

or whatever that calls

[hollow hull]

mind you that flanterm isn't thread safe

there need to be locks around it

but i thought we did have those in the test kernel?

sort of at least

[vale pelican]

well in theory it shouldn't try to access it from multiple threads

since it goes like this: bsp prints info, writes to goto address and waits for counter to increase, and ap prints info, and increments counter

[hollow hull]

yeah

[vale pelican]

i'll investigate this further later

[hexed solstice]

with qemu git master some pleasant progress

unfortunately now i have to wade through the swamp that is risc-v interrupt controllers

[final hatch]

what does "blue port" mean

[hexed solstice]

the colour of the logo

[final hatch]

and AMD64 was red or what?

[hexed solstice]

m68k is a black port

[final hatch]

ah

is the logo normally black, or blue?

[hexed solstice]

it's supposed to be black

but i tolerate it being blue on the blue ports

[final hatch]

ah so the blue is because of the alpha channel

[hexed solstice]

that's what i suspect

it's an unintended endianness indicator probably

[quick saddle]

A cool one for sure though

[earnest salmon]

@hexed solstice why don't you save a0, a1, d0, and d1 in your m68k_context struct

I see you do in the interrupt frame

the only thing I could think of is that since they're scratch registers you don't save them

since the yield function would expect that

actually that makes sense, nvm

[hexed solstice]

Caller saved

Swtch is a function so it works

[earnest salmon]

I figured

anyway, I'm currently implementing my context save function

after that I'm testing my scheduler

[hollow hull]

@vale pelican have you kept looking into that ARM limine issue?

[vale pelican]

not yet

[hexed solstice]

i'm going to limit support for now to what's in the AIA spec

as far as interrupt controllers on riscv go

[hexed solstice]

tonight i might do the APLIC driver

[hollow hull]

have you now?

[vale pelican]

no, sorry

ill get on it in a moment

[hollow hull]

aighty

[hollow hull]

pong

[vale pelican]

shit i fell asleep sorry

[hexed solstice]

i have an initial aplic driver now

only direct mode and only a solitary APLIC supported at present

i wonder if when i hook up the exception handler, things will just work with virtio-9p

[hexed solstice]

i need to reform and clean up the platform devices stuff, cross-platform interrupt management, and i think it will be helpful to make it so that certain devices act as their own roots of the device tree for the purposes of IOP allocation

as right now, IOPs are allocated with enough stack frames to allow them to call into a function for every parent of the device they are allocated to be sent to

totally wasteful - for e.g. acpi root -> pci bus -> NVMe controller -> NVMe namespace -> partition -> filesystem, for e.g. a read IOP sent to the filesystem, it's only necessary to call into the filesystem, then the partition (which can do the equivalent of a tail-call to the NVMe namespace, as it doesn't need to record any state itself, merely adjust the offset being deal with), and then the NVMe namespace code can do the same to submit the request to the controller

so an IOP with 2 stack frames only is sufficient, as neither partition nor nvme namespace need a return into their logic, and going to the PCI bus or ACPI root drivers is totally unneeded

[hollow hull]

@hexed solstice you can try Limine 8.0.3 to see if it fixes that ARM issue

cc @vale pelican

[hexed solstice]

riscv port is up to the first interactions with the UBC (fails as no page fault handler yet)

[final hatch]

yet another blue port I see 👀

[sinful jetty]

Seems like thats any LE port

[final hatch]

yeah

[hexed solstice]

risc-v is a bit more annoying than aarch64 as far as exceptions go with only sscratch to play around with

mips by contrast reserved "k0" and "k1" as the exclusive demesne of the kernel

[sinful jetty]

It was designed by college professors afterall

[hexed solstice]

i've figured out a way to deal with it without too much grief

has MLIBC recently been tested on risc-v?

[hexed solstice]

wtf

_DYNAMIC for ld.so appears empty when loaded

if there is a problem in my ELF loader i should have expected it to appear long before now, 3 existing ports should have experienced it

looks fine in readelf anyway

[hexed solstice]

something goes on during elf loading that ought not to

stuff gets zeroed

stuff should be zeroed but it looks like stuff that not ought to be is also zeroed

[hollow hull]

repost

[hexed solstice]

first i have a problem on risc-v to chase

well that's that

[twin haven]

I had an issue with gcc where it refused to populate PT_DYNAMIC if compiling/linking with an riscv64-elf tool chain. The fix was to change the target triple to riscv64-linux, not sure if that's still a thing.

[hexed solstice]

it's fine, it was my bad arch specific pte creation code for riscv64

now we get as far as

that's unimp in _init

i thought risc-v didn't use that

that's enough for today anyway

[twin haven]

very nice 😄

[hexed solstice]

that's libgcc and family rebuilt now

it seems the DT_INIT_ARRAY functions are using gp but i only install the __global_pointer$ into gp in _start

[hexed solstice]

trying to find the right place to smuggle this in

a preinit_array entry seems appropriate

[vale pelican]

the linux riscv sysdeps also load gp in _start

[hexed solstice]

not workable for me unfortunately

the initarray functions are using it already

[vale pelican]

strange that it works for linux then

i assume it's libgcc stuff in init_array?

[hexed solstice]

yeah, libgcc exception handling stuff

[vale pelican]

looks like that's what glibc does ```
load_gp:
.option push
.option norelax
lla gp, __global_pointer$
.option pop
ret

.section .preinit_array,"aw"
.dc.a load_gp

[hexed solstice]

looks like freebsd rtdl is just setting gp prior to calling any init array functions

#define call_initfini_pointer(obj, target)                \
({                                    \
    uint64_t old0;                            \
    old0 = set_gp(obj);                        \
    (((InitFunc)(target))());                    \
    __asm __volatile("mv    gp, %0" :: "r"(old0));            \
})

#define call_init_pointer(obj, target)                    \
({                                    \
    uint64_t old1;                            \
    old1 = set_gp(obj);                        \
    (((InitArrFunc)(target))(main_argc, main_argv, environ));    \
    __asm __volatile("mv    gp, %0" :: "r"(old1));            \
})

[vale pelican]

interesting

[hexed solstice]

i've just thrown in a preinit_array entry instead

whether FreeBSD's way is more righteous is up to others

[vale pelican]

maybe we should just do that unconditionally in mlibc

it's strange that the riscv tests work considering this

unless riscv64-linux-mlibc libgcc doesn't register a ctor?

[hexed solstice]

in my case it came up because the posix server stub is C++ so it includes some exception handling setup constructor

[vale pelican]

ah

right yeah the tests are in c

the ctor probably comes from libstdc++

[hexed solstice]

frame_dummy to be specific

which calls __register_frame_info

anyway that's keyronex for risc-v making it into userland execution

no SMP yet, no preemption, but the basics of the port are in

[violet tinsel]

nice!

[sinful jetty]

You're speedrunning these ports

PowerPC next week

[hexed solstice]

it's a distinct possibility

[sinful jetty]

damn

[hexed solstice]

4 ports is nice but 5 would be better

loong arch i will have a look at eventually

oh and @hollow hull on updating to latest limine, no further SMP problem on aarch64

however on amd64 an apparent regression

sometimes complaining of this

oh, how interesting, look what's printed

and this happens irrespective of SMP or KVM/not

[violet tinsel]

you‘re not the only one who‘s had this happen

[hollow hull]

who else did?

[hollow hull]

what qemu flags are you using?

[hollow hull]

who's had it happen and under which circumstances?

[hexed solstice]

${QEMU_EXE} -smp 4 -hda test.img -m 128 -M q35 \
  -cdrom build/amd64/barebones.iso \
  ${virtio_gpu_arg} \
  ${virtio_trace_arg} \
  ${qemu_args}  -net tap,ifname=tap0,script=no,downscript=no -net nic,model=e1000e
``` (qemu_args are serial stdio, enable-kvm, -s)

and the problem is obviated by adding -cpu host

[hollow hull]

well if you use -enable-kvm you should always pass -cpu host

[hexed solstice]

the problem remains with no -enable-kvm (and no -cpu host in that case)

[hollow hull]

i can't reproduce

send the kernel over?

[hexed solstice]

4mb! how did it get that big

unbelievable

i will have to chase that

[hollow hull]

ok i can reproduce with this kernel

[violet tinsel]

uhhh i don’t remember sadly i just recall someone mentioning it…

whats the "default" M on qemu?

[hollow hull]

ok i investigated it

my conclusion is that it's not a bug

it's actually intended behaviour

[slender flame]

oiuuuuuuuuuuuuuuuuuuuuuuuujkjjj

I was gonna type something witty but my cat stepped on the keyboard while typing it and did that, so thats what it will be

[hollow hull]

actually i think there is a bug

an off-by-1

[violet tinsel]

where?

I‘m reading through it rn

[hollow hull]

limine

[violet tinsel]

Yeah but could you tell me which line by chance?

[hollow hull]

it's the if block of the panic

it should be > instead of >=

[violet tinsel]

that seems like it could be it yeah

[hollow hull]

i mean it is it

but the issue is overall a symptom of using base revision 0

where all reserved entries above 4G are allocated in the hhdm

including this one big entry that scrapes the top of what's allowable by maxphyaddr

in this case 40 bits

the off by one error made the condition trip

even though it shouldn't even though it's at the tippy top

i never caught this because i don't really test base rev 0 kernels

[violet tinsel]

are they not with rev > 0?

[hollow hull]

they are not

[violet tinsel]

i dont get it, they still have to be in the memmap

wdym by not allocated

[hexed solstice]

i see no reason not to update to base rev 2

[hollow hull]

they are

but they are not mapped in the hhdm

[violet tinsel]

and usable entries above 4gib are?

[hollow hull]

yes

[violet tinsel]

what does it change then

[hollow hull]

it changes that with base rev 0 you have to map reserved entries

[hexed solstice]

no reserved or bad mem mappings

[final hatch]

what means "reserved" in this case?

is MMIO distinct from "reserved"?

[hollow hull]

reserved means nothing, it's really just up to the firmware what it is

[violet tinsel]

oh the point is with a huge reserved one at the end of memory would exceed it

[hollow hull]

it could be anything

[hollow hull]

yes

[final hatch]

but is MMIO counted as reserved

[hollow hull]

no

it can be

but not necessarily

[slender flame]

it usually is, but its not needed, and some fucky firmware wont bother

[hollow hull]

if you are doing mmio it's up to YOU to make sure whatever you're accessing is mapped

[violet tinsel]

i misunderstood that at first ty for explaining

[slender flame]

and I guess all the implications of not allocating MMIO space and all

[hollow hull]

if you follow the Limine hhdm scheme, you can avoid that as long as it's below 4GiB since that's always unconditionally mapped

[hollow hull]

np

[hollow hull]

@hexed solstice can you try 8.0.4

[hexed solstice]

will do shortly

chasing an odd problem in the aarch64 port atm

probably memory ordering messup around timers

feel like pure shit just want TSO back

[hexed solstice]

works fine, cheers

aarch64 port still quite broken under KVM on the pi - missed timer interrupts(? or some problem with the timer logic), timers that should be de-queued from the timer queue attempted to be satisfied

(and i thought the timer dequeuing logic was solid - it's carefully crafted to allow a timer to re-enqueue itself from within its own callback, the principle is that timers have a state field with values either disabled, queued, or currently executing, and every time you (a thread) set a timer, you've got to disable it again when you're running if you were woken up for another reason - that's like a barrier that should make it so that in case e.g. you wait on both a timer and some event, the event is signalled first from another core and you're rescheduled onto that core, away from your previous core, you then disable the timer, and that includes logic to spin when the timer is executing on another core until it's no longer executing - thne depending on whether it is now disabled or instead re-queued, nothing happens, or the timer queue lock of the cpu it's queued on is acquired, and the timer is removed from the queue)

oh...

(```c
bool__atomic_compare_exchange_n (type *ptr, type *expected, type desired, bool weak, int success_memorder, int failure_memorder);

so i transposed false and kTimerExecuting

false being 0 will make it appear like kTimerDisabled - rendering the barrier of calling ke_disable_timer() moot

[hexed solstice]

and apparently that's not the problem

[hexed solstice]

so frustrating, i thought i had this one licked

my best guess is some subtlety of memory ordering i didn't notice on amd64, but a lot of the stuff is going on within locks anyway

and there are other subtle problems on the aarch64 port as well (at least i think they are isolated tot he aarch64 port)

[hexed solstice]

what a nightmare

there are 3 problems i can observe on aarch64 under kvm, how related they are i am not sure: threads being woken when they ought not to be, timers being signalled when they should be de-queued and disabled, and iops never being fulfilled

[hexed solstice]

even with seq_cst everywhere, still seeing the same problems

either there is no use of atomics where there should be somewhere, or my problem is not necessarily rooted in improper memory ordering

[violet tinsel]

is this happening on UP or smp?

[hexed solstice]

smp

i think

i don't remember trying on UP recently

that might be sensible

[violet tinsel]

yeah you should, sounds like you‘re having trouble finding the root cause for this, trying on UP could help

[hexed solstice]

i don't think this is the root of the problem necessarily but this is certainly very stupid

why have i left this here unguarded?

let's see if i can reproduce the uncompleted iop problem now that it's protected

that's in the virtio 9p port driver and may at least explain some of the oddities with iops not being comlpeted

that leaves as remaining aarch64 problems the unknown ESR in userland code problem, the threads being woken when they shouldn't be, and the timers being signalled when they should be gone (the last two are probably related)

(i am NOT going to propose that fixingthis might magically fix the other problems - that's wishful thinking)

another problem with the buddy allocator now fixed also, courtesy of adding randomise memory

very useful limine feature

after fixing the unguarded pending_packets, and having already a) replaced all memory order specifications with SEQ_CST and b) made it so that resuming threads waiting on an object happens under that object's spinlock (something i don't think should be necessary!), i've not been able to replicate any of the problems, other than what looks sort of like missed timer interrupts/wakeups

most interestingly of all

i have not even observed that unknown esr problem

probably because there are full barriers everywhere on account of everything being SEQ_CST

let's now back out everything but the virtio 9p fix and see what happens

as expected, that fix doesn't fix the other stuff

still, i'm happy to have made some progress

and changing every atomic's ordering to SEQ_CST is no help either

i am no longer despondent now though, as i was earlier with the 9p requests being left uncompleted thrown into the mix

[sinful agate]

@hexed solstice unrelated but maybe slightly related, we’ve talked about InitWare on Managarm before and as I kinda want to start working on that again soon, I thought I’d just pop in and see what we can do on that part. For example, eudev lacks sd_notify support (which sucks) and one of the “newer” additions to systemd is tmpfilesd which might be good to add to InitWare. Idk if that’s something that would be possible, and if so, I might be able to help with whatever it is that you need. What do you think?

Oh and I guess expect patches to add Managarm support soonish

[hexed solstice]

tmpfilesd, i have no objection to that, it can be useful

and is small enough

eudev, it gets more tricky there, i think they could benefit from adding sd-notify support

i'm sure i put together a tiny embeddable implementation of sd-notify previously

[sinful agate]

Eudev upstream is kinda dead anyway (last commit 4 months ago)

[hexed solstice]

arsen wanted to rebase it

[sinful agate]

Arsen wants a lot of things. But arsen is also really really really busy. Like actually busy. His TODO list is quite long and due to IRL items he can’t get to any of them (at least, that’s what I gathered from his responses in the Managarm discord)

[sinful agate]

I guess I can just add the udev sources to a new top level directory in InitWare and gate it behind a cmake flag. Then just try to compile and add stuff as needed?

Of course, taking care to gate any BS behind the correct flags

Same for tempfilesd

Might start with that as it’s way less intrusive

[sinful agate]

If only I had a fucking working network connection

Cant even clone a git repo fucking hell

[hexed solstice]

i am preparing to do some refactoring around external interrupt management

at first i was thinking of declaring that on every port i'll have a flat space of wired external interrupts (this would be acpi GSIs on most platforms) and have the platform device provide a method to register some interrupt by flat number

but now i am feeling more like passing structs down, carrying a reference to the relevant interrupt controller device object, and some identification with them so that a "set up interrupt source" message can be sent to the device object that they contain a pointer to

in practice i would be needing that struct anyway to carry details about the interrupt source (edge or level, polarity)

[sinful agate]

network acquired, sources obtained, InitWare fork made. Initial look at tmpfiles config made, I think I know what to ship at least initially. I'll see if I can make a start at porting tmpfiles (directory structure as laid out above) and then we can work from there

@hexed solstice would that be an accepted PR (of course, preliminary testing will be done on linux to see if it at least compiles, considering it works on systemd™️ I somewhat doubt I'll need to actually boot with it, but we can set that up too)

[hexed solstice]

it certainly will

i have taken the liberty of preparing an InitWare thread https://discord.com/channels/440442961147199490/1272279743512707122

[sinful agate]

Excellent. Wouldn’t want to clog up the Keyronex thread. I’ll move further InitWare discussion over to there

[final hatch]

@hexed solstice how do you pronounce keyronex btw

just wondering

[hexed solstice]

how do you pronounce it?

[final hatch]

idk, i guess key (🔑) roh necks

is it meant to have a distinct pronunciation or is this it

[hexed solstice]

that's roughly in line with how i say it

[final hatch]

ah cool

[frank badge]

same here

or wait

roh like roe deer?

or raw

[hexed solstice]

raw

and i don't enunciate the last e much as an e, more as a short i

[frank badge]

oh, ive been saying it as roe

[frank badge]

you did call it keyronix before though right?

could be somewhat used to that name

[hexed solstice]

i think so

i think that was the original but it was keyronex quite quickly

[frank badge]

personally i prefer keyronex but that could just be because thats what im used to

[sinful jetty]

Keyroneecks?

[hexed solstice]

i as in bit or hit

or as in shit

[sinful jetty]

Lol

[hexed solstice]

i wish GCC had blocks

living without them in objc is frustrating

for iteration APIs (PCI caps for example) it's either expose them as functions like create iterator, next; or define a protocol that your object implements and have a method thereof be invoked at each iteration; or call a C callback (in which case you have to define it out-of-line, and potentially just immediately send a message to the object that wanted to do the iteration so you've access to its ivars)

[frank badge]

be the change you want to see

[hexed solstice]

meanwhile the C++ people are eating and drinking and being merry and dancing with their lambada expressions

[hexed solstice]

arsen (veteran managara and gcc contributor) asked me to consider it

i don't rule it out if i ever feel like looking at GCC

but the target configuration alone is hellish and i fear what the rest of it looks like

[twin haven]

lambda expressions are a beautiful thing 😁

[stone orbit]

great way to create dangling pointers if you make a little oopsie

though i don’t remember the last time that happened to me

[twin haven]

Ah I remember causing myself a lot of pain when I first used them

Classic c++ moment

[light pulsar]

Lambdas are a pain to get working fully in a free-standing environment

[zenith karma]

not really no?

they just work lol

[light pulsar]

They work until you want to capture stuff

[vale pelican]

that also just works

lambdas don't allocate

int x = 0;
auto foo = [&x] { x++; };
``` is more or less equivalent to ```cpp
int x = 0;
struct {
    int &x;
    void operator()() {
        x++;
    }
} foo{x};

one pitfall that comes to mind with lambdas for me is if you make a lambda coroutine that captures stuff

[zenith karma]

or just an & capture and use after scope

[vale pelican]

after the lambda suspends and returns to the caller the lambda object potentially goes out of scope

[zenith karma]

see this is why rust is better

[sinful jetty]

Capturing lambdas should work just fine in freestanding I think, aren't they just anonymous structs

[zenith karma]

:^)

[zenith karma]

yep

or templates

[vale pelican]

so now the rest of the lambda body accessing any captures is using this out of scope

[zenith karma]

this->x not this

but either way its usually fine™

[sinful jetty]

They are the same pointer

[vale pelican]

no the lambda object itself goes out of scope in this example

[sinful jetty]

But the member is the object tho

[vale pelican]

as in: ```cpp
void foo() {
async::detach([&my_socket] () -> async::result<void> {
co_await my_socket.send(...); // assuming this suspends the coroutine
// the lambda object is now out of scope
// accessing my_socket (aka this->my_socket) is now invalid
}());
}

because the lambda object is not a part of the coroutine frame

only the hidden this pointer is

there has been a non-zero amount of times where i ran into this without thinking

[frank badge]

mo, they still work

[hexed solstice]

So long as they don't escape

[zenith karma]

even if they escape, i think?

i believe there is no compiler runtime for them

[stone orbit]

no but there isnt one type that all lambdas can be converted to or something

so if they escape you need to allocate

aka std::function or whatever

[zenith karma]

or define your own type erased function type

its not like hard to do

[stone orbit]

is there a way to avoid allocating?

[zenith karma]

store enough bytes for the biggest lambda?

or auto return types lmao

[stone orbit]

ah, so no

[zenith karma]

but you can do allocating yourself?

[stone orbit]

i wanted to add "and have it be functionally equivalent to std::function"

but you replied

[zenith karma]

honestly kinda c++ L

rust impl Trait ftw

[stone orbit]

you still need to Box them if you want to store them

[zenith karma]

(in the positions it is legal which is not all of them lol)

no?

apart from the fact you can* define your own non-Box type that's type erased

you can also do &dyn Trait

[zenith karma]

*except moving out of deref which is not possible to do in user code because yay rust

[stone orbit]

i meant in like a struct or something

[zenith karma]

you can make your struct generic over that?

[stone orbit]

yeah but that is not always a good solution

[zenith karma]

true

sometimes you do need a box

but either way still better than std::function

[stone orbit]

not a big deal though

rust no_std + alloc is so much better than freestanding c++

[zenith karma]

no oom handling though

[stone orbit]

yeah that kinda sucks

[zenith karma]

rust no_std without alloc is still a lot better

[stone orbit]

but there is this new allocator trait

in std tho

[zenith karma]

its in alloc not std

[stone orbit]

oh is it a re-export

fair

[zenith karma]

but either way i can 100% guarantee that no third party libraries will work with only fallible allocs turned on

[sullen fern]

well using a templated function works

template <typename F> void func(F callback)

[stone orbit]

.

[zenith karma]

auto callback works too

[hexed solstice]

i feel in general like keyronex's devicekit needs more structure

soemething a bit like I/O Kit's planes or windows' device stacks

i allocate at least as many stack frames in an IOP as 1 + the number of ancestors of that node in the tree of devices

[frank badge]

which is also just a templated callback

[hexed solstice]

i have a bee in my bonnet about page sizes lately

i feel like i ought to support larger page sizes without requiring that a page's worth of page tables be always allocated

take AArch64 with 16 or even 64k granule size for instance

[violet tinsel]

does keyronex support 2mib and 1gib pages?

or is it not just granularity that‘s not yet been worked on

[hexed solstice]

not yet, but i am thinking of adding them in the near future

[sullen fern]

whats annoying is unmapping and remapping them

[hexed solstice]

i'm considering moving to having a virtually contiguous pfn db and freeing up a word - 12 bits of space for whatever i want to put in pfndb elements

i would want to be considerate about that though as pages belonging to a numa node should be described by pfndb elements allocated on that numa node

assuming 2mib large pages then that's space enough to describe 128mib of memory and the physical address space shouldn't be that fragmented. and i am fairly sure numa affinity regions are going to be more than 128mib aligned themselves, which saves on some bother

if your large pages are 32mib (aarch64 with 16k granule) then that's slightly more annoying, but even then, it's not as though the entirety of the 32mib page has to be used. you can free parts of it and reuse those. i don't think the aliasing should be a problem unless you go around playing with elements of the pfndb which do not exist

and one thing that this figures into is that i want to join up bootloader-reclaimable and usable regions and then unfree the reclaimable ranges early on. later i can free the reclaimable ranges. i could go further and merge in the kernel and modules, and leave those un-freed. it's just nicer this way

in fact i will probably work on this next. then i might merge the new rwlocks, think about my long term plans regarding things like larger page sizes (where page tables != page size in particular), clean up certain areas where the platform-specific page table manipulation functions are inappropriately used where the "tables" being manipulated are not interpreted by hardware but rather software, and introduce memory domains (replicated buddy allocator, dirty/standby page queues, page daemon, etc. for NUMA domains, as if numa scaling is something i should be worrying about, but it's fun)

[violet tinsel]

linked list of numa zones with each one virt. contig. pfndb sounds sensible (especially because that simplifies prioritizing the correct numa node, because you just need a cpu local preferred numa node index/pointer)

[hexed solstice]

very reasonable i think

[sullen fern]

yeah I think that sounds like the simplest design

you keep the list of pages closer to your numa domain in a per-cpu queue

[hexed solstice]

the pfndb elements will have a "to what numa zone do i belong?" field anyway, and that + a global array of numa zones means that finding the physical address a pfndb element corresponds to is like this:

vm_domain_t *dom = &domains[page->domain_dx];
return dom.base_pfn + page - dom.pfndb;

[violet tinsel]

may i ask why you’re worrying about numa? Just for fun or are you planning on making the greatest server os ever

[sullen fern]

I guess same reason why he's worrying about RCU, locks, swap and all the goodies; scalability

realistically there's no use-case for NUMA in a hobby OS but it's all about fun

[violet tinsel]

sure haha
also it‘s cööl

[hexed solstice]

it is exactly for the fun of it

and for doing a little larping

netascale object solutions ltd. only produces the best

some other things i want to look into soon:

• finer locking in the VMM
• memory compression again

[hexed solstice]

disadvantage of being a 4-way portable kernel: making changes to things like this (i.e. adopting a virtually contiguous pfn database, and in service of it, large and huge pages) requires repeated work across architectures

not on m68k anyway, but on all others

but as part of this, i think i'll move over to @twin haven's limine shim

[hexed solstice]

if i make a 5th port, what would people be hoping that 5th port to be?

preferably one for which real hardware is affordably available

[sullen fern]

what about 32-bit x86?

[sinful jetty]

This

[sinful jetty]

Power pc

[hexed solstice]

if i supported 32-bit x86 it would be extraordinarily half-hearted

i assume i can direct map all ram

[sinful jetty]

Is m68k the only 32 bit platform ATM?

[hexed solstice]

and i have no intention to deviate from that (well, i would consider allowing it to be not a proper direct map, but rather several direct maps, for discontiguous ram)

[hexed solstice]

it is

[sinful jetty]

Yeah 32 bit x86 basically supports the same amount of physical memory as 64 bit

So its a good design exercise to support that

[hexed solstice]

it's an interesting design exercise but mostly a useless one

i don't rule out that some day i would consider it

[valid sequoia]

MIPS for cheap routers sounds nice

ARM32 is quite a mess so i'd skip

[sinful jetty]

Also compatibility mode support for 64 bit mode

[sullen fern]

ARM is quite a mess*

[hexed solstice]

there are certain things though that i would want if i wanted to allow no-direct-map operation, like recursive paging, that i simply can't adopt across-the-board because it would clash with the m68k port (i don't think i can do it with the way page tables are arranged on the 68040)

[valid sequoia]

loongson is 64bit and i've seen that devboards and pcs are starting to get available

[sinful jetty]

Anyway I'm rooting for PowerPC and ps3 support

[valid sequoia]

granted it's just medicre riscv so it's not too exciting

[hexed solstice]

mips is an interesting one

powerpc would be very cool

[sinful jetty]

MIPS for PSP support

[sullen fern]

yeah PPC would be nice, dunno which device you could buy tho

[sinful jetty]

Xbox360 or ps3

[hexed solstice]

loongson, they have a reference published

i am very quickly flicking through it

[sullen fern]

idk debugging a ps3 doesnt sound very nice

[sinful jetty]

Well yes

[sullen fern]

xbox360>ps3 btw

[sinful jetty]

Eh ps3 is quirky but cool hw

[hexed solstice]

software refilled TLB, interesting choice today

[valid sequoia]

it's a mips64 copycat

well, not even copycat, it's just mips64 like riscv is mips32

[hexed solstice]

amigas