tabula imaginarium: osdev experiments in zig

[haughty cipher]

in that case, i can just use uefi for all my allocation

and not have to write an allocator in the bootloader

also i should double check what the memmap type for the uefi firmware page tables is at some point

[haughty cipher]

uefi is putting page tables in efi_reclaimable so thats good

thats what i was hoping for

(and what makes the most sense)

but the spec didnt say so figured id check lol

well something im listing as efi reclaimable, so either boot services code/data or loader code/data

[placid swallow]

may I suggest assigning your own ids to the cores (a soft id vs the hardware id you might get from the lapic or similar) since the ones the hardware gives you might not be sequential. So you can have large gaps.

[haughty cipher]

everything i have as per-core is set up page-aligned so i just dont bother mapping anything in the spots for missing cores atm, but i could do my own IDs

and this way i dont need to figure out how (if at all) to tell other hardware/firmware about chaging the IDs, and can use the relatively fast access to the lapic id on x86

though im not sure ill actually use the lapid id, i might use the ACPI processor id

in any case, in zig syntax, my current setup has

const PCBs: [*]align(8192) PCB = @ptrFromInt(0o770_000_000_000_0000);
const kernel_stacks: [*]align(8192) [8192]u8 = @ptrFromInt(0o771_000_000_000_0000);

and the PCB struct has page-aligned size

0o771_xxx_xxx_xxx_0000 gives the kernel stack if you put the processor id shl 1 in for the x's is the idea (shl 1 because its 8k stacks which isnt sized on an octal digit boundary lol)

[haughty cipher]

im actually considering doing a second shift there since ive got extra bits for this still and having each core get a 12k stack with a guard page

tabula imaginarium: osdev experiments in zig

also renaming the post to include the name of the bootloader im making to go along with the imaginarium kernel

[haughty cipher]

guess i should probably work on implementing FAT next

actually get my kernel loaded into memory

and maybe more relevantly, the loader config file

[haughty cipher]

me: why the fuck is the first line of this missing

looks deeper

"\r\r"

ah

thanks uefi, for requiring fuckin carriage returns

anyway now that its not overwriting itself because i mistyped \r instead of \n

we got some basic FAT info being read in by the thing

ok i finally got sick and tired of the uefi carriage return shit and rigged up my log function to fix line endings for me lmao

also allows using multiline string literals

which is nice

got that fixed and that and the initial FAT stuff pushed 🎉

next up the actual FAT and then we'll see about iterating directories

which is a project for not today

[haughty cipher]

to be clear, i like carriage returns as a thing, i just like multiline string literals more than i like carriage returns lol

[haughty cipher]

gonna be a few days before i get any more work done on this

gotta repaste and re-pad my GPU, thing is constantly overheating

[haughty cipher]

FAT12 officially sucks, but only because the fat entry size doesnt cleanly divide sectors

same reason my PFMs are smaller than NT PFNs, to make it divide cleanly a page

[haughty cipher]

well fuck

[haughty cipher]

we can now read the FAT root dir on all FAT types \o/

ignore the addresses on the left, im using zig std code to dump it and you cant give it a fake base address to work from

it takes a noticeable amount of time on FAT12/16 because someone (looks at @hoary aurora) left setting a proper root directory length as a TODO on those fat types, defaulted it to the max value, and left me printout out four whole pages of ram to do this

[haughty cipher]

next up: actual proper directory iteration. ive got the structs for it but ive yet to actually do anything with them

[languid orbit]

Isn't 64 bit NT PFN 48 bytes? Making it PITA.

[languid orbit]

exFat?

[haughty cipher]

exactly, my equivalent is squeezed into 32 bytes so it can align well

[haughty cipher]

does uefi support it? if not then i don't care

[languid orbit]

afaik not (yet). But it's a requirement for sdxc cards over 32GB if you care.)

[haughty cipher]

I'll implement it in my kernel for sure, rn I'm working on bare minimum for my bootloader

need normal fat to load the esp, whenever I get around to designing my own fs I'll need that to load the kernel, but that's all I really plan to do in the bootloader

[languid orbit]

I see, but why do you need FAT code in the loader if UEFI provides it for you? FAT is the only FS guaranteed by the spec to be always supported. You just use simple file system protocol for opening the root directory on a fat partition (any, esp, ordinary) and then file protocol for reading (or even writing if the loader needs that). It's other fss that would need to be supported in one's loader if the OS needs that (most probably it will, since FAT as a boot volume (OS home) is not exactly a good idea). For example, I'm adding ... JFS support to antload.

[haughty cipher]

doing it both to maybe support bios boot later and to start learning how fat works in advance of implementing it for the kernel, plus itll unify my code somewhat

[haughty cipher]

i still want to know who thought that uefi should provide memset and memcpy as function pointers in boot services

actually kinda insane decision

[languid orbit]

All UEFI services are provided via pointers in the Boot Services, Runtime Services and protocols. The other way would be using PE import, but UEFI doesn't use it on the interface side. As of why such an arrangement. Well, at least for you to not mess up things by calling a service after it cannot be called (what especially applies to memory functions). There is a hand off phase, between the fw and loader and after the latter takes control over the memory map, fw cannot provide any services except the runtime ones. It's reasonable.

[haughty cipher]

yeah I know for most things, for memmove and memset it's strange though, I'd just let the languages provide that

[haughty cipher]

who decided that end of stream on uefi file protocol reading should be the same error code as device errors and reading from deleted files

[haughty cipher]

pushed a bunch of misc stuff in advance of whatever i do next in terms of file io

including a first draft at a zig std.Io.Reader that works off of a uefi file protocol, though untested yet

also cleaned up some of my tooling, and added a setup thatll generate the bootloader config file from the kernel build output

thank god for lazypath

best feature of zig build

being able to say hey zig run this executable, pass in files from wherever it caches them, make a temp file with this basename and give an executable the final path of it so itll act as an output file and give me a lazy path to that, capture the stdout of this executable as a temp file and give me a lazy path to that

its so nice

[haughty cipher]

fascinating, the memory map changes noticeably if i use the ovmf-vars.fd compared to when i dont

theres a whole extra entry of type memory_mapped_io starting at 0xffe00000 thats marked with the runtime flag

[haughty cipher]

made the config files more reasonable at the cost of being way more unhinged lmao

old

new

[haughty cipher]

progress proceeding apace

including some unit tests on the bits that i could make do that on native

(dont have a unit test runner yet for the actual loader or kernel, and tbh im ok with that for now)

its reading the config file from the esp using the uefi file protocol

i kinda hate the protocol in a few ways

but for now its fine i guess

most of the issues ive got with the uefi file protocol have a lot more to do with it not being well integrated with zig, and that i could definitely work on

the one unrelated issue ive got is that it doesnt have a unique error code for reading outside of the file's bounds

[haughty cipher]

can iterate directories with the uefi file protocol now

no screenshot yet as i have already shut down the dev machine for the night tho

[haughty cipher]

think im going to actually officially say at this point that im not going to do bios booting with tabula

will clean the code up significantly to throw out all the non-uefi stuff but otoh itll be a no turning back thing, getting non-uefi back in will be way harder

this will allow me to cut and paste my fat code into the kernel and not have it duplicated in the bootloader, as well as remove a LOT of random ifs and switches that can just be simplified out

[haughty cipher]

thinking about how to handle this and i think ill throw a handle to the fs root on any volume i enumerate that has the simple_file_system_protocol

that way if the loader needs to load files from the fs it can check for that and if its not there fall back into the raw fs options (which will be for when i have a custom fs)

[haughty cipher]

one other thing i just realised (as im about to maybe go sleep) is that uefi has a stall function so i can have the bootloader pass a tsc frequency estimate to the kernel too

[haughty cipher]

maybe tomorrow ill actually get some work done on this lol

anyone know a good book on file system design?

[haughty cipher]

the kernel file is in memory

(as long as the kernel file is on a partition that supports SIMPLE_FILE_SYSTEM_PROTOCOL, otherwise it panics atm)

took me a solid ten extra minutes because i forgot to make it change the path separators lmao

[haughty cipher]

if i set the loglevel to info i get this output

with it set to debug its way too spammy to be worth sharing lmao

lmao discord automatically swaps it to attachment if i try to paste in the full debug output

theres no kernel symbols file to load because the zig objcopy implementation was a buggy mess and got thrown out and i aint going to rely on system objcopy when i do all my dev on windows where that doesnt really exist, and without objcopy i cant split out the kernel symbols file easily

[haughty cipher]

ok i got distracted and now the memory map is aligned when printing lmao

way more readable imo

virtual addresses (and page numbers in this case) are still always printed in octal if i can help it

makes it so much easier to think about page tables that way

too hungry to be bothered being precise about what i stage for commits rn so the latest commit is literally everything i did today

but otoh the kernel is in memory

so im hyped about that

[haughty cipher]

got it getting the bspid now, and on x86_64 mapping the lapic block and any ioapic blocks

(well, putting them in the memory map so the paging stuff can do the mapping later)

next up: virtual memory

aka a whole lot of allocating page tables lmao

the fixing up of protocol pointers might end up annoying though

may be easier to basically duplicate everything instead of fixing up all the pointers

the downsides of using a linked list for the memory map

[haughty cipher]

the goal with paging and how the loader arranges virtual memory stuff is that if the literal first thing the kernel does is to wipe the mappings for lower-half memory that nothing should break

id have the loader do it, but the loader is in low memory because thats where uefi puts it and the loader cant really wipe itself out of existence and expect things to work lmao

[haughty cipher]

probably dont need as much memory reserved for loader persistent stuff tbh

[haughty cipher]

will tomorrow be the day i exit boot services? no idea but ig theres a chance lol

will tomorrow be the day i freeze my butt off? seems more likely tbh lmao

[haughty cipher]

oh god i forgot that the arm manual wont let you scroll through pages for some damn reason

[haughty cipher]

wrote out the full memory layout instead of actually loading things into memory lmao

[haughty cipher]

got the kernel image/elf loaded now, mostly

i say mostly because im not applying relocations at this point, at least not in the loader

idk if i actually have any that i need to apply in this

but either way im not doing em as of yet

ok yeah i dont even have a dynamic phdr rn lol

[haughty cipher]

alright, next up will be page tables

and then i can start fiddling with uefi nonsense lmao

pushed everything, and now i can say my kernel is in memory twice finally

once as the file itself as a big ass buffer, the other laid out how the linkerscript says it should be

[haughty cipher]

Update because i forgot one of the entries i need whoops

|-----------------------|-----------------------|------------------------|
|     Start (octal)     |     End (octal)       | Use                    |
|-----------------------|-----------------------|------------------------|
| 000_000_000_000_0000  | 000_000_000_001_0000  | Null Page              |
| 000_000_000_001_0000  | LOWER_HALF_TOP - 1pg  | User-Mode Memory       |
| LOWER_HALF_TOP - 1pg  |    LOWER_HALF_TOP     | User-Mode Env Blocks   |
|-----------------------|-----------------------|------------------------|
|    LOWER_HALF_TOP     |   HIGHER_HALF_BASE    | Non-Canonical (No Use) |
|-----------------------|-----------------------|------------------------|
|   HIGHER_HALF_BASE    | -540_000_000_000_0000 | Nonpaged Pool          |
| -540_000_000_000_0000 | -740_000_000_000_0000 | Paged Pool             |
| -740_000_000_000_0000 | -754_000_000_000_0000 | Per-Cpu Stacks (16K)   |
| -754_000_000_000_0000 | -757_000_000_000_0000 | Per-Cpu Data Blocks    |
| -757_000_000_000_0000 | -760_000_000_000_0000 | Loader Persistent Data |
| -760_000_000_000_0000 | -761_000_000_000_0000 | TMP MMIO Mapping Space |
| -764_000_000_000_0000 | -765_777_000_000_0000 | PFMdb                  |
| -765_777_000_000_0000 | -766_000_000_000_0000 | PFM Presence Bitmap    |
| -766_000_000_000_0000 | -766_000_000_000_0000 | TTBR0 Mirror (Aarch64) |
| -767_000_000_000_0000 | -767_000_000_000_0000 | CR3/TTBR1 Mirror       |
|                       | -767_767_767_767_0000 | root page table        |
|                       | -767_767_767_766_0000 | TTBR0 (aarch64 only)   |
| -770_000_000_000_0000 | -777_000_000_000_0000 | Framebuffer(s)         |
| -777_776_000_000_0000 |     TOP_OF_MEMORY     | Kernel Code/Data Image |
|-----------------------|-----------------------|------------------------|

Negative addresses represent values sign-extended to the architectural address width.

TOP_OF_MEMORY is defined as one above the maximum representable virtual address.

The definitions of LOWER_HALF_TOP and HIGHER_HALF_BASE depend on the architecture, as follows:

|---------|--------------------|---------------------|
|  Arch   |   LOWER_HALF_TOP   |  HIGHER_HALF_BASE   |
|---------|--------------------|---------------------|
| x86_64  |  400_000_000_0000  |  -400_000_000_0000  |
| aarch64 | 1_000_000_000_0000 | -1_000_000_000_0000 |
|---------|--------------------|---------------------|

table at the bottom because aarch64 gives me one extra bit of space

of these, the upper chunk of nonpaged pool is used by the bootloader for mapping reclaimable things that get sent to the kernel, and other than that the loader only maps the recursive tables, kernel, framebuffer, and one of the per-cpu stacks before passing control to the kernel

and whatever it uses of the "loader persistent data" range

uefi runtime services get allocated bottom-up in the loader persistent data range, other persistent stuff the loader needs to leave mapped (like the kernel file itself) get allocated bottom-down in that range

(uefi mapped bottom up just so some of its memmap entries can get merged when i sanitize the memmap)

currently desparately hoping that no extra runtime blocks get allocated by uefi during operation, because if any are then i need to completely redo the order of operations here lmao

(if they dont change then i can allocate all of my initial page tables from the uefi page allocator. if they do change then i need to switch to a memmap allocator instead which kinda sucks but would mean i can definitely allocate the PFMs in the loader at least? tbh i could do that beforehand anyway actually... hmm)

[haughty cipher]

i think ill bite the bullet and make a memmap allocator tbh

it can be the last thing i do before leaving uefi boot services

[haughty cipher]

alright were out of boot services now

i can add more stuff during boot services later™

but for now we're done with boot services

[haughty cipher]

this is lookin kinda right

all things considered

[haughty cipher]

yknow

it might help if i actually applied my new page tables

out here wondering why it keeps page faulting and uh thatll do it

now all this output looks correct and im almost scared about that lmao

[last pebble]

octal lengths is certainly a choice

[haughty cipher]

to match with the octal addresses which is too match with page table sizes

[last pebble]

page table sizes are octal?

[haughty cipher]

every three octal digits is one level of the page table

[last pebble]

oh

[haughty cipher]

because each level is a 4k page with 64bit entries which is 512 or 8^3

so it's really convenient to print them in octal

[last pebble]

that makes sense

[haughty cipher]

anyway everything I've thought to test or double check is working and that has my susssed so I'm taking a break for a bit, plus I can smell dinner from the other room

all that's left to get a minimum working sample is the stack swap and call main in theory

[haughty cipher]

alright had some food, time to find out what the catch it on all this stuff that seems to be working first try lmao

[haughty cipher]

🎉

by god it works

it doesnt have a boot protocol yet though

other than "the stack is reasonable, the gdt is in high mem, and all your shit is mapped as expected"

[placid swallow]

if you do define a protocol for your bootloader, I'd be interested (mainly for using uefi runtime services)

[haughty cipher]

ive got something quick now (though everything is very specific to my os rn, this is bespoke) but once i verify they actually work ill be more than happy to share how i did it

[placid swallow]

sounds good 🙂

btw there's no rush at all, I have a big list of things I want to do with my kernel before this

[haughty cipher]

tryin it now anyway, was one of my next things to attempt

well the zig uefi stdlib continues to be an untested mess with many bugs in all the helper functions but runtime services is defintiely working

or at least the time part

imaginarium.debug: uefi time .{ .year = 2026, .month = 1, .day = 26, .hour = 1, .minute = 41, .second = 18, ._pad1 = 170, .nanosecond = 0, .timezone = 2047, .daylight = .{ .in_daylight = false, .adjust_daylight = false, ._ = 0 }, ._pad2 = 170 } (epoch 1737942078000000000), resolution 1

this is being executed from a virtual address i picked by the kernel

[haughty cipher]

if you can read zig, its all pushed now and working https://codeberg.org/Khitiara/imaginarium

if you have latest(ish) zig master you can even recursive clone and just run "zig build qemu" and the only external deps are zig-master and qemu

[placid swallow]

sweet, thanks

I'll have a look at this tonight

[haughty cipher]

feel free to ping me with any questions btw

(i only figured any of this out because i read the edk2 source lmao, not at all tested on hardware yet)

[placid swallow]

lol

[haughty cipher]

SetVirtualAddressMap has so many non-obvious gotchas too, it kinda sucks

(the tldr is:

1. get the efi memory map. ensure everything in there is identity mapped (base 0). best to do this before swapping off efi page tables
2. set the VirtualStart field of each runtime attribute efi memory map entry to the virtual address you want that block of memory to have. doesnt matter where
3. call SetVirtualAddressMap with that memory map
4. ConvertPointer can only be called DURING SetVirtualAddressMap, so you either have to convert your system table pointer manually or use an efi event to listen for it and use it during (I do the event route)
5. once all addresses are converted, apply the new page mappings which match what you set in the efi memmap earlier
6. now the converted system table pointer can be used to access all of the things. you can also convert any other uefi pointers you want during 4 including the runtimeservices table
   )

setvirtualaddressmap's operation is designed so that the new mappings dont actually need to coexist with the old at least, it does it bottom-up

or more accurately it lets every other component use events to do all the work lmao

any runtime driver's pe image also gets relocations reapplied at that point and instruction caches invalidated, after it gets to run its event

learning about the event so i could use convertpointer made this so much nicer

[placid swallow]

mm ok, good to know thanks

that was my understanding of the process as well, the event is good to know about

do you know if SetVirtualAddressMap can be called after boot services are exited (with recreated identity mappings)?

that was my plan, so it do it from the kernel

assuming it works

[haughty cipher]

it has to be after boot services are exited

errors otherwise

[placid swallow]

ah perfect

[haughty cipher]

but if you recreate the id mappings i think its fine

the thing that threw me off for a while was that when i tried to overwrite the std's system table pointer i forgot that i was referring to the table in my logging code (so it can use conout during boot services) and that was page faulting but debugging uefi loader code is a PITA

[haughty cipher]

i was considering this but already had other things i wanted to do differently from limine so ended up making my own instead of remaking in the kernel what uefi already had page-mapped for me until limine swaps to its page tables lol

anyway turns out among all the other bugs ive found in it, the std.os.uefi efi time to unix timestamp code is a year off

(the other thing i want to do in uefi before booting to the kernel is use efi's stall to get the tsc frequency which will be super nice to have without having to calibrate myself later)

[placid swallow]

yeah fair enough, I've almost made the same decision a few times haha. That or making a prekernel for all the init stuff I want to do different, but for now its not a huge concern for me. Having it in the kernel at least means I can require very little from the bootloader, and anywhere I can get a limited set of the limine protocol I should be able to boot.

[haughty cipher]

yeah

[placid swallow]

oh lmao

getting the tsc frequency early sounds nice

[haughty cipher]

when i convert the unix timestamp it gives me its 2025 but the struct says 2026 lol

[haughty cipher]

yeah boot services has a stall(microseconds) function its great for that

[placid swallow]

I wonder if someone hardcoded 2025 somewhere 😆

[haughty cipher]

nah i checked the code i think its just an off-by-one

it iterates years to make sure it doesnt mess up leap years

[placid swallow]

ah ok

[haughty cipher]

whoever wrote it forgot that its years since the start of 1970, not the end of 1970

[haughty cipher]

@placid swallow fwiw if you do want to try my bootloader it does use a config file from the ESP to load the kernel, albeit only from partitions the uefi itself has file system drivers for, and the boot protocol is that the kernel must be a statically linked 64-bit elf (with no dynamic phdr/section for now) and its entry takes a singly linked list of tagged unions passed in rdi (so standard sysv callconv first parameter. theres a readme in src/ldr with docs on the memory layout the loader gives

the tagged unions are in src/proto.zig but i declared it as a zig tagged union of pointers to the actual union payload types because zig tagged unions dont have a defined memory layout so i had to do it manually

and theres plenty of comments in src/ldr/Config.zig on the config format or you can build it and inspect the config it generates for my kernel, which will be put in the out folder

i wouldnt expect much atm, i dont have framebuffers as a protocol yet lmao, but it should work fine

amusingly the fact that i can use uefi for filesystems means in theory one could make a uefi driver for your custom file system type and have it just work™ if you install that uefi driver

tbh the memory layout is the only strongly hard-coded thing about it atm, being a subset of the memory layout my kernel will use, but thats virtual memory stuff and you can always move things around at will yourself

[placid swallow]

many thanks, this all sounds super useful to know

[placid swallow]

yeah thats fine lol, I dont do anything with framebuffers at the moment anyway

[haughty cipher]

it also gives you an 8k stack

and the kernel base address requirements are the same as limine's (FFFFFFFF8000000 or whatever the exact number is i forget if its 8 Fs or 8 0s)

i could do more of the MM init in the loader but tbh the earlier im in the kernel the happier i am lmao

[lost hawk]

It's 0xffffffff_80000000

[haughty cipher]

ah ok cool my first guess was right (I just woke up and couldnt be assed to actually check lmao)

i think my next work will still be on tabula but itll be making sure the aarch64 side of its few arch-specific things works

[haughty cipher]

ugh apparently WHPX (the qemu hypervisor under windows) cant handle pflash for ovmf so now i need some way to concatenate two files in the build system if i want to use accel on windows

which matters for tsc frequency stability - did ten samples using uefi to stall 1ms each time and got frequencies ranging from 2.5ghz to 4.5ghz

ran it under kvm on wsl (after pulling the files from the windows mount because zig cant build on mounted windows filesystems on wsl rn) and got ten samples of 3600mhz exactly

[haughty cipher]

yknow, if im going to try to build this for aarch64

it might help if i made sure my build system was actually compiling for aarch64

[haughty cipher]

well it started to log something before dying on aarch64 lol

oh

lmao

i forgot to do the equivalent of cli on aarch64 lmao

[haughty cipher]

sometimes the problem is more obvious and stupid than expected lmao

when clearing limine from the aarch64 linkerscript i accidentally deleted the entire data block so it put all the data in rodata and then threw a permissions fault when trying to write a variable in the kernel

anyway here we are from aarch64 this time

[haughty cipher]

fixed another silly bug and now it gets all the way to the end

[haughty cipher]

its easy to tell when it leaves boot services because the color just fuckin stops lmao

[haughty cipher]

ugh i guess i need like half of a MM before i can actually start with uacpi huh

[lost hawk]

Yes and you need interrupts up

[haughty cipher]

for early table access?

i know i need it for the main but for the early?

(i did all of uacpi on my old version but rewriting the kernel and kinda wanted something nice to show for which uacpi could give lol)

[lost hawk]

No not for early table

But if you want devices to actually work properly you will need proper APIC drivers (both local and IOAPIC)

[haughty cipher]

yeah ik

done all of uacpi before as ive said, i literally wrote the zig bindings for it lmao

[lost hawk]

just checking

I, for one, don't support enough of x86 yet to be able to use it

[haughty cipher]

fair lol

[lost hawk]

Yay for RISC-V and DTB

[haughty cipher]

ok i really need to go to bed i meant to do that 90 mins ago

but on the other hand, PROGRESS

this is half of what i need mm-wise to do early page table uacpi

(aka half of what i need to implement map and unmap)

(this half covers the static mappings, the other half will cover temporary mappings)

but yeah thats basically the part that id consider the pmm done

i dont call it the pmm but thats the role it serves

made significantly easier by that this is the fuckin rewrite and the old version worked so all i had to actually do was adapt it to the new entry state

that and i had to rework PTEs to handle not-x86_64 arches. will implement the arm pte type and test that next i think

[haughty cipher]

so i dont lose this, for debugging networking when i get there

[haughty cipher]

i kinda wish there was a true equivalent to port E9 debugcon on aarch64 without using a normal serial port

[haughty cipher]

i should really write exception handlers sooner than later

for x86 it isnt too bad because the no exception handler just turns into a triple fault and qemu exits

but for aarch64 it turns into an infinite loop of instruction fetch faults and floods the qemu logs

anyway this message brought to you by my fuckin thing is page faulting (translation fault level 3) when i try to fill the PFMdb and idk why

ill probably have to double check the bit offsets on my PTE type and then maybe look into my control registers

ive a feeling it isnt attempting to re-do the page table walk on a cache miss

ok did some quick reading and i bet i just need to flush the TLBs

[haughty cipher]

alright aarch64 is now caught back up to x86_64

[placid swallow]

want me to pin it?

[haughty cipher]

would be appreciated yeah

[placid swallow]

https://klipy.com/gifs/yes-done-2

[haughty cipher]

thanks!

[placid swallow]

I havent gotten around to trying your bootloader yet either

soon

[haughty cipher]

called it a night on dev today after realising that i forgot to update SPSR after changing SPSEL to 1 which meant it was always using the old efi stack on aarch64 in the kernel accidentally lmao

[placid swallow]

lol yeah fair

[haughty cipher]

also found a very strange bug with a pointer in the bootloader being misaligned after conversion to virtual address that only happened on aarch64

and fixed it

still zero clue why it was happening lmao

since i was using the same function as is used in the other spot to convert the same structure

so my guess is somehow the pointer itself was invalid for some reason

but yeah at this point i think the only bootloader changes ill be making will be bugfixes until i get around to needing a framebuffer in the kernel

too focused on the kernel init rn, want to get zuacpi updated but zig is too good at lazy evaluation so it doesnt even compile shit unless i use it

and of course now its 00:40 and im reading aarch64 docs instead of going to sleep lmao

[haughty cipher]

i love git filter-repo

just imported a file from the old version of my kernel while preserving that file's history

[haughty cipher]

i hate off by one errors

ok i think i found it?

yep

off-by-one error in refcounting, thats fun

anyway, early tables access on zig master and rewritten kernel \o/

anyway im done for today. next up:

• buddy allocator for nonpaged pool
• slab for nonpaged pool
• put a TODO for numa and then ignore it indefinitely
• initialize nonpaged pool
• uacpi_initialize

[last pebble]

cool!

this is very old uacpi btw

[haughty cipher]

yeah i know, i need to update still

will be one command to update it as long as theres been no breaking abi changes im just lazy

[last pebble]

there was something minor between 2 and 3

[haughty cipher]

do you happen to remember what it was?

because zuacpi is handwritten bindings the only things that break it are function names and function signature changes

the latter is harder to detect

without reading changelog anyway

did that update quickly in any case, and at least for early tables its working still

[haughty cipher]

anyway ive got a headache now and some stuff to do after dinner so calling it a day for now

oh btw ⁨@last pebble⁩ the update of uacpi there is pushed to zuacpi as well

(I tagged the last commit on it before i started updating it to zig master also)

[last pebble]

nice!

[last pebble]

some struct member renames to support watcom

[haughty cipher]

ah ok then zuacpi isn't affected, only relies on the layout not the names

[haughty cipher]

trying to cram extra things into this struct is so much fun lmao

i think i can only use 24 bits for the free count of my slabs

(doing a SLUB allocator here)

the next slab in the root is only 36 bits but i reserve 40 for future 52-bit phys address support

(I only ever do 4k granules so the page shift is 12)

the slabs only need the forward link so i can put the pointer to the first free object in the space where the backwards link would go

and the other two u64s are full up already for the page->pte pointer and the metadata

and i refuse to make my PFMs not be a power of two size

and doubling it def aint happening

past me was def correct to write "Abandon all hope ye who enter here" at the top of the doc comments for the root memory manager file

[haughty cipher]

accurate

[haughty cipher]

oh god i cant use a singly linked list for the slub

i need to be able to remove from the middle if a slab page is fully free

i guess im doing offset central here

[haughty cipher]

I am now dealing with a chicken-and-egg problem, as is normal for osdev tbh

which is that i need to allocate a mapping of acpi processor id to lapic id/mpidr before i have my allocators up

technically i only really need it for mpidr, because the lapic affinity structure uses lapic id

but someone decided that the gicc affinity structure should use acpi id instead of mpidr

or i could just say fuck numa and not do any of this shit

[haughty cipher]

occasionally you find out that your loader has been only giving you the first page of your stack the whole time lmao

oh nvm i was misreading it

[haughty cipher]

fuckin atomics

why does this only work on debug mode? no idea

i kinda hate atomics

[haughty cipher]

oh god this isnt atomics is it huh

this is linker bullshit

no wait

this is loader bullshit

goddamnit i thought my bootloader was working lmao

ok im fuckin mad lmao it was a firmware difference

bug was my mistake for sure but i didnt catch it because the x86 ovmf memsets all new allocations to zero lmao

[haughty cipher]

alright with that fixed we have allocator working and are successfully calling uacpi_initialize

on both arches

next up probably basic interrupt handling

and then stack trace stuff and debug info loading so the interrupt handling can has stack trace

for the first time, imaginarium is running fully on its own allocators too

ive implemented a SLUB allocator on top of my already existing buddy allocator (which I call the pool for reasons derived from my nt inspiration)

took a lot of pain to cram the slab metadata into the PFM but i did it

anyway once ive got interrupts i can do threads and dpcs and that will be enough to get most of the rest of uacpi's kernel api up and running

and once ive got that all done I can get a performance number out of uacpi and learn if i get to be proud of my allocator's perf lmao

[haughty cipher]

starting to poke at interrupts a bit but no progress to show for it today (i got distracted like an hour into dev work and never got back to it before bed)

[haughty cipher]

\o/

[haughty cipher]

ok i dont understand what its talking about with unwind info invalid there

but i am popping off at having stack traces with source locations now

[haughty cipher]

oh right its this bug

i oughta open an issue for it or ask the zig people about how to maybe fix it lmao

[haughty cipher]

fixed it

[haughty cipher]

really wasnt expecting this to work tbh

not complaining though

so thats stack traces with symbol names and source locations for panics and interrupts on all arches i support

this is parsing the DWARF debug and unwind info from the binary and consists entirely of cloning the std implementation to swap out the "dl_iterate_phdr" and file io bits

literally 5 of the 7 files involved are copies from std with like one or two changes, and 3 of those 5 are only needed to work around a subtle overflow bug in the std that only impacts higher half binaries

i think i implemented it right that i can maybe extend it to do stack trace parsing for usermode applications too

[haughty cipher]

got the apic related madt entries parsing on x86_64 and also pulled in mcfg parsing from the old version of the kernel, and calling it a night there

maybe i should give the mcfg its own address space, or maybe make it lazier about mapping, but for now the syspte space is empty when it goes off so it wont fragment that by leaving the long-term mapping there

[haughty cipher]

also solved a bootstrapping problem id been struggling with by finally realizing that the solution is literally always the same: linked lists

[haughty cipher]

ruminating on what i want time sources to be and running into the conflict between my desire to keep "devices" entirely self-contained but needing timing globally and without various bits of the os having to keep a handle to the time source (especially for early boot timekeeping before i have device io up anyway)

so i think im going to just go play terraria instead lmaooo

[haughty cipher]

and its time to see how shit my allocator really is lmao

[haughty cipher]

my allocator is dying

thanks uacpi, for being an allocator stress test

aha i found the bug

forgot to initialize a variable right in pop and then push was failing an assert for it

and now the next bug

i think i see the issue

[haughty cipher]

the fuck

[haughty cipher]

my allocator is fucked it seems

very fucked, dear lord

[haughty cipher]

@last pebble does uacpi guarantee that frees have the same length as allocs?

ok nvm it was a fuckin off by one error

(still curious to know though)

[haughty cipher]

not really sure if my timekeeping is right though

oh right im still compiling -Doptimize=Debug from my allocator hell earlier

maybe releasesafe will work better

turned on all the optimizations

[polar pilot]

pretty sure it does, yes

i remember there being some challenges to this, so i can only assume it actually does track the proper sizes

[haughty cipher]

cool

anyway i was strategically cutting out all this earlier

the namespace init still needs me to implement pci stuff

[haughty cipher]

same speed ish on kvm

not too worried about optimization as of yet but i might look into this sometime

maybe its a caching thing since ive not done anything to set up PAT/MAIR atm and limine used to do those for me

but whatever

its late and im sleepy lol

[haughty cipher]

oh im stupid lmao

im using the reciprocal of the frequency

ill fix that tomorrow sometime

[last pebble]

yeah ofc

[placid swallow]

I went through a similar phase a while back. If its useful to you I decided I would make certain types of devices (timers and interrupt controllers come to mind) part of the kernel core and make them exempt from the usual treatment devices get.

[haughty cipher]

yeah thats what im doing now, was already doing with interrupt controllers - though im leaving e.g. HPET and ACPI PM timers as true devices

since im using uefi boot services stall to calibrate the tsc on x86_64 where needed and the arm generic timer just tells me the frequency already in a register

[haughty cipher]

ok time to swap the numerator and denominator of the frequency calculation and see what my actual time is

and also simplify this fraction for the hell of it

now that its reading in nanos instead of inverse-nanos

anyway, next up

once ive got the energy to dev more anyway

today in mildly annoying things: ive standardized the sub-namespace under arch for interrupt controllers as "pic" since all of these (apic, gic, whatever if i add another arch) qualify as programmable interrupt controllers (if not the og PIC) and the amount of times i mistype pic and pci and vice versa lmao

[haughty cipher]

🎉

mutexes and spinlocks are about half stubbed out

in that they do everything right unless they fail to acquire in which case they panic because i dont have threading yet lmaooo

[haughty cipher]

really ought to work on threading next tbh

[haughty cipher]

thinking i might change my physical page allocator's alloc page zeroed call to instead return a flag if it could get a zero page or not instead of unconditionally doing a temp mapping to zero it

since like 99% of callers immediately map the thing anyway, can save TLB flushes this way and just let them use the flag to decide if the page needs to be zeroed by them

(or more likely ill split that out instead of deleting the old version)

[last pebble]

Do you trim leading whitespace or smth?

[haughty cipher]

yes, though realistically i probably only need to trim trailing because zig's builtin logging all assumes that things dont have the newline already included

i trim both atm

[last pebble]

ah

[haughty cipher]

before i added trimming every uacpi log had a blank line afterwards lmao

[last pebble]

modern langauges dont really want manual newline so makes sense

[haughty cipher]

yeah

i also want do do some form of structural logging at some point here tbh

[haughty cipher]

ruminating on how i plan to do context switching

and i think i can do some truly hilarious nonsense here actually

thread context switching that is

not interrupt context switching

so the idea is

context switching can only ever happen in a DPC

which runs at the dpc irql ofc

which means context switching should only ever happen in one function

which means i can use inline asm clobbers to make the compiler do ALL OF THE REGISTER SAVING

lmao

and therefore the actual thread only needs to save rip/pc, (r)sp, and rbp/fp

and the function that does the context switch just saves the sp and fp in inline asm and saves the pc-relative address of a local label at the end of the function, and then loads the new fp and sp and jumps to the new thread's rip

which will only ever be that same local label or the thread entry point

which means that either it jumps to some new thread entry which will zero registers or whatever, or it jumps to the end of the inline asm which loads all the saved registers for the thread from its own stack

this takes ten lines of inline asm and a bunch of clobbers and includes saving the simd registers too since i can clobber those with inline asm

its all free*

and all the thread struct needs is three usizes

and because it runs at the dpc irql it can be interrupted by normal interrupts which save normal interrupt stack things and now dont need to worry about simd registers or anything

and all you need to do to make a thread yield is be at dpc irql so dpcs are masked, swap out the thread pointers in the percpu block for consistency, and then queue the scheduler dpc

note that technically this is only saving the true thread state for simd registers, its saving the state of within the dpc handler function for the normal registers

could also make this its own interrupt since it should only ever actually be invoked on the local processor, and allow invoking it using the int instruction

[haughty cipher]

tbh dont even need to do it in a dpc or interrupt

the whole point of this weird idea is to do context switching in a regular function anyway

and i can use some inline else trickery with zig to make simd saving conditional on a thread having a user part too

[haughty cipher]

having slept on it, the clobbering isnt great for simd registers because the compiler cant know exactly what combo is supported, so ill still need xsave

also realised i have a shrimple optimization i can do for kernel logging on x86 using rep outsb, rn it just loops over the characters

[last pebble]

Yeah I did something like that, there's a default iowrite_many implementation and there can be a specialization per arch

although I never got around to actually specializing it with rep

[haughty cipher]

yeah

found the rep stosb optimization when randomly scrolling through the pins in #resources lmao

[haughty cipher]

dunno if ill have time to actually implement this stuff today though

was going to do some work last night but instead ended up spending a couple hours helping diagnose a segfault in the zig build command that i found when updating deps lmao

and all because i got a pr merged into zig i wanted to try lmao

and today i gotta go to my brother's basketball game and then visit my grandpa in the hospital so no osdev until late in the day at least

at least i think i only need one thing to enter into a proper thread context and therefore implement the slow path for waithandles

just need the loader to give me the base address and length of the bootstrap stack so i can free it later

that is one thing ive realised, is that switching from a bootstrap context to a thread context really only needs setting up the thread object and putting it in the processor's currentthread pointer. dont need to figure out a base saved context or anything because thats only meaningful if the thread isnt running

the other improvement I want to make is to have the loader pass its logs forward to the kernel so the latter can store them in file or whatever I end up doing

I want to do structured logging but zig's fmt functions are built around comptime known templates so idk how id actually do that really

[haughty cipher]

with allocator counters enabled uacpi is telling me 1.26mil ops/s

which isnt bad for doing an increment or decrement every time i allocate or free something

[lost hawk]

What kind of allocator do you use?

[haughty cipher]

slub

backed by buddy

[lost hawk]

Slab+buddy is goated

[haughty cipher]

really is

ive got a few possible optimizations still (percpu magazines and the buddy is still using a global lock) but its still great

[lost hawk]

You love to see it

[haughty cipher]

also, getting all the way through to my end-of-kmain breakpoint instruction on aarch64 now too

complete with having a thread set as current now, though context switch is untested

[haughty cipher]

so the way i do context switching is kinda fucked up and heavily derived from the zig fibers implementation (thanks jacob young of the zig team for helping me understand that code)

the procedure is:

1. put a struct on the stack of the old thread. holds the old and new contexts and a pointer to the old thread object
2. using inline asm, put a pointer to that struct (on the old thread's stack) in a register, get the old and new context pointers from that, do the stack swap, and jump to the new context's RIP.
3. using inline asm clobbers, make the compiler save all the general purpose registers
4. in the new thread context, using inline asm output constraint, get the address to the struct (still on the old thread's stack) back from a register into a normal local variable
5. use the struct to get a pointer to the old thread (cant use function parameter because we're in the new thread's yield call to context_switch now)
6. schedule the old thread as appropriate

in addition, because we control what register the struct's pointer goes in, we can put it in a normal parameter register for the calling convention of thread start

and then use a naked function to load other parameters from the stack and jump to the kernel thread start and have it do the same post-context switch scheduling

the zig std impl specifically uses the register for the second arg of a function for that reason, and a naked function that puts the main first arg onto the first arg register before calling the entry function

[daring wyvern]

I think inline assembly for step 2 is questionable

Because there is no guarantee the compiler will actually put that pointer into the register, so fetching the old and new context pointers could just result in garbage results

Idk how exactly it works in zig as I’ve never used it but I know in C that would be a catastrophic mistake

[haughty cipher]

using an input constraint to put it in the register and specifying the exact register in the constraint?

[daring wyvern]

The compiler can still decide to not do that though

[haughty cipher]

(in this case zig is processing the inline asm into the llvm backend format and emitting it as part of the bitcode btw)

[daring wyvern]

Oh

I guess it could work then

But I know in C that is catastrophic

So zig inline asm is basically just sneaking asm instructions into the Bitcode

Maybe I should learn zig

[haughty cipher]

zig also has its own full compiler backend but its missing stuff for osdev atm

[daring wyvern]

Yeah I see

Like what?

Does it have raw pointer manipulation?

[haughty cipher]

yep

[daring wyvern]

Cool

[haughty cipher]

its missing some instructions for inline asm and it doesnt have soft floats (plus i dont think theyve got linker scripts fully working for it yet)

but you can just use the llvm backend

[daring wyvern]

Soft floats are unnecessary

[haughty cipher]

which does handle all that

compiler_rt has float functions i cant omit so i need soft floats if im going to disable hardware floats/vectorization to stop the compiler doing that stuff in my kernel

[daring wyvern]

Huh so it has its own mini assembly parser

Ahhh

[haughty cipher]

yeah the backend does it all

[daring wyvern]

I completely forgo the fpu in my kernel

Just let the userspace take care of that

[haughty cipher]

same

the backend gets given the inline asm string, the x86 backend does its own instruction emitting but only supports instructions its already generating from the codegen atm, the llvm backend translates the string into https://llvm.org/docs/LangRef.html#inline-assembler-expressions this format and passes it along

as a result zig actually lacks some of the nicety constraints, e.g. i cant use S or d or whatever i have to specify {rsi} or {dx} etc

the big thing zig has for osdev is first class support for bitfields in the language

[daring wyvern]

That’s actually a nice feature

[haughty cipher]

that and theres no concept of global malloc and very lazy evaluation so every non-io thing in the std works perfectly on freestanding without needing c headers or anything

and as of the master dev builds io also relies on a pluggable implementation which thanks to a pr i sent in like 3 days ago you can swap out the types of not just the vtable so you can even do io in your custom os using the std

also the build system slaps lol

but yeah in any case i think i can rely on the zig compiler adhering to my register constraint here, if only because the zig std is now doing it for fibers for some of the io implementations which means that if it breaks the entire language std will too and someone will notice and fix it

[daring wyvern]

Lazy evaluation is nice

[daring wyvern]

Damn

I might try to learn zig today

I’ve been needing a new lang to add to my portfolio

Because im mainly a C and python guy

With some c++ occasionally

[haughty cipher]

its a fun one. def still in beta so theres breaking changes every so often but its fun

[haughty cipher]

rn if you recursive clone my repo and run zig build qemu, and you have zig and qemu installed, itll do the rest complete with building the disk image, getting ovmf, getting all my deps, everything

wouldnt even need recursive clone but a couple of my deps im also working on while working on the os so i made them submodules to avoid managing all the different repos with different roots etc

ive even got three small tools i build as part of building the disk image that it automatically builds and runs for you in the process, in addition to the one thats putting together the disk image

[daring wyvern]

Will do

[haughty cipher]

zig master that is, i track master not the release versions (at least for now)

[haughty cipher]

oh god i have to tail call optimize this

well dont have to but i dont see any reason i shouldnt

so

my percpu thread priority queues dont need to take the percpu scheduler lock to enqueue

which means that during the process of context switching a higher priority thread could be enqueued to run

i can handle that without releasing the scheduler lock by recursing into yield if thats true after the context switch

but that can eat up stack

so im tail call optimizing that

actually hang on

i can do this much more sanely

[haughty cipher]

aaaand here we go

time for hell debugging

oh thats fun why is that gp faulting

ok its context switching to the idle thread which then immediately calls yield and context switches back to the kernel thread so thats progress

now lets see if i can get a stack trace from a different thread

YO

scheduler rn is priority round-robin. no boosting yet and also no sending threads to other cores

the other cores wouldnt even start up if i had them enabled in qemu rn tho

works on aarch64 too!

[haughty cipher]

I know how I want other processors to be able to detect when sending a preempt ipi is needed during schedule for priority preemption but I'm really not sure how actually do the preempting mechanically

the one downside of not using a full interrupt frame for context switch there is that I can't just use iret to handle it

though actually now that I think about it, I could maybe just use my normal context switch function and just use a target function that sets up an interrupt frame on the stack and does iret?

anyone got any ideas there?

[haughty cipher]

maybe i can just call yield in an interrupt handler tbh

ok after taking a look at obos (thanks obos for the inspiration) ive been reminded that i can signal eoi at the start of the interrupt as long as the irql is raised to mask the interrupt

which means that theres nothing a preempt ipi needs to do after the context switch

and therefore i dont need to completely jury-rig the context switch to carry even more data lol

for posterity's sake
https://codeberg.org/Khitiara/imaginarium/src/commit/9b1dade793b8c592beadb8514b9369e722c5ed61/src/krnl/thread/dispatch.zig#L108-L145
https://codeberg.org/Khitiara/imaginarium/src/commit/9b1dade793b8c592beadb8514b9369e722c5ed61/src/krnl/hal/arch/x86_64/context.zig

[haughty cipher]

next goals when i get to them now that i have context switching are waitformultipleobjects and dpcs

probably waitformultipleobjects first

dpcs will require me to finally get off my ass and figure out the gic after all

also, a caveat to my current context switch code is that xsave/xload or the aarch64 equivalent register saving is a TODO atm

this is ok as i dont have usermode yet so i dont have usermode apps to complain about not having an fpu or vectorization yet lol

[placid swallow]

You may also find this an interesting read: https://github.com/fuga-org/fuga/blob/dufay/kernel/core/sched/ule.c

And the original ule code too, its available on elixir

And mintia2 code is a good read too

[haughty cipher]

thanks for the suggestions

[granite whale]

is this a reference to descartes?

[haughty cipher]

the name? imaginarium is a reference to a fantasy book series by james a owen, tabula is just a latin word for tablet often historically used for book that i pulled out of thin air because i needed a name

[granite whale]

oh ok

cuz Descartes coined the phrase "tabula rasa" so i thought this was a reference to it lol

[haughty cipher]

(which word tabula came to mind because of the tabula rogeriana, which is a book that the latin speaking west decided to call the latin translation of "roger's book")

[granite whale]

i was also gonna say that you probably declined imaginarium wrong but it's voluntary

[haughty cipher]

definitely a fair guess, though i dont know enough philosophy to have heard the latin of that phrase as coming from descartes before

[granite whale]

my latin's a bit rusty lol

[granite whale]

ok ok

[haughty cipher]

yeah the fantasy book series its a reference to used that specific declination, possibly in a more correct context, and i never actually learned latin

technically imaginarium is the name of this project's kernel, and tabula is the name of this project's bootloader

[granite whale]

oh pretty cool

[haughty cipher]

and tbh both are called that because i couldnt think of anything better when starting them and i aint gonna rename the whole thing now 😂

i do like the names tho

[haughty cipher]

persistently running into a bug (in either my code, zig, or llvm, no idea which) where when unwinding a stack trace functions that are inlined replace the frame for whatever called them

which is a problem for assert lmao

[haughty cipher]

alright waits are done now

well sorta

havent tested em yet

kinda hard to

but they do compile

next up, irq handling details

[haughty cipher]

well this is uh clearly incorrect

the ioapic redirecting is right though, and its clearly installed some form of interrupt handler because it didnt go into my undhandled interrupt thing until the handler faulted

im really confused now

its not hitting the body of my interrupt handler

but normal reads of the isr list and the idt all look right

idk why swapping this load of what is guaranteed to fit in a single byte of memory from a load byte instruction to a load word instruction fixed it

but now im really confused

OH IM SO DUMB LMAOOOO

i was sign extending the vector instead of zero extending when getting the index to the interrupt handler array

so when it was interrupt 0x80, it sign extended it to 0xffffffffffffff80

now thats fixed, we got uacpi events

anyway with that done and pushed im done for today i think

next up will be IPIs and interrupt typing (since aarch64 is going to need that) and then i can do DPCs

and preempting threads on priority

after that will be the basic timer interrupt and then we'll start on drivers

[haughty cipher]

i forgot about aarch64 when writing that list of next goals lol

so definitely need interrupt typing first so i can get the gic going

[haughty cipher]

kinda hate that some (most?) PPIs are just. a fixed value you have to deal with now

though i guess thats true for ISA too

its just way harder to find PPI numbers for stuff like the generic timer than to find ISA irq numbers lol, probably due to just age

[haughty cipher]

oh there it is

finally found the table with info about generic timer interrupts on acpi

[steep spear]

Can't you run with the assumption that you have gic?

[haughty cipher]

i do

but i hadnt found the gtdt so i didnt have any way of knowing the gsi for the timer without hardcoding

and the gsi and gic intid have a 1:1 correspondance, found that under the gicd madt struct docs

in fact rn i run under the assumption that i have a GICv3 or GICv4

i dont support GICv2 and acpi doesnt do GICv5 yet

[steep spear]

Yeah intid for timers and such are architected for gic, so I was wondering why do all the hassle

Eh gicv2 is a lot worse, but some targets have it while gicv5 is unrealistic for a long while - I don't even know if gicv4 is implemented in anywhere but qemu

[haughty cipher]

mhm

[haughty cipher]

i could find nothing in the gic or arch docs saying what intids for timers and such are other than they are PPIs and giving a list of recommended values

whereas i already only support acpi atm and hate hardcoding so i can get gsis from that and converting gsi to intid consists of doing nothing

plus its a table so i can access it pretty early and at relatively low cost

could also get the intids from the device tree if i wanted

had to double check that one though in the qemu code since im not parsing or dumping devicetrees atm

[steep spear]

Huh yeah the list in BSA spec is a recommendation, I always thought it was architected. They do mention that not following the table would break compatability with some sw, and I'm more then happy being one of these software

[haughty cipher]

so yknow how i spent yesterday writing a bunch of generic irq handling code?

in my efforts to make something general enough for both arches i support, i managed to make something thats super suboptimal on one of them and still not general enough to support the other properly

time to think some more about design and then try again

[haughty cipher]

tbh the more i think about what i need the more i might end up with something not too dissimilar to what linux does

[haughty cipher]

but also maybe not idk

i dont think i need something that complex

ok why is this now getting stuck

oh

turns out spinlocks arent recursive

who woulda thought

[haughty cipher]

got the current version stuffed into x86 arch files yesterday and cleaned it up a bit, gic will be next but idk if ill work on anything today actually

feeling the pull to terraria lol

[haughty cipher]

might have to actually reserve a block of ram for DMA at some point too

rn my page allocator doesnt do contiguous physmem

only contiguous virtmem

which wont cut it for DMA

[polar pilot]

scatter gather dma should be good enough :^)

xhci and nvme does sg

and surely it's not a problem for anything else

or just do a buddy allocator :^)

[floral prawn]

so does ahci

[polar pilot]

even more of a reason to just let hardware handle non contiguous memory

[floral prawn]

if smth doesn't, then use the iommu

[polar pilot]

or do less than optimal io

if it doesnt do sg dma then its probably old

in that case it probably wont make a difference in performance anyway

[haughty cipher]

for me rn its for allocating tables for the GICR LPI stuff

and the GIC ITS

which mandate physically contiguous

not sure if they can work behind the iommu if that exists

i think theres multi level tables for LPIs but the docs are so hard to read and make sense of that i cant tell which component has them

if its the GICRs then i can just leave the ITS with a single page of command entries and be fine probably

[haughty cipher]

i have a buddy allocator but its only contiguous in virtmem

but yeah the issue here is the GIC

i can probably get away without contiguous physmem for now but he docs are not making it easy to figure out how to handle this stuff lmaao

i might also be able to just require smmu tbh

not sure

[haughty cipher]

i was today days old when i noticed the RSP0-2 fields in the 64-bit tss that have apparently been there the whole damn time lmao

sadly doesnt get set up automatically by an iret

but it exists at least

[haughty cipher]

idk how i just. never noticed the three fields at the bottom there lmao

[haughty cipher]

i love trying to write a struct that has 0x10000 bytes in it and trying to get all the offsets right

[haughty cipher]

gic is making me cranky lol

gicr discovery in this case

not sure how to determine the actual span of the gicr block

[haughty cipher]

slowly fiddling my way forward here

but its taking so long to get all the details of the GIC ruminating in my brain that im not 100% sure yet how i want to do everything

one thing is for sure though, i need to either add a physically contiguous version of the buddy allocator or rework it to be physically contiguous if i want to support the IDT and MSIs

should be able to handle the rest without that though

and maybe even the ITS anyway if i keep numbers small enough, idk

its is a problem for later

ive got types for the GICD and GICR register maps now

at least the base part of the GICR that is

[haughty cipher]

\o/

[haughty cipher]

theres a funny thing here

where my aarch64 interrupt controller code is both more and less far along setup at this point lmao

the x86 code runs sti to let interrupts to the processor but doesnt set the spurious flag to enable the apic fully

and on the other side, the aarch64 code fully enables the gic in all its parts but doesnt actually unmask irqs

[haughty cipher]

out here coming up with insane memory manager ideas

but i think i can cram the buddy order number into the PFM

which means i can refactor out buddy allocator to be a type

which means i can use it for the phys page free list and then use that inside out to fill up the existing virtual buddy allocator for nonpaged pool

or something like that

[haughty cipher]

i do think ill be changing my buddy allocator to use the PFM and reworking the free list of phys pages to be a buddy allocator yeah

esp because then i can use blocks instead of single-page mappings

what i may do because i still dont want a full HHDM is make a chain of usable page regions that are contiguous in virtmem and the nonpaged pool will map pages based on where they fit in that chain

[haughty cipher]

alright @last pebble if youre willing to dig that mac out of storage for this, the latest commit of imaginarium should work under accel on mac if you install zig master, recursive clone imaginarium, and run zig build qemu -Doptimize=ReleaseSafe -Dtarget=aarch64 -Dcpu=host

remove the cpu=host part if that isnt supported obv

anyway, i think the gic is fully initialized now?

so now just to set up actual routing and then i can go parse the generic timer tables

[haughty cipher]

in other news i think i finally have the irq abstraction how i want it

gave the irq struct an intnum field that holds either gsi info (gsi number, polarity, trigger type) or currently nothing else but eventually msi stuff will go there too

and removed all traces of the per-vector irq lists and pointers to the vector itself from the irq

those are now part of a parent struct that has the irq struct as a field, which a pointer to that field is provided by the arch-specific irq code

and then the arch uses containerof to turn the generic thing into whatever its outer struct is

the aarch64 one will have infrastructure for being stored in an intrusive tree (well a treap probably since zig got that in the std but i could make an avl tree or rb tree or even radix trie ig) of irqs since theres a ton of intnums populated sparsely

each irq struct has three function pointers and a few pointers worth of userdata storage

the functions being check, handle, and move

move is only really relevant for x86 rearranging irqs to fit fixed irq number requirements (learned that idea from obos, thanks @floral prawn), check returns a bool if the interrupt is actually from this device, and handle just does the actual work

i could maybe merge check with handle tbh, thatd be more in-line with how uacpi does it anyway, but idk

aarch64 shouldnt need move because theres so many interrupt ids that i dont really expect to have overlaps

(and most of those are told to me by firmware via acpi or devicetree too, the acpi GSIs map one-to-one to arm gic intids and devicetree just gives me pic-specific ids already out of the box)

there's also a feature that's been in the pipeline for a while now for zig that'll be free perf boost: restricted function pointers

which will allow marking function pointer fields with an attribute that says all possible implementations of this are in the compilation to turn it into a jump table switch thing and devirtualize it

so the next objective is the generic timer for sure

just one acpi table to get the interrupt number and getting the rest of the gic programming done

[haughty cipher]

had the idea thanks to some talk in the uacpi channel that i might try parsing the DBG2 table and use that to get early debug logging up on aarch64 without hardcoding or needing proper device discovery

[haughty cipher]

sadly qemu only generates DBG2 on aarch64 though

and in fact i think just the virt machine, though thats also the only one with acpi

[haughty cipher]

reminder to future me: when you get around to the apic timer, its ebx in the hypervisor frequency cpuid to get the apic bus frequency

i can never remember which register is which there lmao

[haughty cipher]

realising i can save myself 0x90 (144) static pointers that all point to the same thing in my x86 interrupt handling stuff

[last pebble]

yeah ill test it out in a bit

[haughty cipher]

not looking like ill get any dev work done today, but ig theres a chance for later this evening

[haughty cipher]

actually probably more than this

also ill be doing something in imaginarium that is technically in violation of the intel ia32e specs

but its a thing that i happen to know microsoft windows nt also does

which means im probably safe

[haughty cipher]

why the fuck did it start randomly putting this variable in .text

that aint right

[haughty cipher]

so i decided to move my IDT thunks into their own section

adding the linksection directive to them in zig caused this random other variable to start getting put in .text

even if i put a linksection directive on it too

which is strange

[haughty cipher]

somehow the irq stuff is slowly turning more complex than the memory manager

[haughty cipher]

ok so the planned interrupt map on x86:

APC LEVEL (0x1)
0x1F - APC

DPC LEVEL (0x2)
0x2F - DPC
maybe some debug stuff here too?

DEVICE LEVEL (0x3 - 0xC)
hardware device interrupts 
acpi SCI goes at one of these levels (maybe 0xB0)

CLOCK LEVEL (0xD)
0xD0 - clock/timer interrupt
0xDF - spurious vector

IPI LEVEL (0xE)
0xE0 - general IPI dispatcher

HIGH LEVEL (0xF)
profiling/perfcounter stuff goes here

on aarch64 (int vector decoupled from priority yay):

APC LEVEL (0xE0)
SGI #0 - APC

DPC LEVEL (0xD0)
SGI #1 - DPC
maybe debug stuff here too?

DEVICE LEVEL (0xC0 - 0x30)
hardware device interrupts
SPIs, LPIs, some PPIs. 
if acpi isnt hardware reduced, sci goes here too

CLOCK LEVEL (0x20)
PPI 0x30 - Generic Timer (may be a different number, see GTDT)
spurious is handled by the global dispatcher so doesnt get a priority on aarch64

IPI LEVEL (0x10)
SGI #F - general IPI dispatcher

HIGH LEVEL (0x00)
profiling/perfcounter stuff goes here

the astute reader will notice that apcs on x86_64 are using a lower intnum than the generally accepted minimum, this is intentional as vector 0x1F is not currently used for an exception and as windows NT relies on this remaining a viable interrupt vector i feel very little risk in doing that

anyone see any glaring issues with this setup?

[haughty cipher]

also, going to be making a RW spinlock soon

itll be useful for this irq stuff and also maybe for kernel debug info

[polar pilot]

why do you need a lock for kernel debug info

just initialize it once and you don't need a lock around it ever

[haughty cipher]

because im using the zig std's dwarf stuff which tries to be lazy about allocating stuff and ive not found a way to force it to preload yet

and im ok with it being lazy because dear lord the std debug info code is so memory inefficient holy shit

i really ought to fork it out into my kernel and rewrite some of it lmao

[polar pilot]

isn't the lazy part on the insides of std dwarf tho? as in, you just use the stack iterator or whatever and it handles mutation internally?

in which case you'd always need to acquire it for writing because you dont know ahead of time when it's going to populate shit internally

but tbh, how often do you need to look up debug info? probably for stack traces or panics

[haughty cipher]

yes. however, that code also relies on a fully functional file system driver so they recently added support to override it and the override part is what has the lock usage in it

[polar pilot]

oh wow

that's nasty

[haughty cipher]

in std it uses an upgradable rwlock

[polar pilot]

i seeeee

that's cool

[haughty cipher]

overridable part, not the override stuff itself

basically theres a type, std.debug.SelfInfo, that handles the load own debug info and then stack iterator (for debug info unwind) and stack symbol printing call into it, and by default SelfInfo is set up to switch on the target and be an elf one that uses posix dl_iterate_phdr on *nix platforms, a macho parser using osx stuff on osx, and a pdb parser on windows

but if the root file of your project defines a namespace called debug and that namespace has a type called SelfInfo in it, then all the std debug code uses your type instead

what i did is i copy-pasted the elf version from std into my kernel, replaced the calls to open and memory-map the binaries with ones that just return the static mapped binaries from the bootloader (multiple in case of split debug info), and replace the call to dl_iterate_phdr with just iterating the phdrs in the file since i dont have any other loaded modules with actual debug info

std.debug.SelfInfo.Elf is where the lock is

and thats the file i copied

atm i replaced the RwLock with a spinlock always used in exclusive mode just to get it working for stack tracing

and i can reuse the dwarf unwinding code that operates underneath it even

works well but if i had an upgradable rw spinlock i could use that to save on possible contention there in the fast path of already cached modules

and the memory usage is abysmal, its trying to use some sort of extra large buffer and somehow tripping the maximum block size of 2mb that my buddy allocator can give it and im shocked by that

only when i compile on full -Doptimize=Debug aka -O0 though, its small enough if theres literally any optimizations on

@polar pilot https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/debug the contents of this folder is sufficient to get fully working debug info with full eh_frame dwarf unwinding and symbol name and source location printing

and technically i could do it in less because the std provides cpu context types for these arches, i just use my own to expose additional registers that dwarf can handle but zig doesnt expose by default there

i only do that on x86_64 but i cant get it to override the cpu context type for only one arch easily

might add ELR to it for aarch64 though, and probably also tpidr when i get to that point

the thing with overriding the dwarf part to optimize the memory is that it would like double the amount of stuff in that folder lmao

[last pebble]

i remember about the macos tests i promised, ill get to it

(a bit later)

[haughty cipher]

no worries lol

[haughty cipher]

wrote the rw spinlock, and even included upgrade and downgrade functions for it that arent tested but i think theyre correct

[haughty cipher]

decided to start on writing the AP trampoline today cause why not

[haughty cipher]

i think the x86 ap trampoline is set up correctly but idk, cant be assed to do the rest of the setup to actualy make it go today

aarch64 ap trampolining is so much easier lmao

the thing i forgot for ap startup that made me go fuck this im not doing this today is i realised i need to actually map the ap trampoline correctly into lowmem lmao

not just put it in the right physical part of lowmem

[haughty cipher]

ended up getting bored on the other things so i guess im doing the ap trampoline now lmao

at least for x86

[haughty cipher]

this is going to drive me insane at this rate

[haughty cipher]

yup

though actually this bit if this fixes it will make this nicer

but i have my doubts

oh yeah no was thinking backwards here

issue was a different spot

so ill still need to do one(1) relocation on the ap trampoline

correction, two (2) relocations

the intial far jump and the initial gdtr

alright we made it out of the goddamn trampoline 🎉

yo we made it through the entire init now \o/

[haughty cipher]

alright next thing i need is log locking lmaoo

but it works

[haughty cipher]

only reason the actual trampoline cant do the relocations btw is its being reused for each AP

so itd compound the adding and mess things up

[haughty cipher]

next project on imaginarium will either be better logging infrastructure or the timers

@last pebble do you actually use your log ring yet or is it just a thing you wrote that has unit testing and nothing else?

in any case not expecting to get anything major actually done today, though also like every time ive said that i get a bunch of shit done late in the evening so who the fuck knows anyway

[last pebble]

Its hooked up to the logging locally, but not yet committed

But its like barely any code to hook it up tbh

[haughty cipher]

the big thing im not sure of wrt lockless log handling is i dont have a good way to get the length of a log message without printing it, and its highly variable

(variable because of stuff like stack tracing)

[last pebble]

doesnt zig have something like snprintf(null)?

this is essentially how I use it

[haughty cipher]

also, i plan on making a bit of code to parse ansi color codes in text and optionally strip them so i can multiplex logs out to file without color codes but also to serial or screen with color

[last pebble]

wouldnt it be better if adding color was the responsibility of the screen printer?

just keep a metadata struct next to the record

like log level etc

[haughty cipher]

oh true, i think theres a thing for that yeah. arguably inefficient to print twice depending on how involved my setup is - wouldnt want to do that for a stack unwinder that writes without storing the trace

[last pebble]

why not?

[haughty cipher]

maybe, though for things like dumping stack traces zig already has a thing for printing them in color based on a mode enum you pass that says how to do it, and parsing ansi out of the things would allow using that color too not just general log formatting color

[haughty cipher]

eh true, though im a bit iffy on doing a full dwarf unwind twice just to get a length

[last pebble]

oh u mean it like does unwind as u print?

[haughty cipher]

yeah

as opposed to capturing a list of frames and then printing later

[last pebble]

u can preallocate a static buffer to dump that to

and then feed that to logging

[haughty cipher]

true

and yeah just double checked, zig does have a discarding writer i can print to

which has a count function at the end

[last pebble]

yeah so u should be fine

[haughty cipher]

yeah

zig std also has MultiArrayList which would slap for this if it was viable to use lockless lmao

(its a type that stores a list of struct as a struct of lists automatically using comptime to get field info)

(and lets you get individual fields by index without reconstructing the whole item if you want, or you can have it reconstruct the struct for an index)

[last pebble]

interesting

are you going to implement a log ring also?

[haughty cipher]

probably yeah

[last pebble]

you can grab mine and translate to zig if u wish

[haughty cipher]

cool, i probably will at least try to yeah

[last pebble]

it has very very simple api so should be easy

and i guess the impl is easy to convert with like an ai agent

[haughty cipher]

i also rigged up a thing for my bootloader to log to memory so i can copy the bootloader log data into the log ring

[last pebble]

oh cool

[haughty cipher]

i have never and will never use ai for anything and i also refuse to permit any ai-involved contributions to any projects i run on principle lol

[last pebble]

lol fair

https://github.com/UltraOS/Ultra/blob/master/kernel/include/log_ring.h the header here is well documented at least

[haughty cipher]

cool

[haughty cipher]

the zig arraylist type has a print function that takes an allocator in case it needs to expand and just does a printf to the contents of the list basically, so my bootloader log setup uses one of those backed by the uefi pool allocator and also an arraylist of log structs, and i just put the index of the start of each print into the logmessage struct and make sure all the strings are null-terminated

its pretty good tbh

[last pebble]

yeah sounds cool

[haughty cipher]

and then theres also xxxBounded versions of print and append that just error if itd need to expand, so before i exit boot services i give it a bunch of extra capacity and then use those afterwards

the idea is the main kernel can then copy all of that into the log ring later and then free the stuff from the bootloader

[last pebble]

makes sense

[haughty cipher]

its just that the bootloader is single-core single-threaded so its completely unsynchronized, so ill need something better for the actual kernel lmao

[last pebble]

a good logging infra is very important imo

[haughty cipher]

yeah

[last pebble]

and it ties into /dev/kmsg later on

if u care about that

[haughty cipher]

idk that ill be putting it in the filesystem as a device like that (im a hater of devices-as-files), but i def want logging to file at the very least

yknow once ive got files lmao

[last pebble]

doesnt have to be a file i suppose

u can have some other syscall for that

[haughty cipher]

yeah

thats the plan

at least tentatively

[haughty cipher]

debating actually storing the log scope as a basic string in this

since the scope is always a comptime-known enum, its tagname is always a comptime known string and will presumably end up in .data

for now im storing it as a proper enum though so i can put numbers to em, store in a u16 instead of two usizes

god sometimes i wish zig had do..while

its like one of the only things i miss here

[haughty cipher]

the amount of integer overflow panics im getting because zig doesnt default to wrapping math unless i specifically use the wrapping operators and the c code im basing this on relies heavily on c operators wrapping lmao

@last pebble if youre willing, could you give https://codeberg.org/Khitiara/imaginarium/src/commit/97d4d1bab2450ca90a92b29a3f0eb2c38b602d54/src/krnl/log_ring.zig a skim and see if the logic matches? i dont have read done yet but the rest should be viable now and its not throwing any errors when i pass the bootloader logs through it, but figure ill ask the expert to give the code a once-over anyway

(as basic initial things to note: zig doesnt have fence so i cant do your relaxed and then fence optimization without writing my own fence in inline asm which i didnt feel like doing today, and zig uses llvm naming so what in c is called relaxed order is in this called monotonic)

(oh and any operator that has a % in it is a wrapping operator, so -%thing means wrapping inverse, +%= means add in place with wrapping, etc)

mostly asking for the sake of atomic ordering correctness lol, i still barely understand those despite now-years of trying

[last pebble]

Wait how did you make one so quickly

[meager mauve]

what the fuck lmao

[haughty cipher]

hyperfixation and zig being not thaaat different from c in this sort of code

[haughty cipher]

they figure that in 90% of cases people using fence really shouldn't be and in the other 10% if you know enough to actually be correct about wanting a fence you also know enough to write your own fence call with inline asm

[last pebble]

you actually wrote all of that by hand?

As far as barriers go its fine i guess

didnt look too deeply

[haughty cipher]

yeah?

took a few hours

[meager mauve]

thats stupid

[last pebble]

damn

[haughty cipher]

not like I had to actually figure out the logic lol, I just copied yours basically

I do understand the logic a lot more now after writing it

[last pebble]

thats cool

[haughty cipher]

just the asterisk that I still have almost no understanding of atomic memory order lmao

[haughty cipher]

take it up with them, I didn't decide that lol

[meager mauve]

well yes i know

its not your fault

its still fucking stupid though and i will make fun of them

[last pebble]

i agree that its a strange thing to do in a systems language

id understand if rust did it

[haughty cipher]

extremely fair

[last pebble]

since it aims for a more high level stuff

[meager mauve]

no, rust isnt fucking stupid

it has a pretty big perf gain in some cases

so it is very much worth caring about

[haughty cipher]

if you actually have evidence for this I think the zig people might reverse course, the logic for removing it was partly based on the gains being marginal and when people said it could be more nobody actually gave any concrete examples

[meager mauve]

on m2 relaxed atomics are significantly cheaper than acquire ones: https://gist.githack.com/orlp/44f35cafe887df34c01df6f4a7e3949f/raw/469750e640786197cee715192f3a737cc72d2757/apple-m2.html

if you only require an acquire ordering rarely (for example, if you have a refcounted pointer and you only need acquire on the final fetch_sub), using a relaxed op is slower by a very nontrivial margin

obviously 0 impact on x86, because it lowers to the same instructions

[haughty cipher]

mhm

[meager mauve]

and in general this kind of thing is what youd expect on like arm

[haughty cipher]

this is what i got when i asked about the decision last september fwiw

[meager mauve]

yeah idk about m3/m4 because i dont have that much money

i would expect stronger orderings to be more expensive

but i think if you find fences hard to reason about you should just. not use them lmao

[haughty cipher]

yeah

ill probably write inline asm fence helpers at some point for my kernel just to work around this but im definitely in the camp of finding them hard to reason about and thus probably shouldnt use them (because i find all atomics hard to reason about and generally avoid using them)

[meager mauve]

im not sure if asm helpers for fences are a great idea tbh

excluding compiler fences maybe

[haughty cipher]

mhm

either way im def just going to live with extra-synchronized reads in the log ring for now

[meager mauve]

makes sense

[haughty cipher]

its just for reading the rest of a struct or writing the whole of the struct, and without the fence i get a bunch of acquire reads/release stores for each field instead of a bunch of relaxed ones followed by a barrier

[meager mauve]

i imagine this kind of "atomic stores in a loop then fence" is exactly the kind of thing where fences are better but idk

[haughty cipher]

yeah

hence why i assume infy used it in the original

https://codeberg.org/Khitiara/imaginarium/src/commit/97d4d1bab2450ca90a92b29a3f0eb2c38b602d54/src/krnl/log_ring.zig#L504-L510 this is the range of lines

in infy's original c version these are relaxed and theres an acquire fence after

[meager mauve]

theres like a good chance it would be faster to load all except the last one relaxed and load the last one seqcst

[haughty cipher]

mhm

i might do that change but at this point atomic optimizations arent the biggest priority, especially since my options to run this myself are x86_64 and aarch64 tcg qemu running on x86_64 lol

[last pebble]

wouldnt that penalize x86

[meager mauve]

yeah dont do that on x86

[haughty cipher]

see this is why i avoid actually having to do reasoning about atomics lmao

i just dont understand this shit

i do think theres some stuff here that can actually be reduced to unordered

the one downside of certain things being the same type for both in and out of the atomic area means its easy to forget and use way too strong of atomics on the outside bits that dont need any synchronization at all except within the same processor

[haughty cipher]

one kinda funky side note is that while technically the way my bootloader protocol is designed and handled doesnt guarantee this, it happens that the implementation of the bootloader always puts the bootloader logs at the front of the linked list of protocol nodes, which means i can have the bootloader log entries be put in the log ring before any of the kernel ones

[haughty cipher]

anyway next up im either setting up the timer(s - one per platform atm) or going to expand log ring usage, not sure yet

if i do the log ring thing ill need to make a resetevent sync primitive too so i can have threads signal the writer thread

tbh not sure i need a log ring reader yet

the plan i think is going to be to have early debugcon in the normal log function instead of using the log ring for that, and then the log ring will get processed later to file or other output places

idk

either that or ill have a global flag that turns off manual debugcon logging and thatll get flipped once threading is up enough to get a log ring reader going

may have a second log reader for the later log readers compared to early debugcon actually? idk

anyone got thoughts there?

[haughty cipher]

another note from qemu-doesnt-adhere-to-standards land:
interpret the namespace string in the DBG2 table as relative to \_SB using the function that allows upward searching ideally

[haughty cipher]

because the DBG2 spec says its supposed to be fully qualified, but qemu goes and puts "COM0" in it for an object actually placed at "\_SB.COM0"

[haughty cipher]

time to rebuild qemu... again

i think the bug might actually have been fixed in qemu master like two weeks ago actually, ill have to test it

but that still means building qemu from scratch lmao

[last pebble]

Yeah uacpi has one

[haughty cipher]

ye i found it when lookin earlier, thanks

anyway, PAIN

thanks qemu

[last pebble]

Why do u need to compile it?

[haughty cipher]

https://gitlab.com/qemu-project/qemu/-/issues/513

this

im sick and tired of having to use this fuckin workaround lmao

it got seemingly fixed by an unrelated patch for arm64 whpx two weeks ago but theres no actual release number with that yet and i want to try it out, plus maybe also fix some other spec deviations in qemu even though i plan to harden imaginarium against them anyway lmao

[last pebble]

Oh

[haughty cipher]

mostly its that my build script currently has a workaround in place for whpx that makes things different on windows with accel on compared to every other possible setup

[haughty cipher]

whoo it was fixed by that commit to master

so next qemu release will have my issue fixed

(well i didnt file it but the issue ive been having)

[haughty cipher]

huh

now the bootloader is giving unaligned access exceptions on aarch64

i didnt change anything in it in a long time

[steep spear]

Qemu added alignment checks to device memory sometime in the last couple of years

Also beware of running on master, I find it sometime broken if youre not running on a specific tag

[haughty cipher]

mhm

so the stuff is actually misaligned after checking, i just thought i was running on a new enough version there

what im really confused about now is why the fuck zig/llvm is generating these misaligned accesses

and yeah apparently i was just using an old enough qemu before

[steep spear]

Idk about zig/llvm but gcc's binutils have unaligned access by default (for example memcpy) even if you pass the force alignment

[haughty cipher]

turns out i didnt have strict-align on even, id assumed it was default

adding that now and we'll see if it helps

[haughty cipher]

zig doesnt use prebuilt binutils, its compiler_rt based

so the memcpy etc get compiled in as part of the build using your build settings

and yeah sure enough turning on strict-align fixed it

[steep spear]

That's a little surprising, for the most part youd like the optimized asm routine in nearly 100% of the cases, so why build it over and over

[haughty cipher]

afaik it is build separate and cached

[steep spear]

Yeah but still you're losing on the optimization

[haughty cipher]

i think it is the builtin still too? not 100% sure. if not that its at least written in asm

compiler-rt is a standard thing on llvm fwiw

not just a zig thing

its where the optimized asm routine you mention is located etc

zig just uses its own because they dont want to depend on any external libraries (so no libc)

and llvm detects the memcpy pattern and adjusts it too afaik

[steep spear]

Oh I see, that's cool

[haughty cipher]

zig does have its own memcpy in its compiler-rt but thats to support non-llvm backends moreso than anything else

and it gets turned into the llvm builtin afaik

on llvm backend anyway

anyway got strict align turned on so now my os works with qemu newer than 9

[haughty cipher]

oh and the kernel writes to the log ring now

i might end up setting up a second serial port going to a file so i can test the log ring that way once i get devices up but idk

[haughty cipher]

ok what i actually need to figure out/set up next is PPIs and DPCs

the more i think bout it

DPCs will be fairly easy i think at least

[haughty cipher]

ok DPCs work on x86 now

really need to get the GIC stuff up though

thatll be next goal i think

rn the gic stuff is all stubs

or at least the routing part is

(and registering)

i do have it initializing

[haughty cipher]

alright we testing the gic now

oh the dpc fired holy shit

wasnt expecting it to just work

(for SGIs that is at this point)

there was a bug in my interrupt return thunk on aarch64 where it wasnt popping enough off the stack but after changing it from 34 to 36 u64s its now fully working??

the top line is in kmain, the second is in the irq dispatcher, the third is in the test DPC function, and the fourth line is back in kmain after the dpc returns

so with that the IRQ infrastructure is fully done, minus MSIs

MSIs will be fine? on x86_64 but hell on aarch64

love being told i have to allocate multiple 64k regions of physically contiguous memory

and my memory manager is really not set up for that rn

ive got ideas for how but low priority atm

calling it a day there for sure

next up will be setting up the timers