#tabula imaginarium: osdev experiments in zig
haughty cipher
last pebble
So tldr if it was booted in xapic mode use xapic format
haughty cipher
yeah i think so
thats what im going to do as a less hacky hardcoded fix than my initial testing version
haughty cipher
alright time to try this shit for aarch64
haughty cipher
alright got the irq rework set up on aarch64 too
may redo some of it a bit because i was being lazy in parts but otherwise its fine
aarch64 was easy because instead of having to write three separate irq domains i need one, total
at least for now
the lazy is making it one instead of two lmao
and the two would basically have the same code
so tbh its probably best not to do it as two just for that sake
(I may tweak somewhat and implement the logically separate second one on top of the first instead of entirely different, undecided)
actually hang on i already have that for the SGI stuff
just gonna reuse that lmao
and with that im done for today
haughty cipher
next up based on the poll results is storage, so itll be time to put all this abstraction to the test and write an NVME driver
nvme because its the most shrimple
nvme also goated by putting specs up on their site for free and easy to find
oh god wait i need MSI for that dont i
which is easily done on x86 but going to be a whole can of worms on aarch64
haughty cipher
not sure which arch's MSI stuff i should start with
aarch64 is gonna be way the fuck harder due to both being more complex and being less well documented
(the spec is pretty good but theres no implementation guidance or samples out there really apart from trying to parse the linux source which uh its the linux source and that part of it is a fuckin mess)
haughty cipher
what do yall think? easy or hard first?
also @last pebble on a technicality i have found a take-any-gsi situation, insofar as MSIs on aarch64 pull from a range in the same pool of interrupt ids as everything else and the acpi spec defines a 1-to-1 correspondance between IntIDs and GSIs
its not actually really the same though, acpi shouldnt ever give you a GSI in that MSI range (arm calls em LPIs)
last pebble
Hm
haughty cipher
also stole @finite ether's rmb/wmb functions since we're sadly stuck in a world without a fence builtin in zig for now
haughty cipher
i think before i do any of the actual MSI stuff im going to make a parser for the IORT table
cause ill need that for aarch64 MSIs and may as well do it now
finite ether
also stole <@379432087620550658>'s rmb/wmb functions since we're sadly stuck in ...
let's form a cabal so they can add it back
haughty cipher
let's form a cabal so they can add it back
if we can provide real benchmarks showing it would generate a significant speedup they might, the reason to remove was that the perf gains were negligible and 90% of uses the devs had found of the fence builtin were being used wrong because atomics are hard to reason about
finite ether
it's not a performance thing it's more of a correctness thing
you cant really implement seqlocks without them
haughty cipher
oh hey seqlocks are exactly what i was talking with the devs about about fences lmao
finite ether
I asked in the zig server and people there seemed like idiots so I left
haughty cipher
id asked in the zulip where they discuss actual dev on zig itself
finite ether
haughty cipher
this was the reply i got when talking about them
finite ether
we'd have to ask @tiny summit can we implement seqlocks without fences?
tiny summit
you can make every load a load-acquire, then you don't need a fence
finite ether
does that apply to the data?
tiny summit
yes
finite ether
then I'd guess this does have considerable overhead
tiny summit
it probably depends on how much data you want to read. if it's only two words or so, the overhead won't be that big
haughty cipher
then I'd guess this does have considerable overhead
everyone guessing that and never producing actual numbers except for some specific cases on m1/m2 that were since fixed by the hardware people is literally the argument the guy in the zig team gave me lol
everyone keeps throwing around probablies and guesses about the perf here and never numbers
tiny summit
but if you want to read, say, a page of memory, that'd be way more expensive than the fence
finite ether
I think I'll take a look at where linux uses seqlocks, take that same amount of data and benchmark it
haughty cipher
I think I'll take a look at where linux uses seqlocks, take that same amount of ...
if the numbers look significant on actual hardware we can put an issue up asking to get fence back and maybe have it accepted
haughty cipher
in the meantime im going to go back to my writing acpi table struct defs lol
tiny summit
on x86 it won't make a difference because every load is a load acquire
on risc-v you'll probably have a lot of overhead since there is no load acquire and it'll do a fence after each instruction 💀
tiny summit
on arm it'll probably depend a lot on the concrete implementation
tiny summit
yeah, that'd be interesting
haughty cipher
apple silicon
m1? m2? m3? which silicon specifically counts, m1 and m2 iirc had some nasty hardware bugs that made atomics slower than they should
finite ether
m3
tiny summit
on x86 the biggest diff that you'd see between fences and no fences is seq cst stores
for
static void WithStore(benchmark::State& state) {
volatile int x = 0;
for (auto _ : state) {
for(int i = 0; i < 10; ++i)
__atomic_store_n(&x, 42, __ATOMIC_SEQ_CST);
}
}
BENCHMARK(WithStore);
static void WithFence(benchmark::State& state) {
volatile int x = 0;
for (auto _ : state) {
for(int i = 0; i < 10; ++i)
__atomic_store_n(&x, 42, __ATOMIC_RELAXED);
__atomic_thread_fence(__ATOMIC_SEQ_CST);
}
}
BENCHMARK(WithFence);
im cooking up a test with seqlocks rn
finite ether
for
```c++
static void WithStore(benchmark::State& state) {
volatile int x = 0...
and for write() in seqlock I can make the sequence increment acquire too?
pub fn store(self: *Self, data: T) void {
const seq = self.sequence.load(.monotonic);
self.sequence.store(seq + 1, .acquire);
self.raw = data;
self.sequence.store(seq + 2, .release);
}
like this?
last pebble
Acquire store is ub lol
finite ether
ah fuck you're right im stupid
it's load-acquire store-release
I'd need a store acquire for this work
I think I have to have a load-acquire here too for the data copying
tiny summit
no for the writer side, acquire -> relaxed (for data) -> release is enough
tiny summit
disregard that ^
finite ether
tiny summit
you need store releases for all the data
if you can't use a fence
or a RMW acquire for the initial + 1 bump
finite ether
```
pub fn store(self: *Self, data: T) void {
const seq = self.sequ...
actually why is it usually done this way (+ a fence) and not with a rmw?
tiny summit
a release fence is free on x86, for example
(i.e., a no-op at the instruction level)
finite ether
ah I see
I know you don't know zig, but this should be correct, right?
fn memcpy_load_acquire(dest: *T, src: *const T, n: usize) void {
var i: usize = 0;
while (i < n) : (i += 1) {
dest[i] = @atomicLoad(u8, &src[i], .acquire);
}
}
pub fn load(self: *Self) T {
var ret: T = undefined;
while (true) {
const seq = self.sequence.load(.acquire);
if (seq & 1 != 0) {
std.atomic.spinLoopHint();
continue;
}
memcpy_load_acquire(&ret, &self.raw, @sizeOf(T));
if (self.sequence.load(.monotonic) == seq)
break;
std.atomic.spinLoopHint();
}
return ret;
}
pub fn store(self: *Self, data: T) void {
const seq = self.sequence.fetchAdd(1, .acquire);
self.raw = data;
self.sequence.store(seq + 2, .release);
}
monotonic is equivalent to c's relaxed fwiw
tiny summit
why is it named like this?
finite ether
no idea
haughty cipher
monotonic? thats the llvm name
tiny summit
yeah looks correct
finite ether
ok good
haughty cipher
this is with 4 u64s <@121733282676211714>
on your m3? send it and open an issue with zig for sure
finite ether
yeah thats on my m3
haughty cipher
thats some data right there
finite ether
well this is single-threaded, but it still shows the latency
haughty cipher
oh fuckin hell i forgot a field no wonder the values coming from this iort parser are nonsense
last pebble
looks like it would benefit from different per-arch impl
haughty cipher
well this is single-threaded, but it still shows the latency
you gonna open an issue on the tracker for this? because this is clearly a big issue at those numbers lol
finite ether
you gonna open an issue on the tracker for this? because this is clearly a big i...
I'm writing better tests first
haughty cipher
in any case i got iort now parsing nicely
finite ether
mhm well this seems better
haughty cipher
in any case i got iort now parsing nicely
logs should be fairly self-explanatory but the tldr is that
- pci requestor ids 0-2ff get mapped to smmu input stream ids 0-2ff
- pci requestor ids 300-ffff get mapped to its device ids 300-ffff
- smmu output stream ids 0-ffff get mapped to its device ids 0-ffff
and that all seems fairly reasonable to me tbh
with no smmu it just maps pci requestor ids 0-ffff to its device ids 0-ffff
which will be useful once i learn what the fuck a pci requester id is lmao
haughty cipher
mhm well this seems better
if youre using zig try with an without llvm also (assuming the aarch64 self-hosted can handle whatever inline asm you need to make the fences happen)
finite ether
I am using ReleaseFast so it should be llvm
yeah now I do multiple runs and get the mean and its even more conclusive
although to be fair, I am copying byte-by-byte while technically I could copy word-by-word but then the implementation would not be as generic
finite ether
you gonna open an issue on the tracker for this? because this is clearly a big i...
do you want to co-sign it or something
i think the more we are the better it will be heard
haughty cipher
do you want to co-sign it or something
yeah ill throw my name in it
feel free to put "co-signed by khitiara" or something like that in there
finite ether
how do I name it, "add @fence back"? lol
haughty cipher
id call it "a performance argument for adding @fence back" or something like that
finite ether
"Restore @fence / expose memory barrier primitives for efficient seqlock-style algorithms"
haughty cipher
(also just to confirm, youve not used the clanker for any of this, have you? cause zig is very very strict about their no-llm policy. im assuming you didnt given you literally contribute to the made-by-humans repo docs but may as well ask)
haughty cipher
"Restore @fence / expose memory barrier primitives for efficient seqlock-style a...
yeah that works
finite ether
(also just to confirm, youve not used the clanker for any of this, have you? cau...
I used it for the graph script
which is python
and unrelated
haughty cipher
then dont include the graph script in the issue ig
(also if you got the raw data do put that in the issue, its always good to have)
finite ether
yeah ill throw my name in it
well I cant create an issue for a language proposal lol
haughty cipher
just make a blank one
the language proposal option doesnt let you to stop people from suggesting random shit with no effort
finite ether
mhm ok
haughty cipher
also fence used to exist which means this is arguably a regression and thus a bug
finite ether
mhm do I attach the files or just copy paste
haughty cipher
idk
finite ether
my benchmark file is like 200 lines lol
haughty cipher
at that point maybe attach
haughty cipher
yeah its khitiara
thats my username basically everywhere
thankfully uncommon enough to not have issues
finite ether
im stupid your project is literally on codeberg
haughty cipher
lmao
im lucky in that khitiara is uncommon enough that if i have an account somewhere its almost always that
finite ether
haughty cipher
id make the image an embed instead of an attachment personally
but yeah cool, subscribed to notifs
finite ether
id make the image an embed instead of an attachment personally
idk how to do that
also I just realized my wmb is wrong so you should change that
make ishst on aarch64
haughty cipher
i think if you drag drop it into the editor itll work
finite ether
its faster
haughty cipher
also I just realized my wmb is wrong so you should change that
fun fact yours is what llvm does
i actually double checked
had to look up the fuckin instruction encoding
finite ether
yeah but here a dmb ishst is fine
since it's a write barrier
llvm has to be more generic for its thread fences
i think
haughty cipher
i ended up adapting for a proper user-code version of @fence instead of a write barrier
haughty cipher
`ret = self.raw;` this is not ub in zig right?
zig tries very hard to never have ub
its either defined or illegal
finite ether
`ret = self.raw;` this is not ub in zig right?
nope
last pebble
so what is a data race defined to
haughty cipher
so the way llvm does it (and i had to double check the encoding) on aarch64 is if the order is acquire it does ishld and otherwise it does ish
last pebble
linux has their own imaginary memory model
finite ether
idk I'll do your thing I think and just steal from llvm
haughty cipher
idk how to do that
 should work
what i couldnt figure out is what llvm does for barriers on x86, but yeah afaik they basically dont exist unless its a full barrier so whatever
finite ether
thanks
finite ether
what i couldnt figure out is what llvm does for barriers on x86, but yeah afaik ...
you dont need any on x86
you just need a compiler barrier
haughty cipher
which asm volatile with memory clobber ought to do
finite ether
yea
haughty cipher
what im doing rn is seq_cst on x86 gets mfence and the rest are compiler barriers, and aarch64 gets the same as llvm does
and then i have a separate function for dsb which is dsb sy on aarch64 and idk what to put in it for x86 rn its nothing
(actually rn on aarch64 its a dsb with two isbs around it because i was having issues)
yknow i might be able to get away with not reworking my buddy allocator if i only support 4096 different MSIs on aarch64
yknow only 4096 of them lmao
haughty cipher
sadge, ill have to actually allocate collection tables smh
alright this looks reasonable too
and everything accepted 4k page size thank god
setting indirection took too
thats good
haughty cipher
ok sick the math does add up here
i can get away with only discontinuous 4k pages here
haughty cipher
wonder if ill be the first hobbyist os to have full gic its/msis
if theres others ive not been able to find any info about it or i would be referencing that to figure this shit out lmao
anyway i should be good to turn the ITS on now
but im done actually coding for tonight
so that can wait for morrow
once i enable it i can start sending it commands
like MAPC and MAPD and MAPTI
and SYNC
SYNC is probably the most important one lmao
haughty cipher
ah fuck
i thought i could get away with this shit and nope
i cant tell this i only support 4096 LPIs, only the number of bits of total intids
which because the first LPI is 8192, means i can only support 8192 LPIs or 0
and the config table is one byte per LPI in required physically contiguous memory
haughty cipher
Me who has 4097 network cards
ig youll be fine then, the architecture said fuck you to me and i have to support 8192 MSIs actually
haughty cipher
well im allocating all the LPI and ITS stuff and enabling all the bits
almost scared to find out what i did wrong 😅
it aint erroring atm
ig i should hook up the interrupt dispatcher to the LPIs tbh
tiny summit
what are you using to test this? is it implemented on qemu?
haughty cipher
what are you using to test this? is it implemented on qemu?
it is implemented on qemu yeah
for the virt board
and tcg supports the ITS
which is what im implementing because im doing GICv3
the problem rn is i uh dont have any drivers that request MSIs yet
so i cant easily test this other than by if it throws an error on my config accesses
the actual config registers work though ive got enough debug printing to be sure of that
haughty cipher
ok so now i can reserve LPIs
and the ITS and LPI tables are all allocated and the ITS is enabled
the next thing then is to write the ITS irqdomain
which will reserve LPIs and then map (device id, event id) pairs to (cpu id, LPI) pairs
(note that LPIs are global numbers but the mapping of LPI to target redistributor is on the ITS so the cpu picks happen there)
once ive got the ITS irqdomain i can make the MSI domain on top of it, which will convert to the address,data pair that you need for MSIs
i think i may make MSI domains their own type actually
either that or make the domain be per-deviceid
honestly might go with the latter
linux gives msi domains their own type tho
idk
anyone got thoughts there?
i think making it per-deviceid makes the most sense with my current irq abstraction setup tho
haughty cipher
what i might actually do
thinking about it
is make an "msi domain" type
that has a single function to create an irqdomain for a given device (aka requester) id
(on x86 itll just have a singleton irqdomain that it returns)
and then each pci root complex or other arbiter of MSIs can have one of those
on aarch64 the msi domains will be each tied to a node in the IORT
and then creating a domain for a given device id will use the iort to figure out the msi target, and then wrap the relevant ITS domain
haughty cipher
ok im now running the MAPC commands
so time to find out if i fucked up
well ok first i gotta fix a billion compiler errors
well
it isnt erroring yet
haughty cipher
alright
MSI domains done
i might actually have the first hobby os implementation of GICv3 ITS MSIs now
i couldnt find any other examples anyway
(i wont say working implementation yet because i need a thing that generates MSIs first)
alright whats the simplest pci device i can put in qemu thatll let me generate MSIs on command
anyone got any ideas there?
tiny summit
One of the pci virtio devices
haughty cipher
eh ig
i dont know anything about virtio stuff maybe i should learn sometime
pulled up the virtio spec this morning, started reading it, and then was like this shit is unreadable and closed it
maybe it was an old version though idk
kindred isle
pulled up the virtio spec this morning, started reading it, and then was like th...
i also did the same 💀
haughty cipher
nvme was the next thing on my main todo list anyway
tiny summit
but still, virtio is probably overall easier
haughty cipher
mhm
if i dont do some simple thing to test MSIs with the next step was going to be writing an nvme driver anyway, so if nvme is relatively simple then i might as well do that?
idk
tiny summit
if you want NVMe anyway, it makes sense to go for nvme
haughty cipher
yeah
(strictly my next goal after the irq abstraction rework and adding msis to that abstraction was to do storage, and i was planning to start with nvme there because ive heard its one of the simpler options)
last pebble
the cool thing about virtio is its the same driver for all devices
the only thing that changes is the request format that u put into the queue
haughty cipher
ah, so i could make a virtio "bus" driver that then all of the individual device types sit on top of then?
last pebble
the spec is a bit all over the place but thats because it tries to be platform abstract
thats how linux does it
last pebble
virtio-blk is very close to nvme, u just have submissions and completion queues
but very much simplified
there isnt an admin queue or anything like that
but yeah nvme if u wanna test on real hw
virtio has nice things like net/blk/gpu/sound/rng/console etc etc
haughty cipher
but just so i have it written down: the idea for virtio then would be to have a driver that detects based on the PCI vendor/devid stuff and then adds a HID of the form VIRTIO\{whatever} to the list and re-queues the device for discovery, and the individual virtio things would discover that hid and use a private dispatch IRP to put stuff in the virtio queue
haughty cipher
i will probably still do nvme first though
not because im equipped for real hardware atm but because my goal with this project is learning and thus i have a lot more interest in emulated hardware than i do in stuff like virtio
haughty cipher
fair
false, it will boot with nvme in bios mode if and only if the total memory is 2G or less
above that and the nvme emulation puts its BAR in high physmem
last pebble
oh ok
haughty cipher
but also my uefi-only bootloader for this project wont boot in bios mode either so 
not exactly limiting myself there by using nvme atm, not any more than already
oh yeah i do also need to make the x86 msi domain too
rn ive only implemented MSIs on aarch64 (because its way fuckin harder there)
tiny summit
how does it boot with other pcie devices and >2G of RAM in bios mode then?
only some of them are put above 4G?
haughty cipher
only some of them are put above 4G?
yup, I've only noticed it with nvme, and also most of the others aren't needed by the bios anyway
it's only a problem when your boot partition is affected
tiny summit
yeah i'm just surprised because i always boot from virtio-block and never noticed that
but ig it depends on the numbering on the pci bus or something like that
last pebble
yeah i'm just surprised because i always boot from virtio-block and never notice...
iirc this has to do with the bar being 64-bit
seabios is more eager to relocate those above 4g
tiny summit
ah that makes sense
last pebble
they had numerous patches dealing with bar relocation blowing up old guests and then fixing blowing up new guests
its a constant back and forth they have
i even fixed one of those bugs myself i think
it was failing to properly map all the bars if u had like 8 nvidia cards there with giant bars
tiny summit
if you have that you probably want uefi anyway
last pebble
u do true
its also useful so u can lock the guest down with secure boot to prevent it from messing with the drivers
since nvidia is really strict about guest driver matching host driver
or everything explodes
haughty cipher
iirc this has to do with the bar being 64-bit
yup exactly, nvme defines it as a 64-bit memory bar
whereas the other storage options dont have to be 64-bit
haughty cipher
anyone know a good reference on PRPs? really struggling to figure those out
last pebble
PRPs?
haughty cipher
an nvme thing
last pebble
Ah
tiny summit
iirc in the simplest case you just have two pointers to data
there is a way to do an indirection and then you get another page (?) of pointers
haughty cipher
yeah that indirection thing is what i dont understand
and it like depends on what the command is? maybe?
ok yeah
for some commands the first pointer is indirected
for others the first is just a pointer and the second depends on how many pages get crossed
tiny summit
for extra fun there is also a SGL (scatter gather list) extension but that's separate from PRPs :^)
haughty cipher
yeah
tiny summit
iirc the SGL is only useful to read/write partial blocks to a buffer though
like, if you want the first half of a sector to go to a different buffer than the second half
haughty cipher
also if its nvme over fabrics then youre required to use SGLs
nvme over memory (aka pcie) requires PRPs for admin commands but doesnt specify anything in particular for IO queues
tiny summit
also if its nvme over fabrics then youre required to use SGLs
yeah, that's also the only case of SGLs that we support in Managarm
haughty cipher
i dont really plan to support nvme over fabrics at this point, at least until i find an actual case for it
tiny summit
our use case is netboot
haughty cipher
ah
finite ether
the cool thing about virtio is its the same driver for all devices
But like isn't it significantly harder to do virtio gpu
last pebble
depends if u mean modeset or 3d
finite ether
They should've made it where you can just send opengl ops directly 
last pebble
commands themselves are easy, the hard part is the byte stream of commands and instructions etc
haughty cipher
i started doing osdev partly because i was trying to learn modern 3d graphics programming and hit the point of desparately wanting to get away from it
id take digging through osdev hell specs and shitty firmware over 3d graphics programming any day personally
finite ether
how does userpace OpenGL/Vulkan interact with the GPU anyway
ioctl?
3d is pretty fun if you do it yourself (entirely in software)
haughty cipher
how does userpace OpenGL/Vulkan interact with the GPU anyway
the actual opengl or vulkan is being loaded dynamically from the driver using a getprocaddress type setup
finite ether
ah
haughty cipher
how the driver actually does it internally is up to it
haughty cipher
yeah its probably ioctl internally unless the driver is fully usermode
haughty cipher
but the actual opengl/vulkan impl is part of the driver
finite ether
(petitgl is my single-header ANSI C opengl 1.1 partial implementation 
)
haughty cipher
lol
id generally take vulkan over opengl at this point
just because its a lot closer to how the actual hardware is set up
finite ether
well vulkan is way more modern
haughty cipher
yeah
last pebble
mesa has generic implementation for all of those
that eventually calls into gpu specific callbacks
finite ether
yeah mesa is great
haughty cipher
side note my bootloader protocol is inspired by vulkan
last pebble
in what way lmao
haughty cipher
in what way lmao
in the way that a lot of vulkan functions support extensibility/adding features by using a singly linked list of structs (which are all basically tagged unions)
my bootloader passes a singly linked list of tagged protocol structures to the kernel and the kernel iterates it and handles any it knows about
haughty cipher
yeah
last pebble
thats the only sane way to do it i think
haughty cipher
its a common and good design, it just happens that vulkan is the first place i personally encountered that pattern
(vulkan is slightly different in that you pass a list of empty tagged structs and it fills the ones you passed whereas this boot protocol just passes all the things)
toxic iris
Isn't multiboot2 more of an array of tags instead of a linked list?
haughty cipher
i wouldnt know tbh, ive not used multiboot anything
toxic iris
As in, mb2 tags don't have a "next tag" pointer or whatever
last pebble
dont think it matters
the idea is u have a tagged union + size
or just tagged union + next link
haughty cipher
i dont actually have a size, just tagged union and next pointer
the size doesnt matter because its allocated from a fixed buffer by the loader anyway
toxic iris
I also do that for my prekernel
toxic iris
i dont actually have a size, just tagged union and next pointer
This specifically.
haughty cipher
(technically what im doing wouldnt be called a tagged union in zig parlance because tagged unions are a zig feature, but they dont currently have a defined ABI which means i cant pass them across the compilation boundary safely so i use an untagged extern union and a separate tag field, and convert to the zig tagged union later)
i may actually rework some of my boot protocol stuff at some point soon tbh
mostly because i realised that some of the stuff im putting in high memory doesnt actually need to be there and could be left in low memory or even on the bootloader stack
since i dont kill the lower half page mappings until after i use most of the boot protocol stuff
im undecided if i want to keep the memory map in highmem tbh
im increasingly thinking i might make imaginarium a serial-terminal-first os tbh
dont have framebuffers even being fetched by the bootloader yet and not yet feeling like i miss that at all
plus if i ever get a teletypewriter itd be sick af to use my os from it
tiny summit
commands themselves are easy, the hard part is the byte stream of commands and i...
virtio-gpu transports Mesa IR to the host
so it only works if both the host and the guest run Mesa
tiny summit
there's also a vulkan variant, that might be more portable / less dependent on Mesa but I'm not sure
last pebble
yeah
there's some feature bit thats set if 3d is supported
in virtio-gpu config space
but it is a pretty niche thing
last pebble
@haughty cipher btw
as far as nvme goes
leo used to recommend that u start with an older version of the spec
newer versions assume u know stuff and leave out a lot of info
haughty cipher
oh good to know, ill look for it
last pebble
u can grep leos messages to check for which versions specifically
haughty cipher
ok the real question is which leo
last pebble
no92_leo
haughty cipher
thanks
last pebble
np
haughty cipher
goddamnit discord
last pebble
lol
tiny summit
I don't think it's that newer specs leave out stuff but they split it over 10 documents instead of 1
haughty cipher
didnt expect it to defocus the search bar when i hit tab on the from bit
last pebble
discord ux strikes again
haughty cipher
yeah
haughty cipher
I don't think it's that newer specs leave out stuff but they split it over 10 do...
also this, ive got three docs i need open to get everything for the spec if i use current versions
last pebble
why the hell did they do that
haughty cipher
idk
theres the "base specification", the "nvm command set specification", and the pcie transport specification
last pebble
thats cool and all but they could also you know... ship a full version as one pdf file
haughty cipher
yep lmao
last pebble
same as amd prms for example
haughty cipher
yeah
last pebble
they have one giant book or small ones
haughty cipher
in this case the "nvm command set" is the stuff that actually deals with nonvolatile memory (yknow the main thing you use nvme for)
the base spec doesnt assume the controllers can actually do anything except basic ident stuff and io controllers can create and destroy io queues
last pebble
yeah
haughty cipher
well and cancel and flush, the base spec assumes the io controllers can cancel and flush
if you want read and write you gotta pull up the nvm command set spec
last pebble
who needs that stuff
haughty cipher
and the pcie spec is self-explanatory except also you gotta reference a bunch of shit on the main spec too because a lot of the pcie transport spec assumes you already know what the base spec requires of memory transports (of which pcie is the one)
last pebble
reminds me of ahci
where the command set lives in a completely different proprietary spec
that u can only find leaked copies of
but it's way worse there in general
haughty cipher
mhm
at least with nvme theres just a bunch of specs on the site you can reference lmao
haughty cipher
but like theres a bunch of cross referencing where the pcie spec says the vector is associated with a queue and then you gotta find the bit in the create io completion queue where theres a field for vector or the random one sentence deep in the config register description for the admin completion queue address that says "this queue is always associated with interrupt vector 0"
yeah ig
last pebble
did u implement anything in code yet?
last pebble
Fair
haughty cipher
and likely to be busy the next three days too but less so lol
last pebble
Ill be curious to see how zig drivers look
haughty cipher
before i actually start it i do need to get x86 msi domains set up
since i got so distracted trying to piece together the hell that is the aarch64 gicv3 msis i forgot that
last pebble
Msi on x86 is incredibly easy so thats good
haughty cipher
yeah
im glad i started with the gic for it, because thats the main driver of how i abstract em
last pebble
What's required for msis there
haughty cipher
the abstraction for normal interrupts is mostly driven by x86, for msis its the other way around
ok so
each msi source has a device id
and then theres this thing called the ITS
and to trigger an msi you write an event id to an mmio register in the ITS
and it uses some implementation defined mechanism to get the device id
then it looks up the device id in one of its tables
(which is a two level table set up so the lower level is exactly one page and the upper level has to be physically contiguous and you have to allocate it yourself but also it might not support the two levels which would make you make a fuckin huge table)
the device's entry in that table is a pointer to another table for that device
the ITS looks up the event id in that second table
(which you also have to allocate yourself)
that second table lets the ITS translate the device id,event id pair into a combination interrupt collection id and intid
the interrupt collection id is looked up in yet another table you allocate yourself
which converts from the ICID to the mmio address of one of the per-cpu redistributors
last pebble
What the actual fuck
haughty cipher
and then the ITS sends the interrupt with that id to that redistributor
that redistributor then looks up the interrupt priority and mask bit for that interrupt id in another table you have to allocate yourself which has a minimum of 8192 bytes and must be physically contiguous
if its too low priority it sets a pending bit in another table you allocate yourself
if its high enough priority to get sent then the IRQ exception vector gets triggered, and you can read the interrupt id from the redistributor
and then you have to use your own data structures to convert the interrupt id back into a combination of device and event id
and then propogate that back
last pebble
Me when irq controller needs 10k loc to send a damn irq
haughty cipher
afaik mine is the first hobby os to implement this lmao
last pebble
Msi for gic?
haughty cipher
Me when irq controller needs 10k loc to send a damn irq
this is only for MSIs, the non-msi interrupts are super super ez
haughty cipher
Msi for gic?
yeah. i couldnt find an example of it anywhere and the docs are kinda bad so the only way i figured this shit out was by cross-referencing the spec with the linux source which means having to decode the mess of duplicated abstractions for bitfields and some macro hell and whatever the linux abstractions are
when i say macro hell
last pebble
Lol
haughty cipher
last pebble
when i say macro hell
What the hell
Did they really not find a way to deduplicate that code
haughty cipher
so this takes a builder for the main command in the arg of the main function, and a builder for the relevant sync command in the arg of the macro, and then makes a routine which sends the main command and then the sync command if the main builder returned a sync object to associate with the command, and then waits for it to finish
last pebble
What is its anyway?
haughty cipher
interrupt translation service
last pebble
Ah
toxic iris
when i say macro hell
haughty cipher
(oh yeah the top level of the device table and the pending and config tables youre allowed to write to with your own code but everything else you cant and have to send commands to the ITS using its ring buffer which you also have to allocate yourself)
and if you ever modify the LPI tables (thats the one the redistributor checks) you have to send an INV command to the ITS to make it tell the redistributors to reload that from cache
haughty cipher
its turning a builder that uses not the normal bitfield helpers even though the normal bitfields are used elsewhere in the same fuckin file into a thing that builds and sends the command to the ring buffer and waits on it
and to wait they have to use stall because the interrupt controller wont interrupt you when it finishes
haughty cipher
there is actually use for the command ring in GICv4 fwiw, since that version adds full virtualization and you can use the commands to send virtual ipis around and reroute virtual msis to other virtual machines
so its got hypervisor use
but also you have to use the command ring to send an inv command if you mask or unmask the interrupts at the controller level
which is the main thing
https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/hal/arch/aarch64/gic/Its.zig#L448 anyway heres my code to reserve an interrupt from the ITS if youre curious
haughty cipher
and the redistributor stuff is https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/hal/arch/aarch64/gic/lpi.zig here
haughty cipher
Did you test on real hw?
dont have any
last pebble
and the redistributor stuff is https://codeberg.org/Khitiara/imaginarium/src/bra...
Do u use atomics for mmio there?
What arm hw has gic3 even?
haughty cipher
linux does that and i was like eh why not
haughty cipher
What arm hw has gic3 even?
no idea lmao, but its the best option ive got on qemu
also gic4 is back compatible with gic3 if theres hardware with that
gic5 looks a lot nicer than gic3/4 but the acpi spec support for gic5 is still in draft and also qemu doesnt support gic5
last pebble
linux does that and i was like eh why not
Never seen atomics being used for mmio on x86, I guess its an arm thing
haughty cipher
yeah ig
i debated between that and just making it volatile but i still dont trust volatile due to past issues with it that probably arent relevant anymore in general let alone in zig lmao
last pebble
Are asm stubs just not popular anymore
haughty cipher
i use those on x86
because even though i probably dont need to worry about that one weird possible amd issue i already have them
last pebble
Does Linux not use them on arm?
haughty cipher
idk about in general
its linux nothing is consistent
but all the GIC code uses relaxed atomics for registers
last pebble
They rolled ad hoc mmio helpers for this specific device?
Wtf
Inconsistency and code duplication final boss
haughty cipher
lemme check the spec actually
maybe it has something to say about accessing the registers
ah yep
there ya go @last pebble
last pebble
That's so strange
But at least explains it
So they dont use the normal stuff like writel() etc
haughty cipher
yeah
i think technically unordered would work here
doesnt have to be relaxed
just needs to be atomic (as in a single instruction)
actually no
it does need to be atomic properly
last pebble
Well thats just talking about the standard mmio memory mappings
They should guarantee order
haughty cipher
mhm
last pebble
just needs to be atomic (as in a single instruction)
Usually arm does those in multiple or smth?
last pebble
Can u send a link to where they use atomic stuff for that on Linux
last pebble
Interesting, writel actually looks like an even stricter version
haughty cipher
if i follow that chain as best i can
it is an asm helper
as it turns out
i may make an mmio helper thing at some point
got some ideas for that now that im thinkin bout it
last pebble
The normal one also slaps some barrier on top
haughty cipher
mhm
in that case im probably best leaving this as atomic for now
wish i had real hardware for this
just a basic sbc with an armv8 cpu a gic3 acpi and pci and im set
and a serial port ofc
last pebble
Yep
haughty cipher
so a thing im considering atm
i could theoretically whip up an extern or packed struct in zig that holds a single int and has read and write methods that act as mmio helpers
probably extern so it can't get out into a packed struct
there already is one for atomic values after all
could even use that to set up pci config space as a struct etc because its an extern struct and then just inherits size and alignment from its one (1) field
downside for pci config space would be support for legacy access
tiny summit
ARM does not support atomic access to arbitrary MMIO
By atomic, I mean ll/sc
Access to the same mmio region are strongly ordered on ARM but they are not strongly ordered with respect to other memory accesses
such as cached accesses or accesses to other mmio regions
tiny summit
ARM does not support atomic access to arbitrary MMIO
Or are we talking about atomic access to tables in RAM?
placid swallow
pulled up the virtio spec this morning, started reading it, and then was like th...
older versions of the virtio spec are more concise.
hoary aurora
pulled up the virtio spec this morning, started reading it, and then was like th...
i found it "not good, but implentable"
luckily they just ship the full inizialization routines for most stuff
languid orbit
that u can only find leaked copies of
ata acs? but that is not ahci (in that it's not a part of it and it could be used with something else, like ide). however it's definitely that t13 are greedy pvssies. worse than them are only t10.
haughty cipher
By atomic, I mean ll/sc
then how would you interpret #1312232715193683988 message as far as requirements?
tiny summit
I think that's referring to normal loads/stores, not rmw
last pebble
i think it just means regs must be written and read via their real type and not split up into multiple
especially since from looking at linux source it uses relaxed mmio helpers for it
haughty cipher
yeah
I'm def gonna do my mmio value type at some point then
itll be an extern struct with a single field so itll inherit its representation fully from that field
and then have methods that take self-pointers and do the ops on that
haughty cipher
alright did that change now
haughty cipher
I think that's referring to normal loads/stores, not rmw
double checked and linux is using normal load stores except for one weirdly specific range of processors where the loads are acquire lmao
love fuckin hardware quirks
tiny summit
Yeah that's a funny quirk
haughty cipher
https://codeberg.org/ziglang/zig/issues/31022 now if only this bug didnt exist so i could be slightly more general with my asm constraints
(fyi @finite ether on that one if you didnt know about it - "m" contraints in inline asm are currently broken in zig and cause a double-dereference if you pass something.* in)
haughty cipher
realised i wasnt parsing pci capabilities yet so now im trying to add those but also got stuff going on later so probably wont get that done today
finite ether
(fyi <@379432087620550658> on that one if you didnt know about it - "m" contrain...
It's not even a thing anymore no?
Now it's a struct with memory = true
haughty cipher
constraint not clobber
(where youd put register or whatever that you want that thing to be put in)
this
haughty cipher
things tonight got canceled so ig i can keep working on pci cap iteration stuff
haughty cipher
🎉
haughty cipher
alright my brain is slowly turning to mush so im gonna quit while im ahead
tomorrow: ima send a command to the controller and see what happens
haughty cipher
realised a stupid mistake i made that ill have to fix tomorrow
which is that its currently throwing the virtual addresses for the completion and submission queues in there lmao
haughty cipher
ive gotta make a generic virt_to_phys function tbh
i keep forgetting to use phys address when setting stuff up because i dont have an easy helper
haughty cipher
alright the pci driver is now wrapping the msi irqdomain with a per-function one that handles masking
and made a phys_from_virt function to make it easier to set up device stuff
got shit going on in 15 mins but after im done all that ill try sending a command in the admin queue
polar pilot
i think you should have a dma memory allocation api
haughty cipher
i think you should have a dma memory allocation api
i do sorta now, had to make one for the GIC on arm
it just reserves a block of physically contiguous memory that you can pull from
rn on x86 it defaults to reserving nothing, on aarch64 it allocates 16mb
polar pilot
i was thinking about something that'd let you abstract iommu nicely from drivers
haughty cipher
thats def something ill do eventually
haughty cipher
i actually do sorta already
in that the io resource manager takes stuff like BARs or acpi memory resources, maps them, and then provides both a physical address and pointer to the driver
and it could easily just provide the address in the iommu instead of the physical address later
polar pilot
yeah but what about allocating buffers at runtime? for stuff like queues
haughty cipher
ill provide a dedicated allocator for that
and an iommu version of phys_from_virt
polar pilot
ideally it should all be abstracted away from drivers, they shouldn't have to distinguish between memory in iommu and host address space if that makes sense
haughty cipher
thats where the allocator interface in zig comes in handy
polar pilot
driver code written "with iommu in mind" should work without an iommu without changing anything
i think i'm stating obvious facts lol
haughty cipher
so the plan is to provide two allocator instances to the driver, one for device-facing and one for host-only. if theres no iommu then the two will be the same but if theres an iommu the device-facing one will use it
if that makes sense
haughty cipher
(technically the device one wont be a true zig allocator instance, itll be a wrapper that adds a virt-to-device function too, but the idea is the same and if no iommu then virt-to-device is just virt-to-phys)
(or actually more likely ill make the alloc function return both a pointer and the device-facing address)
polar pilot
that sounds nice
haughty cipher
may actually make that abstraction after my appointment today tbh
just to not need to rework everything in the kitchen sink later when i add iommus
haughty cipher
that sounds nice
got that code up now, thanks for the prodding to actually go make that
maybe now ill go make the x86 msi stuff that ive been ignoring lmao
my brain: i should do the aarch64 MSI stuff first cause its harder
also my brain: alright aarch64 msi stuff is done lets jump right into nvme
in the process ignoring all of the x86 stuff lmao
haughty cipher
alright we got x86 MSIs now
haughty cipher
ok this is very broken lmao
x86 is just freezing and never getting the interrupt
and aarch64 is timing out in the ITS stuff
im hungry though so taking a break for dinner
haughty cipher
oh i think i found the ITS issue
very dumb bug lmao
ok now both of them hang when waiting on the nvme controller to process the command
oh
im stupid
maybe enabling msi/msix is a good idea if i want to get interrupts from it
ok the aarch64 its stuff is still at least part broken but i can try this on x86 now
its at least dispatching the interrupt but only on xapic not x2apic which is weird
haughty cipher
pcie is hell actually
and i hate how hard it is to find the specs
turns out i need to walk upward to find possible alias conditions for the pci requester id
haughty cipher
ok got that sorted
now to see if i cant get it to release the thread
oh hey it did pog
oh lmao no wonder the nvme controller is returning an error
i had the wrong opcode
haughty cipher
popping off actually
now why the fuck does it break under whpx
gonna test kvm before drawing too many conclusions
current guess is x2apic related
actually hang on i have a way to turn off x2apic
lets test that
nope that changed nothin
haughty cipher
oh damn thats a funky race condition
haughty cipher
anyway calling it a night now
so to be specific, it now works perfectly on TCG
but not on any other accel even if i turn off x2apic
the MSI just never gets delivered
i even have qemu trace logging to tell me that it sent the msi
it just never actually interrupts in a way i can detect
tomorrow im going to try stalling instead of blocking and see if its just getting masked out for some reason
(im suspecting a race condition with the context switch stuff, which i need to redo my wait function to handle better)
haughty cipher
anyone got experience with msis on qemu have ideas why this would break?
haughty cipher
ok it is a race condition
scheduler code is going to kill me
ok i think its fixed?
im scared that ill get hit with some stupid race condition later and im just getting lucky rn
i smuggled the mcs lock token through to the new thread on a yield-for-wait so the context causing a thread to wake cant fuck with the thread wait list until after the context switch
and then because thats now synchronized more correctly i can make the context switch not raise irql above .dispatch
which means the MSI isnt getting lost due to that lol
id have expected the apic to buffer that msi but ig it doesnt?
i dont fuckin know
haughty cipher
continuing to pop off
last pebble
Letsgo
haughty cipher
idk why the capacity thing is set to 0 tbh
haughty cipher
Letsgo
if i multiply the size in blocks by the lba size i get the exact same number as my build script passes to the disk image tool i use in the "disk size" argument btw
so im very happy there
last pebble
Cool
haughty cipher
hmm its returning "invalid queue size" as an error when i try to make the queues
oh oops
i forgot to initialize the entry size in the config register
alright that works now hell yeah
haughty cipher
poppin off again
now lets see if it works on aarch64
having good abstractions: still goated
didnt have to change anything it just worked
anyway now itll be time to figure out how i want to structure the IRPs for block io
initial plan is to have fully separate operations for streaming and positional io
block devices only support positional, stuff like serial only supports streaming
yknow what fuck this i should probably be using a DPC to signal sync objects anyway
haughty cipher
well i think i fixed the race condition
but in the process i discovered a really annoying edge case and now im mad
not actually an edge case tbh
more something i didnt know about the type system
tbh maybe just a weird edge case on calling conventions idk
i think its all workin now tho
callin it a night my brain is turning to mush again lmao
and as usual the literal first thing that happens after i shut down the dev machine is i realise something i did wrong lmao
a bug in waiting as it were, as circumstantially it cant break rn but is def wrong
kindred isle
Quick question. For my kernel would I be able to make it with C and Zig? And have some parts in both?
For my arch abstractions I wanted to use weak symbols but I’m not sure if that would work with zig?
Eg I want to make low level hardware stuff in C and other things since I may port them from my old kernel
(Doing a rewrite)
kindred isle
My kernel is a pile of dookie for a lot of things
So I lowkey need to rewrite
What’s the starters guide packet for zig
haughty cipher
Quick question. For my kernel would I be able to make it with C and Zig? And hav...
weak symbols do work in zig but why would you want to do it that way when zig's lazy eval is so good for this exact case
kindred isle
So unimplemented functions for an arch can trigger a panic?
haughty cipher
Eg I want to make low level hardware stuff in C and other things since I may por...
mixing c and zig does work actually quite well but again why would you want to do that
kindred isle
Yeah I’m considering just maybe not doing that?
Idk
I’ve been cleaning my desk and computer to get stated with this 
haughty cipher
So unimplemented functions for an arch can trigger a panic?
using zig lazy eval will make it compile error if a function gets used that isn't implemented but it's like 3 lines to make a panic stub for it?
finite ether
So unimplemented functions for an arch can trigger a panic?
they should trigger a linker error not a panic
kindred isle
mixing c and zig does work actually quite well but again why would you want to d...
Mainly just scared of not being able to use zig proper
kindred isle
they should trigger a linker error not a panic
Couldn’t that get annoying for arch ports depending on how much i need to have?
I could just have a template of stubs tho :p
kindred isle
What’s that do?
finite ether
well it would panic with something like "panic: not implemented"
kindred isle
Ahh
haughty cipher
Mainly just scared of not being able to use zig proper
polar pilot
there isn't a todo() in zig but you can just as easily do @unimplemented() or @panic("todo")
or rather @unreachable()
@unimplemented() is not a thing lol, mb
haughty cipher
unreachable is a keyword not a builtin
polar pilot
oh it is? yeah my memory is a bit spotty
finite ether
unreachable is also UB
polar pilot
but tbh unreachable might be bad
@breakpoint() if anything
or whatever it's called
finite ether
there isn't a todo() in zig but you can just as easily do @unimplemented() or @p...
so when are you joining zag
haughty cipher
and I wouldn't use it for todo stubs
kindred isle
Okay so. To get started I should get it loading with Limine and flantern and overwriting print functions to work right?
finite ether
and I wouldn't use it for todo stubs
the compiler can assume unreachable is unreachable so it's UB to actually reach it
polar pilot
in debug it'll panic but in other modes it probably will just generate code that's ub
finite ether
yes
haughty cipher
the compiler can assume unreachable is unreachable so it's UB to actually reach ...
sorta - if youre in a mode with runtime safety unreachable is a panic
polar pilot
so don't rely on that obviously
kindred isle
Also I did hear backtraces in zig are much easier?
polar pilot
std.dwarf
haughty cipher
Also I did hear backtraces in zig are much easier?
yeah i got a fully working setup in mine that wraps std.dwarf to just load from the kernel file memory mapped blob that limine or whatever gives you
finite ether
I'm not skill issued so my panic is empty
haughty cipher
even supports loading split debug info from a module but the zig build system cant do the splitting without system objcopy so i cant be assed to actually use that yet
like seriously, do just steal https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/debug this folder and the debug.zig in the level above
itll give you stack tracing
kindred isle
even supports loading split debug info from a module but the zig build system ca...
So I could make the debug info seperate?
That sounds useful and somthing I’d want
haughty cipher
its not that useful unless your kernel is huge and you want to strip it to save space
polar pilot
it's a thing that linux distros do nowadays, they ship separate -debug packages that contain just the elf files with debug info and no code
kindred isle
True but it’s cool
finite ether
windows historically did that
polar pilot
well, it has no other way
haughty cipher
all you have to do to use my debug thing is set SelfInfo.kernel_bin to the kernel file from limine and if you split debug info set SelfInfo.debug_bin to that loaded as an optional module from limine
polar pilot
i don't think you can embed debug info inside the PE itself
haughty cipher
ah PE cant embed debugging info
yeah its always PDBs separate
finite ether
before pdbs they had dbg
windows nt 3.1 is a .dbg file
that's not a pdb
they probably switched to pdb because it's fancier
haughty cipher
yeah
either way its always been separate debug info for windows
i think nowadays they do something even fancier for kernel debugging but idk how that works
polar pilot
from my experience there's also just plain pdbs for kernel itself and the drivers
windbg downloads them from msft servers and loads them automatically
haughty cipher
ah cool
wasnt sure if it was still a plain pdb file
knew windbg downloaded them for you tho
kindred isle
this is melting my brain a little 
looking at the build scrips and the eld stuff
100% a skill issue tho
haughty cipher
this is melting my brain a little <a:wahhgone:801309853116923924>
looking at th...
whats melting about the build scripts?
i get the elf stuff i barely understand that and i wrote it lmao
kindred isle
i dont see where the limine things get put into the right sections (looking at the zag kernel for that)
haughty cipher
i dont see where the limine things get put into the right sections (looking at t...
well i dont use limine
so thats probably why
haughty cipher
ahh
kindred isle
wait why does zag have stuff for SDL??
haughty cipher
wait why does zag have stuff for SDL??
zag has a user-mode emulated version
kindred isle
ahhh
kindred isle
true sorry
haughty cipher
no worries
i dont mind offtopic here, youre just more likely to get an answer over there
kindred isle
i do sadly think i wont be able to do zig
though the more i read it the more its making a little sesne
haughty cipher
rip
kindred isle
maybe my next reweite
haughty cipher
lol
finite ether
i dont see where the limine things get put into the right sections (looking at t...
I dont have limine support
yet
there's a limine zig template tho
haughty cipher
yup
languid orbit
i don't think you can embed debug info inside the PE itself
PE has means to embed debugging information inside the executable. There's a dedicated .debug section for this and a special Debug Data Directory entry in the Optional Header pointing to that. In fact debug info can be placed in .rdata or anything else. pdbs are just most commonly used to strip the executable size. Then internal debug section references this external file.
haughty cipher
TIL
also just realised i never made a post in #introduce-yourself here lmao
fixed that now
languid orbit
I can't write there somehow. Even though I've never introduced myself too. Not like I'm going to though.
haughty cipher
I can't write there somehow. Even though I've never introduced myself too. Not l...
well someone gave you the introduced role at some point is why
haughty cipher
I can't write there somehow. Even though I've never introduced myself too. Not l...
#introduce-yourself message
you wrote in the introductions channel in a reply and someone probably looked at it and gave you the role without reading lmao
languid orbit
I forgot about it. Well, if i forgot that i set a password to one of my laptops fw, then.
finite ether
you wrote in the introductions channel in a reply and someone probably looked at...
its automatic
haughty cipher
its automatic
well a bot automating it def wont read the message to realise its a reply not an introduction so my point still stands
haughty cipher
also yeah i thought it was manual
tiny summit
it used to be automatic but the bot feature got lost in some rewrite of the bot
haughty cipher
saw something about that when looking through random channels earlier
oh thats a mood lmao
anyway i cleaned up how drivers access device properties so its harder to accidentally fuck it up like i did last night
and moved the namespace identify command to the namespace driver
haughty cipher
ok how the fuck do i want to handle read/write IRPs
the big questions:
- do i want to support scatter/gather stuff or only contig buffers
- do i want the nvme driver to allocate dmamem for transfers or rely on the caller provided buffer
- do i pass offset by bytes or make a separate block read/write that uses lba offsets
haughty cipher
anyone have thoughts on those questions?
polar pilot
i think SG-first design would be nice, especially since you can easily just pass an SG list of 1 buffer and achieve the same behavior
and i think that the driver should take byte offsets + sizes that are multiples of the block size and then you can translate that to LBA + block count yourself
and i’d say rely on passed in buffers for zero copy IO
haughty cipher
i think SG-first design would be nice, especially since you can easily just pass...
only possible risk there is if the nvme controller doesnt support SGLs i need to do temp buffers depending on the setup
i do agree about going SG-first with the design tho
esp because zig's reader and writer types are built on it (well built on being able to transparently use preadv and pwritev on posix platforms)
polar pilot
but in case NVMe does not support SG you can have the IO subsystem not do proper SG
idk how to explain it better
haughty cipher
yeah itll just have to allocate a dma buffer in that case, and i dont really expect the partmgr to be asking for SG io anyway tbh
polar pilot
i mean you can still do IO to contiguous physical memory without SGL
haughty cipher
(fwiw windows sidesteps this by requiring SG io to have each buffer be a full page which is just kinda dumb and makes it useless so i dont want to do that)
polar pilot
with NVMe
haughty cipher
yeah
ig ill make an error code if SGLs arent supported to tell the caller to do contig buffers
if the buffer passed isnt aligned right to just use anyway
haughty cipher
alright i wrote a read dispatch for nvme namespaces (only PRPs for now because lazy), not tested yet though ill do that after dinner when i start writing the partition manager driver and try to read the mbr/gpt
one thing ill definitely be working on at some point (probs as part of the partition manager driver actually, might make it a filter driver tho but then ill need to rig up the io manager to also do filter driver attachment) is a cache system
basically just cache page-size groups of blocks in some form of basically non-intrusive radix tree
haughty cipher
🎉
polar pilot
biiiig
haughty cipher
nothing im doing actually sets that guid but it is the same one the bootloader logs at least
haughty cipher
🔥
haughty cipher
Ill be curious to see how zig drivers look
https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/dev Nvme.zig and the nvme subfolder comprise the nvme driver for this if youre curious to look at it
and im happy to explain things
nvme read is also the first true async-capable driver call ive implemented
well nvme commands in general ig but read is the only one so far that operates truly async instead of just using a dpc to release a manual reset event that the caller blocks on on completion
last pebble
https://codeberg.org/Khitiara/imaginarium/src/branch/rewrite/src/krnl/dev Nvme.z...
thanks
haughty cipher
(also the other stuff in that dir is other drivers fwiw)
ok time to test all this on aarch64
should? be fine?
this stuff isnt touching arch things
but who knows
last pebble
did qemu actually generate some sort of an aml device for nvme?
haughty cipher
haughty cipher
did qemu actually generate some sort of an aml device for nvme?
for every pci slot actually, but i skip nodes unless theres a HID/CID or existing device object to attach the node to based on ADR (which the pci driver dispatches to the acpi driver to request)
last pebble
oh ok
haughty cipher
if i dont theres however many empty devices with only an ADR under the pcie-pci bridge
i think its to support pci hotplug using acpi hotplug?
not sure
haughty cipher
yeah
i wanted to make sure my pci bridge stuff is fully fledged so i attach a pcie root port to the root complex and then a pcie-pci bridge to that and the nvme controller under that bridge
last pebble
it would be more realistic if it was
pcie -> pcie bridge -> pcie root port -> nvme dev
nvme is pretty much always behind a pcie root port
haughty cipher
i had to implement a painful edge case in the MSI requester id to make this work lol
haughty cipher
nvme is pretty much always behind a pcie root port
good to know
i might do that then idk
haughty cipher
unfortunately "pcie-pcie-bridge" isnt a device type qemu knows about
pcie-root-port is though, im using that
last pebble
lemme find what its called
haughty cipher
👍
last pebble
this doc is really good https://gitlab.com/qemu-project/qemu/-/blob/v9.0.1/docs/pcie.txt
2.1.2 To expose a new PCI Express Root Bus use:
-device pxb-pcie,id=pcie.1,bus_nr=x[,numa_node=y][,addr=z]
pcie-pcie bus is this basically
it also requires aml to detect i think
last pebble
the pxb device is a device that actually exists on bus 0
it is a bridge
pxb -> pci express bridge
and on real hw it's like that also usually
haughty cipher
oh hm ok
last pebble
basically the root complex aka bus 0 almost never has actual pcie devices attached to it
only legacy endpoints like chipset devices or other pci bridges or pci express bridges
haughty cipher
mhm
im gonna go with the pcie switch thing instead for plugging the nvme into
i knew the nvme usually isnt direct on the pcie.0 bus just didnt know that it wouldnt normally be behind a pcie-pci bridge
anyway plugging it into a downstream port of a switch it all still works
on both platforms
last pebble
yeah because its a pci express device so it cant be on a pci bridge
haughty cipher
yeah because its a pci express device so it cant be on a pci bridge
well it can but you lose the extended part of extended configuration space
which im not using yet so it didnt impact me yet
haughty cipher
yeah
last pebble
whats your current topology?
haughty cipher
- pcie.0 root complex
- pcie-pci bridge (nothing behind atm but there for later)
- x3130-upstream
- xio3130-downstream
- nvme controller
+ whatever the qemu machine does by default
ah u did the switch thing
haughty cipher
q35 puts the legacy stuff behind a pci-isa bridge and a few things direct on pcie0, virt on aarch64 puts the hardwired stuff direct under \_SB on acpi with no pci intermediaries
last pebble
the sad part about these switches is u run out of bus numbers very quickly, since every downstream port takes 1 extra bus number since its actually a bridge also
its kind of dumb
haughty cipher
yeah
what threw me off at start is a lot of what im pulling from is what windows will give me as a device node tree
but windows calls all of the kinds of pci(e)-bridge/port/whatever "PCI-PCI bridge" in its ui
last pebble
lol
haughty cipher
well apparently upstream and downstream are labeled correctly too
but the naming is based on the HID and CID
which are based on the pci class code
which is the same for pcie-pci bridge, pci-pci bridge, and pcie root port
so it uses the original name for it, which is PCI-to-PCI Bridge because history
in any case i got the partitions enumerated by my partmgr driver and nvme is reading blocks for me successfully so rn im happy enough lol
last pebble
Cool
haughty cipher
the sad part about these switches is u run out of bus numbers very quickly, sinc...
well i have a limited number of devices here so less worried about running out of bus numbers and i want a complex enough topology to actually test my pci code
last pebble
Yeah
That's why they added these segment group numbers
Which is only used by cursed arm devices for some reason
haughty cipher
im glad i did, i found out about this bullshit that i have to do now
(which i do)
last pebble
What does this even do
haughty cipher
What does this even do
so do you know what requester id is in pci?
last pebble
Nope
haughty cipher
ok
its a 16-bit mask made from the bus number, device number, and function number
taking up 8, 5, and 3 bits respectively
and its the id that goes over the pci bus to say where a message came from
some interrupt controllers use that id as part of the lookup from msi to actual vector to send
but
the bridges and root ports and the like change the id when they forward a message
so in my old setup with the pcie-pci bridge shit, the nvme controller had requester id 0x0208 (bus 2, device 1, function 0)
but the message the interrupt controller got had requester id 0x200
because the bridge nuked the device and function bits
last pebble
Is this requester thing only relevant for the insanely complex arm irq controllers?
haughty cipher
which then caused the interrupt controller to look up in the wrong table, error, and not forward the irq
haughty cipher
Is this requester thing only relevant for the insanely complex arm irq controlle...
uh it might be relevant for iommu stuff too?
last pebble
Ah ok
haughty cipher
i dont have iommu yet so im not sure
last pebble
Probably
last pebble
im glad i did, i found out about [this bullshit ](<https://elixir.bootlin.com/li...
Is this generic PCI code?
haughty cipher
Is this generic PCI code?
drivers/pci/search.c so yeah, though its got a pluggable callback
last pebble
Interesting